launcher/.hermes/plans/sglang-support.md

SGLang Support Implementation Plan

Background

vllm-spark-launcher is a Rust CLI for launching vLLM/SGLang containers across DGX Spark GPU clusters via SSH. The current SGLang support is incomplete — it only changes the engine name but uses identical docker commands as vLLM. SGLang has different defaults and CLI flags that need proper handling.

Problem

Current gaps:

1. VLLM_HOST_IP is always set — SGLang uses SGLANG_WORKER_IP instead
2. No SGLang-specific CLI flags (--tp, --dp, --pp vs vLLM's --tensor-parallel-size)
3. No distributed backend selection (Ray vs MP) — currently hardcoded to MP
4. No way to distinguish SGLang-specific distributed config from vLLM's
5. Container naming doesn't reflect engine type (always uses recipe name)
6. No health check/repair infrastructure for container issues

Implementation Plan

Phase 1: Recipe & Config Changes

r1. Add distributed_backend field to Recipe struct

• Add distributed_backend: String with default "mp" (matches current behavior)
• Supported values: "mp" (default), "ray"
• This controls docker flag composition in build_docker_command

r2. Add distributed_backend to recipe TOML schema

• Document in AGENTS.md: distributed_backend = "mp" (or "ray")
• Ray mode: --tensor-parallel-size=<nnodes> + RAY_ADDRESS env var on worker nodes
• MP mode: --tensor-parallel-size=1 + --pipeline-parallel-size + --nnodes + --node-rank + --master-addr + --distributed-executor-backend=mp

Phase 3: SGLang-Specific Docker Command Composition

r6. Update build_docker_command for Ray/MP backend switch

• When distributed_backend == "ray":
  • Head node: normal vLLM/SGLang command with --tensor-parallel-size=<nnodes>
  • Worker nodes: --tensor-parallel-size=<nnodes> + RAY_ADDRESS=<head_ip> env var
  • No --nnodes, --node-rank, --master-addr, --pipeline-parallel-size, --distributed-executor-backend flags
• When distributed_backend == "mp" (current behavior):
  • All nodes: --tensor-parallel-size=1 + --pipeline-parallel-size=<computed> + --nnodes + --node-rank + --master-addr + --distributed-executor-backend=mp

Environment variable changes:

• If engine is "vllm": set VLLM_HOST_IP
• If engine is "sglang": set SGLANG_WORKER_IP (instead of VLLM_HOST_IP)

Phase 4: CLI Enhancements

r4. Implement doctor subcommand (checks node readiness + container state)

• Concurrent node checks: docker --version, nvidia-smi, NCCL env var validation
• Container checks: docker ps --filter label=spark-launcher, engine health endpoints, Ray cluster status
• Output: pass/fail per node, actionable suggestions

r5. Implement --repair flag (builds .repaired image on head node)

• Generates Dockerfile fixing common issues (missing deps, wrong entrypoint)
• Builds <image>.repaired on head node, tags it
• serve auto-detects and prefers <image>.repaired if it exists

Phase 5: Wire Up & Testing

r3. Wire up doctor handler in main()

r7. Update serve() to prefer repaired images

• Before build_docker_command, check if <image>.repaired exists
• If yes, use it instead (log it)

r8. Integration testing

• cargo build succeeds
• Dry-run with --recipe glm51 -d shows correct flags for both engines
• Dry-run with distributed_backend = "ray" shows Ray-specific flags
• Health check on empty cluster

r9. Update AGENTS.md

• Document new recipe fields, commands, and behavior changes

File Changes

File	Changes
src/main.rs	Recipe struct, distributed_backend, env var logic, build_docker_command rewrite, doctor handler, repair handler
recipes/*/recipe.toml	Add distributed_backend field (optional, defaults to mp)
AGENTS.md	Document new fields, commands, SGLang behavior

Risks & Pitfalls

• NEVER use write_file on src/main.rs — accidentally wiped it before. Always use patch() for incremental changes.
• Rust 2015 linter noise (E0670 async fn not permitted) — ignore, Cargo.toml uses edition 2024.
• SGLang command-line flags differ significantly from vLLM — --tp/--dp/--pp vs --tensor-parallel-size/--pipeline-parallel-size.
• Ray mode requires head node to start first (Ray GCS), then workers join. May need sequential startup for Ray.