|
|
@@ -1,87 +0,0 @@
|
|
|
-# SGLang Support Implementation Plan
|
|
|
-
|
|
|
-## Background
|
|
|
-
|
|
|
-`vllm-spark-launcher` is a Rust CLI for launching vLLM/SGLang containers across DGX Spark GPU clusters via SSH. The current SGLang support is incomplete — it only changes the engine name but uses identical docker commands as vLLM. SGLang has different defaults and CLI flags that need proper handling.
|
|
|
-
|
|
|
-## Problem
|
|
|
-
|
|
|
-Current gaps:
|
|
|
-1. `VLLM_HOST_IP` is always set — SGLang uses `SGLANG_WORKER_IP` instead
|
|
|
-2. No SGLang-specific CLI flags (`--tp`, `--dp`, `--pp` vs vLLM's `--tensor-parallel-size`)
|
|
|
-3. No distributed backend selection (Ray vs MP) — currently hardcoded to MP
|
|
|
-4. No way to distinguish SGLang-specific distributed config from vLLM's
|
|
|
-5. Container naming doesn't reflect engine type (always uses recipe name)
|
|
|
-6. No health check/repair infrastructure for container issues
|
|
|
-
|
|
|
-## Implementation Plan
|
|
|
-
|
|
|
-### Phase 1: Recipe & Config Changes
|
|
|
-
|
|
|
-**r1. Add `distributed_backend` field to Recipe struct**
|
|
|
-- Add `distributed_backend: String` with default `"mp"` (matches current behavior)
|
|
|
-- Supported values: `"mp"` (default), `"ray"`
|
|
|
-- This controls docker flag composition in `build_docker_command`
|
|
|
-
|
|
|
-**r2. Add `distributed_backend` to recipe TOML schema**
|
|
|
-- Document in AGENTS.md: `distributed_backend = "mp"` (or `"ray"`)
|
|
|
-- Ray mode: `--tensor-parallel-size=<nnodes>` + `RAY_ADDRESS` env var on worker nodes
|
|
|
-- MP mode: `--tensor-parallel-size=1` + `--pipeline-parallel-size` + `--nnodes` + `--node-rank` + `--master-addr` + `--distributed-executor-backend=mp`
|
|
|
-
|
|
|
-### Phase 3: SGLang-Specific Docker Command Composition
|
|
|
-
|
|
|
-**r6. Update `build_docker_command` for Ray/MP backend switch**
|
|
|
-- When `distributed_backend == "ray"`:
|
|
|
- - Head node: normal vLLM/SGLang command with `--tensor-parallel-size=<nnodes>`
|
|
|
- - Worker nodes: `--tensor-parallel-size=<nnodes>` + `RAY_ADDRESS=<head_ip>` env var
|
|
|
- - No `--nnodes`, `--node-rank`, `--master-addr`, `--pipeline-parallel-size`, `--distributed-executor-backend` flags
|
|
|
-- When `distributed_backend == "mp"` (current behavior):
|
|
|
- - All nodes: `--tensor-parallel-size=1` + `--pipeline-parallel-size=<computed>` + `--nnodes` + `--node-rank` + `--master-addr` + `--distributed-executor-backend=mp`
|
|
|
-
|
|
|
-**Environment variable changes:**
|
|
|
-- If engine is `"vllm"`: set `VLLM_HOST_IP`
|
|
|
-- If engine is `"sglang"`: set `SGLANG_WORKER_IP` (instead of `VLLM_HOST_IP`)
|
|
|
-
|
|
|
-### Phase 4: CLI Enhancements
|
|
|
-
|
|
|
-**r4. Implement `doctor` subcommand** (checks node readiness + container state)
|
|
|
-- Concurrent node checks: `docker --version`, `nvidia-smi`, NCCL env var validation
|
|
|
-- Container checks: `docker ps --filter label=spark-launcher`, engine health endpoints, Ray cluster status
|
|
|
-- Output: pass/fail per node, actionable suggestions
|
|
|
-
|
|
|
-**r5. Implement `--repair` flag** (builds `.repaired` image on head node)
|
|
|
-- Generates Dockerfile fixing common issues (missing deps, wrong entrypoint)
|
|
|
-- Builds `<image>.repaired` on head node, tags it
|
|
|
-- `serve` auto-detects and prefers `<image>.repaired` if it exists
|
|
|
-
|
|
|
-### Phase 5: Wire Up & Testing
|
|
|
-
|
|
|
-**r3. Wire up doctor handler in `main()`**
|
|
|
-
|
|
|
-**r7. Update `serve()` to prefer repaired images**
|
|
|
-- Before build_docker_command, check if `<image>.repaired` exists
|
|
|
-- If yes, use it instead (log it)
|
|
|
-
|
|
|
-**r8. Integration testing**
|
|
|
-- `cargo build` succeeds
|
|
|
-- Dry-run with `--recipe glm51 -d` shows correct flags for both engines
|
|
|
-- Dry-run with `distributed_backend = "ray"` shows Ray-specific flags
|
|
|
-- Health check on empty cluster
|
|
|
-
|
|
|
-**r9. Update AGENTS.md**
|
|
|
-- Document new recipe fields, commands, and behavior changes
|
|
|
-
|
|
|
-## File Changes
|
|
|
-
|
|
|
-| File | Changes |
|
|
|
-|------|---------|
|
|
|
-| `src/main.rs` | Recipe struct, `distributed_backend`, env var logic, `build_docker_command` rewrite, `doctor` handler, `repair` handler |
|
|
|
-| `recipes/*/recipe.toml` | Add `distributed_backend` field (optional, defaults to `mp`) |
|
|
|
-| `AGENTS.md` | Document new fields, commands, SGLang behavior |
|
|
|
-
|
|
|
-## Risks & Pitfalls
|
|
|
-
|
|
|
-- **NEVER use `write_file` on `src/main.rs`** — accidentally wiped it before. Always use `patch()` for incremental changes.
|
|
|
-- Rust 2015 linter noise (`E0670 async fn not permitted`) — ignore, `Cargo.toml` uses edition 2024.
|
|
|
-- SGLang command-line flags differ significantly from vLLM — `--tp`/`--dp`/`--pp` vs `--tensor-parallel-size`/`--pipeline-parallel-size`.
|
|
|
-- Ray mode requires head node to start first (Ray GCS), then workers join. May need sequential startup for Ray.
|
