launcher/AGENTS.md

AGENTS.md — vllm-spark-launcher

Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over russh.

Project Layout

• Cargo.toml — binary crate vllm-spark-launcher. Edition 2024.
• src/main.rs — single-file binary: CLI parsing, SSH, docker orchestration.
• config.toml — runtime config: SSH key, cluster definitions, per-cluster NCCL settings.
• recipes/<name>/recipe.toml — recipe files.

No build.rs, no workspace, no additional modules. Everything is in main.rs.

Commands

cargo run serve --recipe glm51                             # Auto-cluster selection
cargo run serve --recipe glm51 --cluster 4nodes-switch     # Explicit cluster
cargo run -- -d serve --recipe glm51                       # Dry-run (print only)
cargo run stop --recipe glm51                              # Stop containers on all nodes
cargo run health                                           # Health check all clusters
cargo run health --cluster 2nodes-dac                      # Single cluster
cargo run logs --recipe glm51                              # Last 100 lines per node
cargo run logs --recipe glm51 --follow                     # Follow head node logs

All subprocess calls support -d / --dry-run as a global flag.

config.toml

User-specific. Has hardcoded SSH key paths — won't work without editing. Structure:

ssh_user = "username"
ssh_key_path = "/absolute/path/to/key"

[[clusters]]
name = "my-cluster"
master_ib_ip = "1.2.3.4"
nodes = [
    { name = "dgx1.lan", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0 },
    { name = "dgx2.lan", ssh_ip = "10.0.0.2", ib_ip = "1.2.3.5", rank = 1 },
]

[nccl."my-cluster"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0

recipes/

TOML format. Directory name is intentionally misspelled. Each recipe lives in its own subdirectory:

engine = "vllm"                        # or "sglang" (default: "vllm")
name = "Human Readable Name"
docker_image = "my-image:tag"
model = "org/model-name"
port = 8000
required_nodes = 4                     # 0 = any cluster size
pipeline_parallel_size = 0             # 0 = auto (equals nnodes)

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
SOME_VAR = "value"

How launch works

build_docker_command in main.rs constructs the docker run command for each node. It always appends these flags (cannot be overridden by recipe):

--tensor-parallel-size 1
--pipeline-parallel-size <computed>
--nnodes <total>
--node-rank <rank>
--master-addr <cluster.master_ib_ip>
--distributed-executor-backend mp

NCCL env vars come from the cluster's [nccl."<name>"] section in config.toml.

--cluster auto picks the first cluster in config.toml that has an NCCL section.

Dependencies

• clap 4.6 — derive-based CLI
• russh 0.60 + russh-keys 0.49 — SSH client
• tokio full — async runtime
• toml 1 + serde 1 — config/recipe deserialization
• tracing + tracing-subscriber — logging
• anyhow — error handling