# AGENTS.md — vllm-spark-launcher

Rust CLI tool for launching vLLM/SGLang containers across a DGX Spark cluster. SSH-based provisioning over `russh`.

## Project Layout

- `Cargo.toml` — binary crate `vllm-spark-launcher`. Edition 2024.
- `src/main.rs` — single-file binary: CLI parsing, SSH, docker orchestration.
- `config.toml` — runtime config: SSH key, cluster definitions, per-cluster NCCL settings.
- `recipies/<name>/recipe.toml` — recipe files (note: directory is misspelled "recipies").

No `build.rs`, no workspace, no additional modules. Everything is in `main.rs`.

## Commands

```
cargo run serve --recipe glm51                             # Auto-cluster selection
cargo run serve --recipe glm51 --cluster 4nodes-switch     # Explicit cluster
cargo run -- -d serve --recipe glm51                       # Dry-run (print only)
cargo run stop --recipe glm51                              # Stop containers on all nodes
cargo run health                                           # Health check all clusters
cargo run health --cluster 2nodes-dac                      # Single cluster
cargo run logs --recipe glm51                              # Last 100 lines per node
cargo run logs --recipe glm51 --follow                     # Follow head node logs
```

All subprocess calls support `-d` / `--dry-run` as a global flag.

## config.toml

User-specific. Has hardcoded SSH key paths — won't work without editing. Structure:

```toml
ssh_user = "username"
ssh_key_path = "/absolute/path/to/key"

[[clusters]]
name = "my-cluster"
master_ib_ip = "1.2.3.4"
nodes = [
    { name = "dgx1.lan", ssh_ip = "10.0.0.1", ib_ip = "1.2.3.4", rank = 0 },
    { name = "dgx2.lan", ssh_ip = "10.0.0.2", ib_ip = "1.2.3.5", rank = 1 },
]

[nccl."my-cluster"]
ib_hca = "rocep1s0f0,roceP2p1s0f0"
socket_ifname = "enp1s0f0np0"
ib_gid_index = 3
ib_disable = 0
```

## recipies/

TOML format. Directory name is intentionally misspelled. Each recipe lives in its own subdirectory:

```toml
engine = "vllm"                        # or "sglang" (default: "vllm")
name = "Human Readable Name"
docker_image = "my-image:tag"
model = "org/model-name"
port = 8000
required_nodes = 4                     # 0 = any cluster size
pipeline_parallel_size = 0             # 0 = auto (equals nnodes)

args = [
    "--kv-cache-dtype", "fp8",
    "--max-model-len", "32768",
]

[env]
SOME_VAR = "value"
```

## How launch works

`build_docker_command` in `main.rs` constructs the docker run command for each node. It always appends these flags (cannot be overridden by recipe):

```
--tensor-parallel-size 1
--pipeline-parallel-size <computed>
--nnodes <total>
--node-rank <rank>
--master-addr <cluster.master_ib_ip>
--distributed-executor-backend mp
```

NCCL env vars come from the cluster's `[nccl."<name>"]` section in `config.toml`.

`--cluster auto` picks the first cluster in `config.toml` that has an NCCL section.

## Dependencies

- `clap` 4.6 — derive-based CLI
- `russh` 0.60 + `russh-keys` 0.49 — SSH client
- `tokio` full — async runtime
- `toml` 1 + `serde` 1 — config/recipe deserialization
- `tracing` + `tracing-subscriber` — logging
- `anyhow` — error handling