Run NemoClaw with a Local LLM

Multi-node inference for NemoClaw

NOTE

Multi-node currently applies to DGX Station only. Single-node setup for all supported hardware platforms is covered in the Instructions tab.

Deploy NVIDIA Nemotron 3 Ultra across two DGX Station nodes, then point NemoClaw on the head node at that shared vLLM endpoint.

Phase 1: Prepare both nodes

Step 1
Prerequisites

Complete the connectivity playbook for your DGX Station pair before starting this tab. Both high-speed fabric rails must be configured and validated.
Hugging Face Access Token for downloading the model.
Docker and NVIDIA Container Toolkit installed on both nodes.
Both nodes must be connected to each other via the validated fabric.

NOTE

The initial model download, weight loading, kernel compilation, autotuning, and CUDA graph capture can take more than an hour. Later starts are faster when the model cache and the existing containers are retained. Recreating the containers can rebuild compilation artifacts.

Step 2
Verify the fabric

On both nodes, confirm that the high-speed interfaces are active:

Bash

ibdev2netdev
ip -br link show
ip -br address show
show_gids

The examples in this guide use the following direct-attach network. Use your configured addresses if they differ.

Rail	`node-1`	`node-2`
0	`192.168.240.1/30`	`192.168.240.2/30`
1	`192.168.240.5/30`	`192.168.240.6/30`

From node-1, verify both peers with jumbo packets:

Bash

ping -c 4 -M do -s 8972 192.168.240.2
ping -c 4 -M do -s 8972 192.168.240.6

Do not continue until both pings succeed and the two-node fabric playbook's RDMA and NCCL checks pass.

Step 3
Download and copy the model cache

NOTE

This step takes more than an hour depending on the internet speed and requires substantial storage (on the order of hundreds of GB) on both nodes.

Authenticate to Hugging Face on node-1. The model is gated, so the account must have access before the download starts.

Bash

ssh node-1
hf auth login
hf auth whoami

export HF_MODEL=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4

hf download "${HF_MODEL}" --cache-dir "${HOME}/.cache/huggingface"

Copy the complete Hugging Face cache layout to node-2. Copying only the model shards is not sufficient.

Bash

ssh node-2 'mkdir -p ~/.cache/huggingface'

rsync -aH --info=progress2 "${HOME}/.cache/huggingface/" node-2:.cache/huggingface/

Confirm that the model snapshot exists on both nodes:

Bash

find "${HOME}/.cache/huggingface" -maxdepth 1 -type d \
  -name 'models--nvidia--NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4'

Phase 2: Launch the multi-node inference server

Step 4
Start the head container on node-1

Connect to node-1 and set the deployment variables. Discover fabric devices and the first fabric interface from ibdev2netdev (use the values from your validated connectivity playbook if discovery differs):

Bash

ssh node-1

mapfile -t IB_LINES < <(ibdev2netdev)
export HEAD_HCA0=$(awk '{print \$1; exit}' <<< "\${IB_LINES[0]}")
export HEAD_HCA1=$(awk '{print \$1; exit}' <<< "\${IB_LINES[1]}")
export HEAD_IFACE=$(awk '{print \$5; exit}' <<< "\${IB_LINES[0]}")
export NCCL_IB_HCA="${HEAD_HCA0},${HEAD_HCA1}"
export UCX_NET_DEVICES="${HEAD_HCA0}:1,${HEAD_HCA1}:1"
export HEAD_IP=$(ip -4 -o address show dev "${HEAD_IFACE}" \
  | awk '{split($4, address, "/"); print address[1]; exit}')
export GPU_UUID_HEAD=$(nvidia-smi \
  --query-gpu=name,uuid --format=csv,noheader \
  | awk -F', ' '/GB300|B300/ {print $2; exit}')
export IMG=vllm/vllm-openai:v0.25.1-aarch64
export HF_MODEL=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
export SERVED_MODEL=nemotron-ultra

Review the discovered values before starting the container:

Bash

printf 'HEAD_IFACE=%s\nHEAD_IP=%s\nGPU_UUID_HEAD=%s\nNCCL_IB_HCA=%s\nUCX_NET_DEVICES=%s\n' \
  "${HEAD_IFACE}" "${HEAD_IP}" "${GPU_UUID_HEAD}" "${NCCL_IB_HCA}" "${UCX_NET_DEVICES}"

Start the head container. It creates the Ray cluster and waits up to one hour for the worker GPU before launching vLLM.

Bash

sudo docker rm -f nemotron-ultra-head 2>/dev/null || true

sudo docker run -d --name nemotron-ultra-head \
  --restart unless-stopped --init \
  --network host --shm-size 16g \
  --gpus "device=${GPU_UUID_HEAD}" \
  --device=/dev/infiniband/uverbs0 \
  --device=/dev/infiniband/uverbs1 \
  --ulimit memlock=-1 \
  -e HEAD_IP="${HEAD_IP}" \
  -e HF_MODEL="${HF_MODEL}" \
  -e SERVED_MODEL="${SERVED_MODEL}" \
  -e VLLM_HOST_IP="${HEAD_IP}" \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e NCCL_IB_HCA="${NCCL_IB_HCA}" \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ROCE_VERSION_NUM=2 \
  -e NCCL_IB_TC=106 \
  -e NCCL_NET_GDR_LEVEL=PHB \
  -e NCCL_SOCKET_IFNAME="${HEAD_IFACE}" \
  -e GLOO_SOCKET_IFNAME="${HEAD_IFACE}" \
  -e TP_SOCKET_IFNAME="${HEAD_IFACE}" \
  -e NCCL_IB_QPS_PER_CONNECTION=4 \
  -e NCCL_IB_PCI_RELAXED_ORDERING=1 \
  -e UCX_NET_DEVICES="${UCX_NET_DEVICES}" \
  -e HF_HOME=/models/huggingface \
  -v "${HOME}/.cache/huggingface:/models/huggingface" \
  --entrypoint bash "${IMG}" -lc '
    set -euo pipefail
    python3 -m pip install --break-system-packages "ray==2.56.0"
    python3 -m pip install --break-system-packages --ignore-installed "blinker==1.9.0" "aiperf==0.11.0"
    ray start --head --node-ip-address="${HEAD_IP}" --port=6379 --num-gpus=1

    python3 - <<"PY"
import time
import ray

ray.init(address="auto")
deadline = time.time() + 3600
while ray.cluster_resources().get("GPU", 0) < 2:
    if time.time() >= deadline:
        raise TimeoutError("node-2 GPU did not join Ray within 3600 seconds")
    time.sleep(5)
print(ray.cluster_resources())
PY

    exec vllm serve "${HF_MODEL}" \
      --served-model-name "${SERVED_MODEL}" \
      --host 0.0.0.0 --port 8000 \
      --trust-remote-code \
      --tensor-parallel-size 1 \
      --pipeline-parallel-size 2 \
      --distributed-executor-backend ray \
      --kv-cache-dtype fp8 \
      --max-model-len 262144 \
      --gpu-memory-utilization 0.9 \
      --max-num-seqs 256 \
      --distributed-timeout-seconds 7200 \
      --enable-prefix-caching \
      --enable-auto-tool-choice \
      --tool-call-parser qwen3_coder \
      --reasoning-parser nemotron_v3
  '

Step 5
Start the worker container on node-2

Connect to node-2 and set the worker variables:

NOTE

HEAD_IP value should be fetched from node-1 Step 4

Bash

ssh node-2

mapfile -t IB_LINES < <(ibdev2netdev)
export WORKER_HCA0=$(awk '{print \$1; exit}' <<< "\${IB_LINES[0]}")
export WORKER_HCA1=$(awk '{print \$1; exit}' <<< "\${IB_LINES[1]}")
export WORKER_IFACE=$(awk '{print \$5; exit}' <<< "\${IB_LINES[0]}")
export NCCL_IB_HCA="${WORKER_HCA0},${WORKER_HCA1}"
export UCX_NET_DEVICES="${WORKER_HCA0}:1,${WORKER_HCA1}:1"
export WORKER_IP=$(ip -4 -o address show dev "${WORKER_IFACE}" \
  | awk '{split($4, address, "/"); print address[1]; exit}')
export GPU_UUID_WORKER=$(nvidia-smi \
  --query-gpu=name,uuid --format=csv,noheader \
  | awk -F', ' '/GB300|B300/ {print $2; exit}')

export HEAD_IP="<node-1 fabric IP>" # e.g. 192.168.240.1
export IMG=vllm/vllm-openai:v0.25.1-aarch64

Review the values and then start the worker:

Bash

printf 'WORKER_IFACE=%s\nWORKER_IP=%s\nGPU_UUID_WORKER=%s\nHEAD_IP=%s\nNCCL_IB_HCA=%s\n' \
  "${WORKER_IFACE}" "${WORKER_IP}" "${GPU_UUID_WORKER}" "${HEAD_IP}" "${NCCL_IB_HCA}"

sudo docker rm -f nemotron-ultra-worker 2>/dev/null || true

sudo docker run -d --name nemotron-ultra-worker \
  --restart unless-stopped --init \
  --network host --shm-size 16g \
  --gpus "device=${GPU_UUID_WORKER}" \
  --device=/dev/infiniband/uverbs0 \
  --device=/dev/infiniband/uverbs1 \
  --ulimit memlock=-1 \
  -e HEAD_IP="${HEAD_IP}" \
  -e WORKER_IP="${WORKER_IP}" \
  -e VLLM_HOST_IP="${WORKER_IP}" \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e NCCL_IB_HCA="${NCCL_IB_HCA}" \
  -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ROCE_VERSION_NUM=2 \
  -e NCCL_IB_TC=106 \
  -e NCCL_NET_GDR_LEVEL=PHB \
  -e NCCL_SOCKET_IFNAME="${WORKER_IFACE}" \
  -e GLOO_SOCKET_IFNAME="${WORKER_IFACE}" \
  -e TP_SOCKET_IFNAME="${WORKER_IFACE}" \
  -e NCCL_IB_QPS_PER_CONNECTION=4 \
  -e NCCL_IB_PCI_RELAXED_ORDERING=1 \
  -e UCX_NET_DEVICES="${UCX_NET_DEVICES}" \
  -e HF_HOME=/models/huggingface \
  -v "${HOME}/.cache/huggingface:/models/huggingface" \
  --entrypoint bash "${IMG}" -lc '
    set -euo pipefail
    python3 -m pip install --break-system-packages "ray==2.56.0"
    exec ray start --address="${HEAD_IP}:6379" --node-ip-address="${WORKER_IP}" --num-gpus=1 --block
  '

Phase 3: Verify the multi-node inference server

Step 6
Monitor startup without interrupting it

Open one terminal for each node to monitor the startup:

Bash

# node-1
sudo docker logs -f nemotron-ultra-head

Bash

# node-2
sudo docker logs -f nemotron-ultra-worker

During a first start, vLLM loads model shards, compiles kernels, builds the KV cache, autotunes FlashInfer/TRT-LLM kernels, and captures CUDA graphs. The containers can remain Up while port 8000 returns HTTP 000. This is expected while the logs continue to advance.

Do not restart merely because the API is not yet listening. Investigate when a container exits, its restart count increases, an explicit error appears, or the worker logs stop advancing for an extended period.

Step 7
Validate the API, reasoning, and tool calls

Wait for this message in the head logs:

Text

Application startup complete.

Then run all API checks on node-1.

Confirm the model alias:

Bash

curl -fsS http://127.0.0.1:8000/v1/models | jq

Send a short chat request:

Bash

curl -fsS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "nemotron-ultra",
    "messages": [
      {"role": "user", "content": "Reply with exactly: READY"}
    ],
    "max_tokens": 16,
    "temperature": 0
  }' | jq

Step 8
Perform routine health checks

Run these checks from node-1:

Bash

sudo docker inspect \
  --format 'head={{.State.Status}} restarts={{.RestartCount}}' \
  nemotron-ultra-head

ssh node-2 sudo docker inspect \
  --format 'worker={{.State.Status}} restarts={{.RestartCount}}' \
  nemotron-ultra-worker

curl -fsS http://127.0.0.1:8000/v1/models \
  | jq -e '.data[] | select(.id == "nemotron-ultra")'

All three checks must succeed before treating the agent service as healthy.

When the endpoint is healthy, install NemoClaw on node-1 (see the Instructions tab) and point it at this local vLLM endpoint with NVIDIA Nemotron 3 Ultra as the inference backend.

Phase 4: Troubleshoot or remove the deployment

Step 9
Troubleshoot common failures

Symptom	Likely cause	Corrective action
API responds, but tool calls appear as text	Tool parser flags are missing or NemoClaw is using the Responses API	Confirm `--enable-auto-tool-choice`, `--tool-call-parser qwen3_coder`, `--reasoning-parser nemotron_v3`, and Chat Completions.
Direct `curl` works, but NemoClaw inference is unhealthy	OpenShell cannot reach the host endpoint	Confirm vLLM listens on `0.0.0.0`, inspect the OpenShell subnet, review firewall rules, and rerun onboarding.
NemoClaw reports the wrong model	The provider route or served-model alias is stale	Verify `/v1/models`, then rerun onboarding or use `nemoclaw inference set` with `nemotron-ultra`.

Useful diagnostics:

Bash

# node-1
sudo docker logs --tail 200 nemotron-ultra-head
sudo docker exec nemotron-ultra-head \
  ray status --address=${HEAD_IP}:6379
ss -ltnp | grep ':8000' || true

Bash

# node-2
sudo docker logs --tail 200 nemotron-ultra-worker
nvidia-smi

Step 10
Stop or remove the deployment

Remove the inference containers in this order:

Bash

# node-1
sudo docker rm -f nemotron-ultra-head

Bash

# node-2
sudo docker rm -f nemotron-ultra-worker

Keep the Hugging Face caches unless reclaiming disk is intentional. Retaining the model cache makes a controlled relaunch substantially faster. Reusing the existing containers also preserves their compilation artifacts.

Run NemoClaw with a Local LLM

Multi-node inference for NemoClaw

Phase 1: Prepare both nodes

Step 1
Prerequisites

Step 2
Verify the fabric

Step 3
Download and copy the model cache

Phase 2: Launch the multi-node inference server

Step 4
Start the head container on node-1

Step 5
Start the worker container on node-2

Phase 3: Verify the multi-node inference server

Step 6
Monitor startup without interrupting it

Step 7
Validate the API, reasoning, and tool calls

Step 8
Perform routine health checks

Phase 4: Troubleshoot or remove the deployment

Step 9
Troubleshoot common failures

Step 10
Stop or remove the deployment

Related resources

Resources

Run NemoClaw with a Local LLM

Multi-node inference for NemoClaw

Phase 1: Prepare both nodes

Step 1Prerequisites

Step 2Verify the fabric

Step 3Download and copy the model cache

Phase 2: Launch the multi-node inference server

Step 4Start the head container on node-1

Step 5Start the worker container on node-2

Phase 3: Verify the multi-node inference server

Step 6Monitor startup without interrupting it

Step 7Validate the API, reasoning, and tool calls

Step 8Perform routine health checks

Phase 4: Troubleshoot or remove the deployment

Step 9Troubleshoot common failures

Step 10Stop or remove the deployment

Related resources

Resources

Step 1
Prerequisites

Step 2
Verify the fabric

Step 3
Download and copy the model cache

Step 4
Start the head container on node-1

Step 5
Start the worker container on node-2

Step 6
Monitor startup without interrupting it

Step 7
Validate the API, reasoning, and tool calls

Step 8
Perform routine health checks

Step 9
Troubleshoot common failures

Step 10
Stop or remove the deployment