What happened?

On an 8× NVIDIA H200 (Hopper, sm_90) machine, every *-cuda-gpu model variant produces deterministically corrupted output (token doubling / gibberish), while the CPU variants of the same models are perfectly clean. The corruption is stable across runs at temperature=0, so it is not a sampling issue — it points to a CUDA compute/kernel problem in the bundled ONNX Runtime.

Clean vs. corrupted (identical prompt, temperature=0, max_tokens=25)

Prompt: Count from 1 to 5

The GPU result is bit-for-bit identical across 3 consecutive runs ('Sure (lyyes))' each time), confirming it is deterministic, not random sampling.

Other prompts show the classic token-doubling signature:

Prompt: Count: 1 2 3 4 5 → GPU returns:

The:\n count is is 11 112  11131 111144  11

(note is is, 11 112 11131 111144 — tokens are being duplicated/interleaved)

Same corruption reproduced on Phi-4-mini-instruct-cuda-gpu as well (pure gibberish). It affects all GPU variants tested.

Environment

• Foundry Local: 0.10.0+174be11ea7aeacd8d0d67b0ba1daebec615284b1
• Bundled ONNX Runtime: 1.26.0 (from /usr/local/lib/foundry-cli/libonnxruntime.so), ONNX Runtime GenAI (Foundry build)
• GPU: 8× NVIDIA H200 (Hopper, compute capability 9.0)
• NVIDIA driver: 580.105.08
• CUDA EP libs present: libonnxruntime_providers_cuda.so, libonnxruntime-genai-cuda.so
• OS: Linux

How to reproduce

1. On an H200 (or other Hopper sm_90) host, foundry model load qwen2.5-1.5b-instruct-cuda-gpu.
2. Send a completion at temperature=0:

   curl -s http://127.0.0.1:<port>/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"qwen2.5-1.5b-instruct-cuda-gpu",
          "messages":[{"role":"user","content":"Count from 1 to 5"}],
          "max_tokens":25,"temperature":0}'

3. Observe garbled output; repeat and observe it is identical each time.
4. Load the CPU variant of any model and run the same prompt → clean output.

nvidia-smi confirms the model is resident in GPU memory and inference is fast (~2–3 s), so the CUDA EP is genuinely active — it is computing, just incorrectly.

Analysis / suspected cause

The GenAI/CUDA execution path in the bundled ONNX Runtime 1.26.0 (preview) appears to miscompute on Hopper (sm_90). The corruption is:

• deterministic (independent of sampling),
• present on every GPU model variant,
• absent on all CPU variants,
• independent of cuDNN/cuBLAS/cudart minor versions (tested several CUDA 12.x runtime + cuDNN 9.x combinations on the linker path — corruption is unchanged).

This strongly suggests an ONNX Runtime / ONNX Runtime GenAI CUDA kernel issue on Hopper rather than a Foundry Local SDK/daemon-level bug.

Questions for maintainers

1. Is this a known issue with the ORT 1.26.0-preview CUDA EP on Hopper (sm_90)?
2. Would bumping the bundled ONNX Runtime / ONNX Runtime GenAI to a newer (or different stable) build fix this? Since flcore links the bundled ORT .sos directly, users can't safely swap them without an ABI-matched build — a Foundry Local release with an updated ORT would be the clean fix.
3. Should this be tracked/forwarded to microsoft/onnxruntime-genai if it is confirmed to be a kernel-level bug?

Happy to run additional diagnostics on the H200 host.