[Bug]: DeepSeek-V4-Flash hangs after ~6 requests with cudagraph_mode=FULL_AND_PIECEWISE + chunked prefill on SM 12.x (GB10)

Your current environment

- Hardware: 2× DGX Spark (NVIDIA GB10, SM 12.1) connected via 100Gb RoCE
- vLLM: built from `4d51588` (V4-Flash main, post #40860 merge), aarch64
- Image: `vllm/vllm-openai:deepseekv4-cu130`
- Model: `deepseek-ai/DeepSeek-V4-Flash` (mxfp4 MoE)
- Backend: `--moe-backend marlin` (with #40923 SM12.x cubin patch applied locally)
- KV cache: `--kv-cache-dtype fp8_ds_mla`
- TP=2, max_model_len=32768, prefix-caching on
- PyTorch 2.10.0+cu130 (jasl's `torch_compat_v3` shim for `torch.float8_e8m0fnu`)

🐛 Describe the bug

V4-Flash with --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' (the implicit default once --enforce-eager is dropped) loads, captures graphs cleanly, and serves the first 5–6 requests correctly. On the 6th–7th request, the engine silently hangs: requests timeout with no Python exception, no NCCL timeout, no OOM. nvidia-smi shows ~100% SM utilization on both ranks but nvtop decode tok/s is zero. Recovery requires container restart.

The model produces coherent text on the requests that do succeed (so it isn't a numerical/quant issue), and the same workload runs cleanly when restricted to cudagraph_mode: PIECEWISE only — or with --enforce-eager. Workaround for our deployment is cudagraph_mode: PIECEWISE (4.99 tok/s steady on dual GB10 TP=2).

Reproduction

1. Build vLLM at 4d51588 for SM 12.x (apply #40923 for native Marlin SM12.x cubin)
2. Launch:

vllm serve /models/DeepSeek-V4-Flash \
  --tensor-parallel-size 2 \
  --distributed-executor-backend ray \
  --max-model-len 32768 \
  --moe-backend marlin \
  --kv-cache-dtype fp8_ds_mla \
  --gpu-memory-utilization 0.78 \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}'

1. Send 8+ sequential /v1/chat/completions requests, each ~50–100 tokens. Around request #6–7 the worker stops emitting tokens but the API connection stays open until client timeout.

What works (control matrix)

Suspected area

DeepseekV4Indexer / sparse_swa.py declares _cudagraph_support = AttentionCGSupport.UNIFORM_BATCH. With chunked prefill enabled, batches mix prefill (variable length) with decode (length=1), which violates uniform-query-length. Either:

• (a) the dispatcher fails to detect mixed batches on V4's sparse-MLA path and replays a wrong-shape FULL graph, OR
• (b) the metadata builder produces inconsistent state between capture and replay for V4's indexer top-k buffers.

I do not yet have py-spy stack data to confirm which (we briefly attempted py-spy capture but our diagnostic harness wasn't ready). Posting this with the empirical control matrix in case the failure mode is already familiar to V4 contributors — happy to run further diagnostics on dual GB10.

cc @zyongye @jasl @tlrmchlsmth @LucasWilkinson — V4-Flash / V32 indexer area.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.