vllm - 💡(How to fix) Fix [Bug]: Worker state update can raise KeyError on prev_req_id_to_index for a resumed AsyncLLM request

vllm2026-05-15 16:59:37

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

I found an async streaming_update bug in the AsyncLLM path that can crash the engine with a worker-side KeyError.

The reproducer keeps one other live request active so speculative decoding stays enabled, then starts one resumable live request (req0) and sends two legal streaming_updates to that same live request.

Under that combination, the scheduler still emits speculative decode tokens for req0, for example:

scheduled_spec_decode_tokens={req0-...: [-1, -1, -1, -1]}

But by the time the worker updates the cached request state for the next async iteration, that same req0 is no longer present in prev_req_id_to_index. On the re-verified standalone AsyncLLM run from 2026-05-16, the reproducer then crashes in gpu_model_runner.py:1104 with:

KeyError: 'req0-...'

This is not the same issue as:

the Qwen double-streaming_update row-mapping issue that was already filed
the shared-prefix -1 placeholder leak issue

In this reproduction, the visible sink is: scheduled_spec_decode_tokens still present for req0 + missing prev_req_id_to_index entry -> KeyError.

Error Message

KeyError: 'req0-...'

Root Cause

This is an async resumed-request state mismatch bug, not a malformed input bug.

The important distinction is:

the request is frontend-legal
the same live request is resumed twice through legal streaming_updates
speculative decode state for that request is still being carried forward
but the worker's previous-batch-row mapping no longer contains that request id

So the real problem is that the resumed request state and the worker's cached state from the previous batch no longer agree. The scheduler still emits speculative decode tokens for req0, but the worker no longer has a previous batch row entry for that same request and crashes when it assumes the mapping still exists.

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 384 On-line CPU(s) list: 0-383 Vendor ID: AuthenticAMD Model name: AMD EPYC 9654 96-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 2 Stepping: 1 Frequency boost: enabled CPU max MHz: 3707.8120 CPU min MHz: 1500.0000 BogoMIPS: 4792.57 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d Virtualization: AMD-V L1d cache: 6 MiB (192 instances) L1i cache: 6 MiB (192 instances) L2 cache: 192 MiB (192 instances) L3 cache: 768 MiB (24 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-95,192-287 NUMA node1 CPU(s): 96-191,288-383 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Code Example

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 13.1.0-8ubuntu1~22.04) 13.1.0
Clang version                : 16.0.6 (++20231112100510+7cbf1a259152-1~exp1~20231112100554.106)
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-6.5.0-35-generic-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.61
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090

Nvidia driver version        : 570.86.10
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.3.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             384
On-line CPU(s) list:                0-383
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9654 96-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 2
Core(s) per socket:                 96
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        3707.8120
CPU min MHz:                        1500.0000
BogoMIPS:                           4792.57
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                     AMD-V
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           192 MiB (192 instances)
L3 cache:                           768 MiB (24 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-95,192-287
NUMA node1 CPU(s):                  96-191,288-383
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.0.2
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] optree==0.15.0
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu128
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu128
[pip3] torchvision==0.25.0+cu128
[pip3] transformers==4.56.1
[pip3] triton==3.6.0
[conda] flashinfer-python         0.6.4                    pypi_0    pypi
[conda] numpy                     2.0.2                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.8.4.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.8.90                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.8.93                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.8.90                  pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.10.2.21                pypi_0    pypi
[conda] nvidia-cudnn-frontend     1.18.0                   pypi_0    pypi
[conda] nvidia-cufft-cu12         11.3.3.83                pypi_0    pypi
[conda] nvidia-cufile-cu12        1.13.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.9.90                pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.7.3.90                pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.5.8.93                pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.7.1                    pypi_0    pypi
[conda] nvidia-cutlass-dsl        4.4.2                    pypi_0    pypi
[conda] nvidia-cutlass-dsl-libs-base 4.4.2                    pypi_0    pypi
[conda] nvidia-ml-py              13.590.48                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.27.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.8.93                  pypi_0    pypi
[conda] nvidia-nvshmem-cu12       3.4.5                    pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.8.90                  pypi_0    pypi
[conda] optree                    0.15.0                   pypi_0    pypi
[conda] pyzmq                     27.1.0                   pypi_0    pypi
[conda] torch                     2.10.0+cu128             pypi_0    pypi
[conda] torch-c-dlpack-ext        0.1.5                    pypi_0    pypi
[conda] torchaudio                2.10.0+cu128             pypi_0    pypi
[conda] torchvision               0.25.0+cu128             pypi_0    pypi
[conda] transformers              4.56.1                   pypi_0    pypi
[conda] triton                    3.6.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.1
vLLM Build Flags:
  CUDA Archs: 8.9; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    96-191,288-383  1               N/A
GPU1    NODE     X      96-191,288-383  1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
TORCH_CUDA_ARCH_LIST=8.9
CUDA_PATH=/usr/local/cuda
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/neil/code/llm/llama.cpp/build-cuda/bin
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
CUDAToolkit_ROOT=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_neil

---

scheduled_spec_decode_tokens={req0-...: [-1, -1, -1, -1]}

---

KeyError: 'req0-...'

---

# vllm/v1/engine/output_processor.py
   stream_input=request.resumable

---

# vllm/v1/engine/output_processor.py
   def apply_streaming_update(self, update: StreamingUpdate) -> None:
       if self.prompt_token_ids:
           self.prompt_token_ids.extend(update.prompt_token_ids or ())
       else:
           self.prompt_token_ids = update.prompt_token_ids or []
       self.prompt_len = len(self.prompt_token_ids)

---

# vllm/v1/core/sched/scheduler.py
   kept_output_tokens = session._all_token_ids[
       session.num_prompt_tokens : num_computed_tokens
   ]
   session.prompt_token_ids.extend(kept_output_tokens)
   session._all_token_ids.extend(update.prompt_token_ids or ())
   session.prompt_token_ids.extend(update.prompt_token_ids or ())

---

# vllm/v1/core/sched/async_scheduler.py
   cur_num_spec_tokens = len(spec_decode_tokens.get(req_id, ()))
   request.num_output_placeholders += 1 + cur_num_spec_tokens
   request.spec_token_ids = self._spec_token_placeholders

---

# vllm/v1/core/sched/scheduler.py
   scheduled_spec_decode_tokens[request.request_id] = spec_token_ids

---

# vllm/v1/worker/gpu_model_runner.py
   prev_req_id_to_index: dict[str, int] = {}
   for i, req_id in enumerate(self.input_batch.req_ids):
       if i in discard_req_indices_set:
           continue
       prev_req_id_to_index[req_id] = i
   self.input_batch.prev_req_id_to_index = prev_req_id_to_index

---

# vllm/v1/worker/gpu_model_runner.py
   if req_state.prev_num_draft_len and self.use_async_scheduling:
       assert self.input_batch.prev_req_id_to_index is not None
       prev_req_index = self.input_batch.prev_req_id_to_index[req_id]
       num_accepted = valid_sampled_token_count[prev_req_index] - 1
       req_state.output_token_ids.extend([-1] * num_accepted)

---

KeyError: 'req0-...'

---

export POC_PY=/path/to/python3
export G12_ASYNC=/path/to/repro_g12_granite_req0_keyerror_asyncllm.py
export VLLM_POC_G12_MODEL=/path/to/granite

CUDA_VISIBLE_DEVICES=0 "$POC_PY" "$G12_ASYNC" \
  --model "$VLLM_POC_G12_MODEL" \
  --run-name g12_granite_req0_keyerror_asyncllm

---

CUDA_VISIBLE_DEVICES=0 CUDA_LAUNCH_BLOCKING=1 "$POC_PY" "$G12_ASYNC" \
  --model "$VLLM_POC_G12_MODEL" \
  --run-name g12_granite_req0_keyerror_blocking \
  --blocking

---

File ".../vllm/v1/worker/gpu_model_runner.py", line 1104, in _update_states
    prev_req_index = self.input_batch.prev_req_id_to_index[req_id]
KeyError: 'req0-...'

---

vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

---

scheduled_spec_decode_tokens={req0-...: [-1, -1, -1, -1]}
num_invalid_spec_tokens={req0-...: 4}

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 13.1.0-8ubuntu1~22.04) 13.1.0
Clang version                : 16.0.6 (++20231112100510+7cbf1a259152-1~exp1~20231112100554.106)
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.10.0+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] (64-bit runtime)
Python platform              : Linux-6.5.0-35-generic-x86_64-with-glibc2.35
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.61
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA GeForce RTX 4090
GPU 1: NVIDIA GeForce RTX 4090

Nvidia driver version        : 570.86.10
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.3.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      52 bits physical, 57 bits virtual
Byte Order:                         Little Endian
CPU(s):                             384
On-line CPU(s) list:                0-383
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 9654 96-Core Processor
CPU family:                         25
Model:                              17
Thread(s) per core:                 2
Core(s) per socket:                 96
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        3707.8120
CPU min MHz:                        1500.0000
BogoMIPS:                           4792.57
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                     AMD-V
L1d cache:                          6 MiB (192 instances)
L1i cache:                          6 MiB (192 instances)
L2 cache:                           192 MiB (192 instances)
L3 cache:                           768 MiB (24 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-95,192-287
NUMA node1 CPU(s):                  96-191,288-383
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.4
[pip3] numpy==2.0.2
[pip3] nvidia-cublas-cu12==12.8.4.1
[pip3] nvidia-cuda-cupti-cu12==12.8.90
[pip3] nvidia-cuda-nvrtc-cu12==12.8.93
[pip3] nvidia-cuda-runtime-cu12==12.8.90
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.3.3.83
[pip3] nvidia-cufile-cu12==1.13.1.3
[pip3] nvidia-curand-cu12==10.3.9.90
[pip3] nvidia-cusolver-cu12==11.7.3.90
[pip3] nvidia-cusparse-cu12==12.5.8.93
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.8.93
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.8.90
[pip3] optree==0.15.0
[pip3] pyzmq==27.1.0
[pip3] torch==2.10.0+cu128
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.10.0+cu128
[pip3] torchvision==0.25.0+cu128
[pip3] transformers==4.56.1
[pip3] triton==3.6.0
[conda] flashinfer-python         0.6.4                    pypi_0    pypi
[conda] numpy                     2.0.2                    pypi_0    pypi
[conda] nvidia-cublas-cu12        12.8.4.1                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.8.90                  pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.8.93                  pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.8.90                  pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.10.2.21                pypi_0    pypi
[conda] nvidia-cudnn-frontend     1.18.0                   pypi_0    pypi
[conda] nvidia-cufft-cu12         11.3.3.83                pypi_0    pypi
[conda] nvidia-cufile-cu12        1.13.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.9.90                pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.7.3.90                pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.5.8.93                pypi_0    pypi
[conda] nvidia-cusparselt-cu12    0.7.1                    pypi_0    pypi
[conda] nvidia-cutlass-dsl        4.4.2                    pypi_0    pypi
[conda] nvidia-cutlass-dsl-libs-base 4.4.2                    pypi_0    pypi
[conda] nvidia-ml-py              13.590.48                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.27.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.8.93                  pypi_0    pypi
[conda] nvidia-nvshmem-cu12       3.4.5                    pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.8.90                  pypi_0    pypi
[conda] optree                    0.15.0                   pypi_0    pypi
[conda] pyzmq                     27.1.0                   pypi_0    pypi
[conda] torch                     2.10.0+cu128             pypi_0    pypi
[conda] torch-c-dlpack-ext        0.1.5                    pypi_0    pypi
[conda] torchaudio                2.10.0+cu128             pypi_0    pypi
[conda] torchvision               0.25.0+cu128             pypi_0    pypi
[conda] transformers              4.56.1                   pypi_0    pypi
[conda] triton                    3.6.0                    pypi_0    pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.17.1
vLLM Build Flags:
  CUDA Archs: 8.9; ROCm: Disabled; XPU: Disabled
GPU Topology:
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    96-191,288-383  1               N/A
GPU1    NODE     X      96-191,288-383  1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
TORCH_CUDA_ARCH_LIST=8.9
CUDA_PATH=/usr/local/cuda
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/neil/code/llm/llama.cpp/build-cuda/bin
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
CUDAToolkit_ROOT=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_neil

</details>

🐛 Describe the bug

Describe the bug

Version: vLLM 0.17.1
Model: local GraniteMoE checkpoint used for repro
Hardware reproduced on: NVIDIA GeForce RTX 4090, single GPU

Summary

I found an async streaming_update bug in the AsyncLLM path that can crash the engine with a worker-side KeyError.

Under that combination, the scheduler still emits speculative decode tokens for req0, for example:

scheduled_spec_decode_tokens={req0-...: [-1, -1, -1, -1]}

KeyError: 'req0-...'

This is not the same issue as:

the Qwen double-streaming_update row-mapping issue that was already filed
the shared-prefix -1 placeholder leak issue

In this reproduction, the visible sink is: scheduled_spec_decode_tokens still present for req0 + missing prev_req_id_to_index entry -> KeyError.

Trigger chain

Start one other live request so speculative decoding stays active.
Submit one resumable structured-output request (req0).
Let req0 produce one output event.
Send a first legal streaming_update to that same live req0.
Let req0 produce the next output window.
Send a second legal streaming_update to the same live req0.
The scheduler later still emits speculative decode positions for req0, including scheduled_spec_decode_tokens={req0: [-1, -1, -1, -1]} and num_invalid_spec_tokens={req0: 4}.
During async worker state update, gpu_model_runner tries to look up the previous batch row for req0, but that request id is no longer present in prev_req_id_to_index, and the engine dies with KeyError.

Details

Trigger path in code

The public AsyncLLM streaming interface exposes resumable continuation through StreamingInput.
```
# vllm/v1/engine/output_processor.py
stream_input=request.resumable
```

Each legal streaming_update extends the live request prompt state.

# vllm/v1/engine/output_processor.py
def apply_streaming_update(self, update: StreamingUpdate) -> None:
    if self.prompt_token_ids:
        self.prompt_token_ids.extend(update.prompt_token_ids or ())
    else:
        self.prompt_token_ids = update.prompt_token_ids or []
    self.prompt_len = len(self.prompt_token_ids)

In the scheduler's resumable-session path, vLLM rebuilds the same session by keeping already-produced output tokens and appending the new update tokens.

# vllm/v1/core/sched/scheduler.py
kept_output_tokens = session._all_token_ids[
    session.num_prompt_tokens : num_computed_tokens
]
session.prompt_token_ids.extend(kept_output_tokens)
session._all_token_ids.extend(update.prompt_token_ids or ())
session.prompt_token_ids.extend(update.prompt_token_ids or ())

Under async scheduling, vLLM also adds output placeholders and stores placeholder speculative token ids on the live request.

# vllm/v1/core/sched/async_scheduler.py
cur_num_spec_tokens = len(spec_decode_tokens.get(req_id, ()))
request.num_output_placeholders += 1 + cur_num_spec_tokens
request.spec_token_ids = self._spec_token_placeholders

The scheduler can still keep speculative decode tokens for that resumed request in later iterations.

# vllm/v1/core/sched/scheduler.py
scheduled_spec_decode_tokens[request.request_id] = spec_token_ids

On the worker side, the previous batch row mapping is cached in prev_req_id_to_index.

# vllm/v1/worker/gpu_model_runner.py
prev_req_id_to_index: dict[str, int] = {}
for i, req_id in enumerate(self.input_batch.req_ids):
    if i in discard_req_indices_set:
        continue
    prev_req_id_to_index[req_id] = i
self.input_batch.prev_req_id_to_index = prev_req_id_to_index

Later, when async speculative state from the previous batch is updated, the worker assumes the resumed request id is still present in that mapping.

# vllm/v1/worker/gpu_model_runner.py
if req_state.prev_num_draft_len and self.use_async_scheduling:
    assert self.input_batch.prev_req_id_to_index is not None
    prev_req_index = self.input_batch.prev_req_id_to_index[req_id]
    num_accepted = valid_sampled_token_count[prev_req_index] - 1
    req_state.output_token_ids.extend([-1] * num_accepted)

In this issue, after the second streaming_update, the worker still reaches that async speculative state-update path for req0, but req0 is no longer present in prev_req_id_to_index. The re-verified sink is:
```
KeyError: 'req0-...'
```

AsyncLLM script breakdown

repro_g12_granite_req0_keyerror_asyncllm.py is a standalone local reproducer that uses only the public AsyncLLM interface.

It reads the Granite checkpoint path from VLLM_POC_G12_MODEL or the built-in /path/to/granite placeholder.
It creates one AsyncLLM with:
- async scheduling enabled
- speculative decoding enabled
- xgrammar structured output enabled
It starts:
- one live peer request (spec0)
- one streaming-input request (req0)
The live streaming request is sent through AsyncLLM.generate(AsyncGenerator[StreamingInput], ...) as:
- chunk 0
- first streaming_update
- second streaming_update
It progress-gates both updates on actual req0 output events instead of using blind sleeps.
It writes:
- repro_config.json
- engine_args.json
- request_payloads.json
- spec0_outputs.json
- req0_outputs.json
- task_results.json
- post_run_summary.json
- error.txt on failure

AsyncLLM repro

Official Hugging Face repo: granite-3.0-1b-a400m-base
Please download granite-3.0-1b-a400m-base and run my reproduce script

repro_g12_granite_req0_keyerror_asyncllm.py

export POC_PY=/path/to/python3
export G12_ASYNC=/path/to/repro_g12_granite_req0_keyerror_asyncllm.py
export VLLM_POC_G12_MODEL=/path/to/granite

CUDA_VISIBLE_DEVICES=0 "$POC_PY" "$G12_ASYNC" \
  --model "$VLLM_POC_G12_MODEL" \
  --run-name g12_granite_req0_keyerror_asyncllm

Optional blocking attribution:

CUDA_VISIBLE_DEVICES=0 CUDA_LAUNCH_BLOCKING=1 "$POC_PY" "$G12_ASYNC" \
  --model "$VLLM_POC_G12_MODEL" \
  --run-name g12_granite_req0_keyerror_blocking \
  --blocking

Reproduce Environment

Item	Value
OS	Ubuntu 22.04.5 LTS
Kernel	Linux 6.5.0-35-generic
GPU	2 x NVIDIA GeForce RTX 4090
GPU memory	24564 MiB each
Driver	570.86.10
CUDA runtime	12.8 (`nvidia-smi`)
CUDA toolkit	12.8.61 (`nvcc`)
Python	3.12.9
vLLM	0.17.1
PyTorch	2.10.0+cu128
transformers	4.56.1
tokenizers	0.22.0
flash_attn	2.8.3
triton	3.6.0
numpy	2.0.2

Observed result

Representative traceback from the re-verified non-blocking standalone AsyncLLM run:

  File ".../vllm/v1/worker/gpu_model_runner.py", line 1104, in _update_states
    prev_req_index = self.input_batch.prev_req_id_to_index[req_id]
KeyError: 'req0-...'

The same run also leaves the user-facing surface:

vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue.

And the last scheduler dump before the sink shows that the same resumed request still has speculative decode tokens scheduled:

scheduled_spec_decode_tokens={req0-...: [-1, -1, -1, -1]}
num_invalid_spec_tokens={req0-...: 4}

Root cause

This is an async resumed-request state mismatch bug, not a malformed input bug.

The important distinction is:

the request is frontend-legal
the same live request is resumed twice through legal streaming_updates
speculative decode state for that request is still being carried forward
but the worker's previous-batch-row mapping no longer contains that request id

Attachments

The attachment bundle for this report should contain:

repro_g12_granite_req0_keyerror_asyncllm.py

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #retriever error #indexing error #inference speed #environment variable

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: Worker state update can raise KeyError on prev_req_id_to_index for a resumed AsyncLLM request

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

Describe the bug

Summary

Trigger chain

Details

Trigger path in code

AsyncLLM script breakdown

AsyncLLM repro

Reproduce Environment

Observed result

Root cause

Attachments

Before submitting a new issue...

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: Worker state update can raise KeyError on prev_req_id_to_index for a resumed AsyncLLM request

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

Describe the bug

Summary

Trigger chain

Details

Trigger path in code

AsyncLLM script breakdown

AsyncLLM repro

Reproduce Environment

Observed result

Root cause

Attachments

Before submitting a new issue...

Still need to ship something?

RELATED_DISCOVERY

TRENDING