Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [Bug]: enable_qk_norm_rope_fusion error on DeepSeek-V3.2-NVFP4 [1 pull requests, 2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#40587•Fetched 2026-04-23 07:24:11
View on GitHub
Comments
2
Participants
2
Timeline
10
Reactions
0
Author
carlyoucarlyou
Participants
carlyoucarlyouEricccYangEricccYang
Timeline (top)
commented ×2mentioned ×2renamed ×2subscribed ×2

Error Message

... [test 1] (EngineCore pid=2328) File "/tmp/vllm-bench/repos/carlyou/vllm/repro--sparse-mla-rotary-bug/.venv/lib/python3.12/site-packages/torch/_inductor/pattern_matcher.py", line 316, in replace_by_example [test 1] (EngineCore pid=2328) assert len(graph_with_eager_vals.graph.nodes) == len( [test 1] (EngineCore pid=2328) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [test 1] (EngineCore pid=2328) torch._inductor.exc.InductorError: AssertionError: [test 1] (EngineCore pid=2328) [test 1] (EngineCore pid=2328) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo" [test 1] (EngineCore pid=2328)

Root Cause

Root cause is in upstream torch (over-strict assertion), but vLLM's flashinfer_rotary_embedding custom op is the trigger because it takes query and key as two independent mutated Tensor args that happen to share a symbolic leading dim.

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 192 On-line CPU(s) list: 0-191 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8559C CPU family: 6 Model: 207 Thread(s) per core: 2 Core(s) per socket: 48 Socket(s): 2 Stepping: 2 BogoMIPS: 4800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Hypervisor vendor: KVM Virtualization type: full L1d cache: 4.5 MiB (96 instances) L1i cache: 3 MiB (96 instances) L2 cache: 192 MiB (96 instances) L3 cache: 640 MiB (2 instances) NUMA node(s): 2 NUMA node0 CPU(s): 0-47,96-143 NUMA node1 CPU(s): 48-95,144-191 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected

PR fix notes

PR #38877: [compile] mla + group fp8 fusion

Description (problem / solution / changelog)

Purpose

Completes phase 1 (group fp8) of #35792

  • Adds MLAAttnFp8GroupQuantPattern for DeepSeekV3 MLA + group FP8 quant fusion
    • pattern match covers group fp8 flags of col_major, e8m0 and TMA alignment
    • Leverage existing output_scale and output_block_scale to derive group FP8 quant key
  • Fix pattern match and test for nvfp4
    • fixed pattern match on previous nvfp4
    • use DeepSeek-R1-0528-NVFP4-v2 in e2e test, previous DeepSeek-V3.2-NVFP4 uses F8_e4m3 for kv cache which caused test failure (after fixing pattern^)

Test Plan

  • Unit tests pass for group fp8
  • E2E test pass for tp1 and tp2
  • gsm8k eval DeepSeekV3-FP8
  • bench DeepSeekV3-FP8 with and without fusion

Test Result

H100 FP8 per-group

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /tmp/vllm-bench/repos/carlyou/vllm/test--mla-quant-fusion-group-fp8/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/vllm-bench/repos/carlyou/vllm/test--mla-quant-fusion-group-fp8
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 360 items / 208 deselected / 152 selected

tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,-rms_norm-6-TRITON_MLA-RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8-<lambda>-model_kwargs3-<l
ambda>-False] PASSED [ 34%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,-rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED 
[ 34%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,+rms_norm-6-TRITON_MLA-RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8-<lambda>-model_kwargs3-<l
ambda>-False] PASSED [ 46%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,+rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED 
[ 46%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,-rms_norm-6-TRITON_MLA-RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8-<lambda>-model_kwargs3-<lam
bda>-False] PASSED [ 81%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,-rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 
82%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,+rms_norm-6-TRITON_MLA-RedHatAI/DeepSeek-Coder-V2-Lite-Instruct-FP8-<lambda>-model_kwargs3-<lam
bda>-False] PASSED [ 93%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,+rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 
94%]

B100 FP8 per-group

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /tmp/vllm-bench/repos/carlyou/vllm/test--mla-quant-fusion-group-fp8/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/vllm-bench/repos/carlyou/vllm/test--mla-quant-fusion-group-fp8
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 372 items / 276 deselected / 96 selected

tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,-rms_norm-6-FLASHINFER_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 34%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,-rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 36%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,+rms_norm-6-FLASHINFER_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 46%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[inductor_partition-+quant_fp8,+rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 48%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,-rms_norm-6-FLASHINFER_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 84%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,-rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 86%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,+rms_norm-6-FLASHINFER_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 96%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp8_fusions[dynamo_partition-+quant_fp8,+rms_norm-6-TRITON_MLA-deepseek-ai/DeepSeek-V3-<lambda>-model_kwargs4-<lambda>-False] PASSED [ 98%]

B100 DeepSeekR1-NVFP4

============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /tmp/vllm-bench/repos/carlyou/vllm/test--mla-quant-fusion-group-fp8/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/vllm-bench/repos/carlyou/vllm/test--mla-quant-fusion-group-fp8
configfile: pyproject.toml
plugins: anyio-4.13.0
collecting ... collected 360 items / 352 deselected / 8 selected

tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[inductor_partition--rms_norm-6-FLASHINFER-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] SKIPPED [ 12%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[inductor_partition--rms_norm-6-FLASHINFER_MLA-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] PASSED [ 25%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[inductor_partition-+rms_norm-6-FLASHINFER-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] SKIPPED [ 37%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[inductor_partition-+rms_norm-6-FLASHINFER_MLA-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] PASSED [ 50%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[dynamo_partition--rms_norm-6-FLASHINFER-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] SKIPPED [ 62%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[dynamo_partition--rms_norm-6-FLASHINFER_MLA-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] PASSED [ 75%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[dynamo_partition-+rms_norm-6-FLASHINFER-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] SKIPPED [ 87%]
tests/compile/fusions_e2e/test_tp1_quant.py::test_tp1_fp4_fusions[dynamo_partition-+rms_norm-6-FLASHINFER_MLA-nvidia/DeepSeek-R1-0528-NVFP4-v2-<lambda>-model_kwargs2-<lambda>] PASSED [100%]
<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.
</details>

Changed files

  • docs/design/fusions.md (modified, +2/-2)
  • tests/compile/fusions_e2e/conftest.py (modified, +16/-0)
  • tests/compile/fusions_e2e/models.py (modified, +19/-5)
  • tests/compile/fusions_e2e/test_tp1_quant.py (modified, +3/-2)
  • tests/compile/fusions_e2e/test_tp2_ar_rms.py (modified, +3/-2)
  • tests/compile/passes/test_mla_attn_quant_fusion.py (modified, +99/-18)
  • vllm/compilation/passes/fusion/mla_attn_quant_fusion.py (modified, +259/-18)
  • vllm/model_executor/layers/attention/mla_attention.py (modified, +96/-10)
  • vllm/v1/attention/backend.py (modified, +14/-2)

Code Example

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.28.3
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar  3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.8.0-1046-aws-x86_64-with-glibc2.39
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA B200
Nvidia driver version        : 580.126.09
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.14.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) Platinum 8559C
CPU family:                              6
Model:                                   207
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
Stepping:                                2
BogoMIPS:                                4800.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               4.5 MiB (96 instances)
L1i cache:                               3 MiB (96 instances)
L2 cache:                                192 MiB (96 instances)
L3 cache:                                640 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.5.4
[pip3] triton==3.6.0
[conda] pyzmq                      27.1.0           pypi_0                pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.1.dev15995+g307116eae (git sha: 307116eae)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-47,96-143	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576
NCCL_VERSION=2.28.3-1
NVIDIA_DRIVER_CAPABILITIES=all
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=13.0.2
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/x86_64-linux-gnu
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

---

vllm serve nvidia/DeepSeek-V3.2-NVFP4 --compilation-config '{"pass_config":{"enable_qk_norm_rope_fusion": true}}'

---

...
[test 1] (EngineCore pid=2328)   File "/tmp/vllm-bench/repos/carlyou/vllm/repro--sparse-mla-rotary-bug/.venv/lib/python3.12/site-packages/torch/_inductor/pattern_matcher.py", line 316, in replace_by_example
[test 1] (EngineCore pid=2328)     assert len(graph_with_eager_vals.graph.nodes) == len(
[test 1] (EngineCore pid=2328)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[test 1] (EngineCore pid=2328) torch._inductor.exc.InductorError: AssertionError:
[test 1] (EngineCore pid=2328)
[test 1] (EngineCore pid=2328) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
[test 1] (EngineCore pid=2328)
RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary> 
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 24.04.4 LTS (x86_64)
GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
Clang version                : Could not collect
CMake version                : version 3.28.3
Libc version                 : glibc-2.39

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.3 (main, Mar  3 2026, 12:15:18) [GCC 13.3.0] (64-bit runtime)
Python platform              : Linux-6.8.0-1046-aws-x86_64-with-glibc2.39
    
==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : GPU 0: NVIDIA B200
Nvidia driver version        : 580.126.09
cuDNN version                : Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.14.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.14.0
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) Platinum 8559C
CPU family:                              6
Model:                                   207
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
Stepping:                                2
BogoMIPS:                                4800.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               4.5 MiB (96 instances)
L1i cache:                               3 MiB (96 instances)
L2 cache:                                192 MiB (96 instances)
L3 cache:                                640 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] pyzmq==27.1.0
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu130
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.5.4
[pip3] triton==3.6.0
[conda] pyzmq                      27.1.0           pypi_0                pypi

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.1.dev15995+g307116eae (git sha: 307116eae)
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-47,96-143	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=void
NVIDIA_REQUIRE_CUDA=cuda>=13.0 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571 brand=unknown,driver>=575,driver<576 brand=grid,driver>=575,driver<576 brand=tesla,driver>=575,driver<576 brand=nvidia,driver>=575,driver<576 brand=quadro,driver>=575,driver<576 brand=quadrortx,driver>=575,driver<576 brand=nvidiartx,driver>=575,driver<576 brand=vapps,driver>=575,driver<576 brand=vpc,driver>=575,driver<576 brand=vcs,driver>=575,driver<576 brand=vws,driver>=575,driver<576 brand=cloudgaming,driver>=575,driver<576
NCCL_VERSION=2.28.3-1
NVIDIA_DRIVER_CAPABILITIES=all
NVIDIA_PRODUCT_NAME=CUDA
CUDA_VERSION=13.0.2
LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NVIDIA_CTK_LIBCUDA_DIR=/usr/lib/x86_64-linux-gnu
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
</details>

🐛 Describe the bug

TL;DR

"Inductor replace_by_example node-count mismatch on multi-arg mutating custom ops with shared symbolic dims (triggered by vLLM sparse-MLA rotary fusion)"

Details

torch._inductor.exc.InductorError: AssertionError File ".../torch/_inductor/pattern_matcher.py", line 316, in replace_by_example assert len(graph_with_eager_vals.graph.nodes) == len(replacement.graph.nodes)

Root cause is in upstream torch (over-strict assertion), but vLLM's flashinfer_rotary_embedding custom op is the trigger because it takes query and key as two independent mutated Tensor args that happen to share a symbolic leading dim.

Code pointers:

Reproduce

  • Enable enable_qk_norm_rope_fusion for DSv3.2 in e2e tp1 test (or use branch in vllm-project/vllm/pull/40586), and then Run test_tp1_quant.py on B200
  • Alternatively, run model serve with cc config should also surface the error:
vllm serve nvidia/DeepSeek-V3.2-NVFP4 --compilation-config '{"pass_config":{"enable_qk_norm_rope_fusion": true}}'

Example error:

...
[test 1] (EngineCore pid=2328)   File "/tmp/vllm-bench/repos/carlyou/vllm/repro--sparse-mla-rotary-bug/.venv/lib/python3.12/site-packages/torch/_inductor/pattern_matcher.py", line 316, in replace_by_example
[test 1] (EngineCore pid=2328)     assert len(graph_with_eager_vals.graph.nodes) == len(
[test 1] (EngineCore pid=2328)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[test 1] (EngineCore pid=2328) torch._inductor.exc.InductorError: AssertionError:
[test 1] (EngineCore pid=2328)
[test 1] (EngineCore pid=2328) Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
[test 1] (EngineCore pid=2328)

Proposed fixes

(from claude code, to be double checked)

  1. vLLM (production guard, near-term): in compilation.py post_init, when enable_qk_norm_rope_fusion=True is requested for a sparse MLA model, log-warn and force it off — same logic the test conftest applies. Avoids silent crash for users who flip the fusion flag on.
  2. vLLM (proper fix, mid-term): rewrite flashinfer_rotary_embedding to take a bundled tensor (concat q/k along head dim, split after the in-place call), eliminating the multi-arg-mutated case that triggers the upstream bug. Removes the need for any special casing.
  3. Upstream torch (root cause, long-term): relax the replace_by_example assertion. Track separately as a pytorch issue; reference: torch/_inductor/pattern_matcher.py:316.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The issue can be temporarily mitigated by disabling the enable_qk_norm_rope_fusion flag for DSv3.2 in the e2e tp1 test or model serve configuration.

Guidance

  1. Disable fusion flag: Set enable_qk_norm_rope_fusion to False in the compilation configuration to avoid the crash.
  2. Rewrite custom op: Consider rewriting the flashinfer_rotary_embedding custom op to take a bundled tensor, eliminating the multi-arg-mutated case that triggers the upstream bug.
  3. Track upstream issue: Report the root cause to the PyTorch team and track the issue separately, referencing torch/_inductor/pattern_matcher.py:316.

Example

No code snippet is provided as the issue is related to a specific custom op and PyTorch internals.

Notes

The proposed fixes are based on the information provided in the issue and may require further verification and testing.

Recommendation

Apply the near-term fix by disabling the enable_qk_norm_rope_fusion flag to avoid the crash, and track the upstream issue for a long-term solution.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#model save/load#optimization#mixed precision#environment variable

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

vllm - ✅(Solved) Fix [Bug]: enable_qk_norm_rope_fusion error on DeepSeek-V3.2-NVFP4 [1 pull requests, 2 comments, 2 participants]