Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [vllm IR]: Replace`per_token_group_quant_fp8` with `ir.ops.dynamic_group_quant_fp8` [2 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#40607•Fetched 2026-04-23 07:23:59
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Author
BadrBasowidBadrBasowid
Participants
BadrBasowidBadrBasowid
Timeline (top)
cross-referenced ×2labeled ×1

Fix Action

Fixed

PR fix notes

PR #39481: [vllm IR] Port FP8 Quantization to vLLM IR Ops

Description (problem / solution / changelog)

Purpose

Ports quant fp8 ops to IR following #38745 . The table below tracks all quant fp8 IR ops and their provider implementations. ✅ : Ported ❌: No existing implementation

Opnativeaitertritonvllm_cxpu
static_quant_fp8✅✅❌✅❌
dynamic_quant_fp8✅✅❌✅❌
dynamic_group_quant_fp8✅✅✅✅✅
static_group_quant_fp8✅❌❌✅❌

Notes:

  • per_token_group_quant_fp8_packed_for_deepgemm: Has different semantics from dynamic_group_quant_fp8 and requires a separate IR op.
  • per_token_group_quant_fp8 : not removed because deep_gemm_moe.py calls it with a pre-allocated out_q output buffer. IR ops must be purely functional and cannot write into caller-provided output tensors, so this usage cannot be expressed as an IR op without refactoring the call site.

Follow ups

  • Aiter dynamic quant pattern: Subclass RMSNormDynamicQuantPattern once it's refactored to use VllmFusionPatternMatcherPass to eliminate the duplicate.
  • Aiter group quant pattern: Same as above for RMSNormGroupQuantPattern.
  • deep_gemm_moe: Replace the per-token group quant call in fp8_utils.py with a direct IR op.

Test Plan

Test Result

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.
</details>

Changed files

  • tests/compile/passes/distributed/test_fusion_all_reduce.py (modified, +1/-3)
  • tests/compile/passes/test_fusion.py (modified, +10/-21)
  • tests/compile/passes/test_fusion_attn.py (modified, +3/-7)
  • tests/compile/passes/test_mla_attn_quant_fusion.py (modified, +2/-6)
  • tests/compile/passes/test_silu_mul_quant_fusion.py (modified, +14/-16)
  • tests/kernels/ir/test_quant.py (added, +588/-0)
  • tests/rocm/aiter/test_grouped_quant.py (modified, +15/-15)
  • vllm/_aiter_ops.py (modified, +0/-134)
  • vllm/compilation/passes/fusion/act_quant_fusion.py (modified, +14/-43)
  • vllm/compilation/passes/fusion/allreduce_rms_fusion.py (modified, +10/-15)
  • vllm/compilation/passes/fusion/attn_quant_fusion.py (modified, +3/-5)
  • vllm/compilation/passes/fusion/matcher_utils.py (modified, +11/-171)
  • vllm/compilation/passes/fusion/mla_attn_quant_fusion.py (modified, +33/-55)
  • vllm/compilation/passes/fusion/rms_quant_fusion.py (modified, +51/-90)
  • vllm/compilation/passes/fusion/rocm_aiter_fusion.py (modified, +51/-47)
  • vllm/compilation/passes/fusion/sequence_parallelism.py (modified, +8/-10)
  • vllm/config/kernel.py (modified, +12/-0)
  • vllm/ir/ops/__init__.py (modified, +13/-1)
  • vllm/ir/ops/quant.py (added, +180/-0)
  • vllm/kernels/__init__.py (modified, +2/-2)
  • vllm/kernels/aiter_ops.py (modified, +193/-0)
  • vllm/kernels/triton/__init__.py (added, +7/-0)
  • vllm/kernels/triton/quant.py (added, +210/-0)
  • vllm/kernels/vllm_c.py (modified, +128/-0)
  • vllm/kernels/xpu_ops.py (modified, +41/-0)
  • vllm/model_executor/layers/quantization/input_quant_fp8.py (modified, +31/-123)
  • vllm/model_executor/layers/quantization/utils/fp8_utils.py (modified, +2/-0)
  • vllm/platforms/cuda.py (modified, +5/-1)
  • vllm/platforms/rocm.py (modified, +20/-2)
  • vllm/platforms/xpu.py (modified, +4/-2)

PR #1: [vLLM IR] Add optional output tensor support to dynamic_group_quant_fp8

Description (problem / solution / changelog)

<!-- markdownlint-disable -->

Purpose

Resolves #40607

This PR replaces duplicate legacy quantization logic by leveraging standardized IR ops naturally.

  • Extended ir.ops.dynamic_group_quant_fp8 to support an optional out tensor (bypassing pre-allocation overhead).
  • Migrated the fallback quantization in deep_gemm_moe.py to use ir.ops.dynamic_group_quant_fp8(..., out).
  • Extracted and renamed the old duplicate utility function to _execute_per_token_group_quant_fp8 inside fp8_utils.py, allowing it to serve cleanly as the internal execution layer for the IR Op without exposing legacy endpoints.

AI Assistance Statement

  • This contribution was made with AI assistance (Cursor / Antigravity AI).
  • Duplicate checking: Performed duplicate PR searches on #40607. No existing open PR provides this exact IR Op output substitution.

Test Plan

Ran local unit tests targeting the new out buffer capabilities for dynamic_group_quant_fp8 and confirmed deep_gemm_moe fallbacks process the outputs identically safely.

pytest tests/kernels/ir/test_dynamic_group_quant_fp8.py -v
pytest tests/kernels/moe/test_block_fp8.py -v

## Changed files

- `benchmarks/fused_kernels/layernorm_rms_benchmarks.py` (modified, +11/-5)
- `benchmarks/fused_kernels/silu_mul_block_quant_benchmark.py` (modified, +4/-5)
- `benchmarks/kernels/benchmark_per_token_group_quant.py` (modified, +16/-6)
- `benchmarks/kernels/deepgemm/benchmark_fp8_block_dense_gemm.py` (modified, +9/-7)
- `benchmarks/kernels/ir/shapes.py` (modified, +14/-0)
- `cursor_instructions.md` (added, +68/-0)
- `tests/kernels/core/test_fused_quant_layernorm.py` (modified, +11/-5)
- `tests/kernels/core/test_fused_silu_mul_block_quant.py` (modified, +4/-5)
- `tests/kernels/ir/test_dynamic_group_quant_fp8.py` (added, +53/-0)
- `tests/kernels/moe/test_deepep_moe.py` (modified, +5/-4)
- `tests/kernels/moe/test_deepgemm.py` (modified, +5/-4)
- `tests/kernels/quantization/test_block_fp8.py` (modified, +18/-11)
- `tests/kernels/quantization/test_per_token_group_quant.py` (modified, +23/-9)
- `vllm/ir/ops/__init__.py` (modified, +2/-1)
- `vllm/ir/ops/dynamic_group_quant_fp8.py` (added, +65/-0)
- `vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py` (modified, +9/-7)
- `vllm/model_executor/layers/fused_moe/experts/deep_gemm_moe.py` (modified, +10/-3)
- `vllm/model_executor/layers/fused_moe/utils.py` (modified, +2/-4)
- `vllm/model_executor/layers/quantization/input_quant_fp8.py` (modified, +9/-7)
- `vllm/model_executor/layers/quantization/utils/fp8_utils.py` (modified, +5/-17)
- `vllm/model_executor/models/deepseek_v2.py` (modified, +8/-6)
RAW_BUFFERClick to expand / collapse

🚀 The feature, motivation and pitch

The per_token_group_quant_fp8 function defined in vllm/model_executor/layers/quantization/utils/fp8_utils.py duplicates logic that will be expressed via ir.ops.dynamic_group_quant_fp8 once #39481 is merged. This function should be removed in favour of the IR op.

Blocker

deep_gemm_moe.py passes a pre-allocated out_q tensor to per_token_group_quant_fp8. IR ops do not support output variants for now, making a direct substitution impossible at present.

Two paths:

  • Remove out_q: if deep_gemm_moe.py can be refactored to allocate the output internally, the call site can be migrated to the IR op without further changes.
  • Extend the IR op: if out_q cannot be eliminated, IR ops have to be extended with an optional output-tensor argument (something similar to the maybe_inplace variant that has been proposed, see #36823) before migration proceeds.

Once unblocked, per_token_group_quant_fp8 should be removed from fp8_utils.py and replaced with the IR op.

Alternatives

No response

Additional context

Keep per_token_group_quant_fp8 as-is, this leaves duplicate quantization logic in fp8_utils.py.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

Refactor deep_gemm_moe.py to allocate the output internally or extend the IR op to support an optional output-tensor argument to replace per_token_group_quant_fp8 with the IR op.

Guidance

  • Identify if deep_gemm_moe.py can be refactored to allocate the output internally, allowing for a direct substitution with the IR op.
  • If refactoring is not possible, consider extending the IR op to support an optional output-tensor argument, similar to the maybe_inplace variant proposed in #36823.
  • Once the blocker is resolved, remove per_token_group_quant_fp8 from fp8_utils.py and replace it with the IR op to eliminate duplicate quantization logic.
  • Verify the replacement by testing the functionality of deep_gemm_moe.py with the IR op.

Example

No code snippet is provided as the issue does not contain sufficient code details.

Notes

The solution depends on the feasibility of refactoring deep_gemm_moe.py or extending the IR op, which may require additional discussions or design decisions.

Recommendation

Apply a workaround by refactoring deep_gemm_moe.py to allocate the output internally, as this approach avoids modifying the IR op and allows for a more straightforward replacement of per_token_group_quant_fp8.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#memory leak#API versioning#request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING