PR #39481: [vllm IR] Port FP8 Quantization to vLLM IR Ops

• Repository: vllm-project/vllm
• Author: BadrBasowid
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/39481

Description (problem / solution / changelog)

Purpose

Ports quant fp8 ops to IR following #38745 . The table below tracks all quant fp8 IR ops and their provider implementations. ✅ : Ported ❌: No existing implementation

Notes:

• per_token_group_quant_fp8_packed_for_deepgemm: Has different semantics from dynamic_group_quant_fp8 and requires a separate IR op.
• per_token_group_quant_fp8 : not removed because deep_gemm_moe.py calls it with a pre-allocated out_q output buffer. IR ops must be purely functional and cannot write into caller-provided output tensors, so this usage cannot be expressed as an IR op without refactoring the call site.

Follow ups

• Aiter dynamic quant pattern: Subclass RMSNormDynamicQuantPattern once it's refactored to use VllmFusionPatternMatcherPass to eliminate the duplicate.
• Aiter group quant pattern: Same as above for RMSNormGroupQuantPattern.
• deep_gemm_moe: Replace the per-token group quant call in fp8_utils.py with a direct IR op.

Test Plan

Test Result

---

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

</details>

Changed files

• tests/compile/passes/distributed/test_fusion_all_reduce.py (modified, +1/-3)
• tests/compile/passes/test_fusion.py (modified, +10/-21)
• tests/compile/passes/test_fusion_attn.py (modified, +3/-7)
• tests/compile/passes/test_mla_attn_quant_fusion.py (modified, +2/-5)
• tests/compile/passes/test_silu_mul_quant_fusion.py (modified, +14/-16)
• tests/rocm/aiter/test_grouped_quant.py (modified, +15/-15)
• vllm/_aiter_ops.py (modified, +0/-134)
• vllm/compilation/passes/fusion/act_quant_fusion.py (modified, +14/-43)
• vllm/compilation/passes/fusion/allreduce_rms_fusion.py (modified, +10/-15)
• vllm/compilation/passes/fusion/attn_quant_fusion.py (modified, +3/-5)
• vllm/compilation/passes/fusion/matcher_utils.py (modified, +11/-171)
• vllm/compilation/passes/fusion/mla_attn_quant_fusion.py (modified, +3/-5)
• vllm/compilation/passes/fusion/rms_quant_fusion.py (modified, +51/-90)
• vllm/compilation/passes/fusion/rocm_aiter_fusion.py (modified, +52/-47)
• vllm/compilation/passes/fusion/sequence_parallelism.py (modified, +8/-10)
• vllm/config/kernel.py (modified, +12/-0)
• vllm/ir/ops/__init__.py (modified, +13/-1)
• vllm/ir/ops/quant.py (added, +148/-0)
• vllm/kernels/__init__.py (modified, +2/-2)
• vllm/kernels/aiter_ops.py (modified, +193/-0)
• vllm/kernels/triton/__init__.py (added, +7/-0)
• vllm/kernels/triton/quant.py (added, +212/-0)
• vllm/kernels/triton_ops.py (added, +212/-0)
• vllm/kernels/vllm_c.py (modified, +154/-0)
• vllm/model_executor/layers/quantization/input_quant_fp8.py (modified, +29/-123)
• vllm/model_executor/layers/quantization/utils/fp8_utils.py (modified, +2/-0)
• vllm/platforms/cuda.py (modified, +8/-1)
• vllm/platforms/rocm.py (modified, +21/-5)
• vllm/platforms/xpu.py (modified, +7/-1)