vllm - 💡(How to fix) Fix [Feature]: Fuse QK Norm + mRoPE + KV cache write + FP8 quant

Purpose

For the QK-RMSNorm + mRoPE models (such as Qwen3-VL), the operations for Q/K RMSNorm, mRoPE, and the KV cache write with FP8 quantization currently run as separate kernels. This results in intermediate global memory round-trips and multiple kernel launches.

This PR introduces a kernel fusion that combines these operations into a single fused CUDA kernel. This new kernel will handle Q/K RMSNorm, mRoPE, KV cache writing, and per-tensor FP8 quantization, all in a single op.

As part of https://github.com/vllm-project/vllm/issues/36066

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.