vllm - 💡(How to fix) Fix [Bug]: Gemma4MultimodalEmbedder normalization order different from Transformers, causing bad audio inference [1 comments, 2 participants]

TL;DR

The most likely fix is to update the normalization order in Gemma4MultimodalEmbedder to match the implementation in Transformers.

Guidance

• Verify the normalization order in the Transformers library to ensure it is RMSNorm(multimodal_dim) → Linear.
• Update the Gemma4MultimodalEmbedder in gemma4_mm.py to use the correct normalization order.
• Test the updated embedder with short audio clips to verify that ASR accuracy has improved and hallucinations are reduced.
• Review the documentation for any other potential discrepancies between the VLLM and Transformers implementations.

Example

No code snippet is provided as the issue does not contain sufficient information to generate a accurate example.

Notes

The fix assumes that the normalization order is the only discrepancy between the VLLM and Transformers implementations. Other differences may exist and should be investigated.

Recommendation

Apply workaround: Update the Gemma4MultimodalEmbedder to match the Transformers implementation, as this is likely to resolve the regression issue with ASR accuracy.