🐛 Describe the bug

For Gemma4 checkpoints whose text config has use_bidirectional_attention="vision" (e.g. gemma-4-31B-it, gemma-4-26B-A4B-it), vLLM's Gemma4 implementation silently ignores the flag and runs standard causal attention on vision tokens. HF transformers' Gemma4Model.forward calls create_causal_mask_mapping, which makes tokens inside the same vision group bidirectionally visible to each other. The vLLM forward diverges from HF on multimodal inputs as a result.

This affects any multimodal use case (generation, logprobs) and causes significant numerical divergence compared to the HF reference, concentrated on image token positions.

Reproduction

Running a teacher-forcing forward pass on gemma-4-31B-it (dense, use_bidirectional_attention="vision") and gemma-4-26B-A4B-it (MoE, same flag) with a multimodal prompt (a synthetic 224×224 image + text instruction), and comparing HF logprobs to vLLM logprobs on the generation tokens:

| model | KL(P_HF || P_vLLM) on generation tokens | |---|---| | gemma-4-E4B-it (no bidir) | 0.0004 | | gemma-4-31B-it (bidir=vision) | ~0.03 | | gemma-4-26B-A4B-it (bidir=vision) | ~0.09 |

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.