vllm - 💡(How to fix) Fix [Bug]: Gemma-4 fails to start on GPUs with < 70GB memory due to max_num_batched_tokens < multimodal token size

Your current environment

<details>

Environment:

• vLLM: 0.20.0
• Model: google/gemma-4-31b-it

</details>

🐛 Describe the bug

Gemma-4 (google/gemma-4-31b-it) fails to start on any GPU instance with less than 70GB memory per device when using the default max_num_batched_tokens configuration.

The problem:

Gemma-4 is a bidirectional multimodal model. Because of its bidirectional attention for vision tokens, vLLM disables chunked prefill for multimodal inputs. The model's transformers config defines max_multimodal_token_per_item = 2496, meaning a single multimodal input (e.g., one image) requires at least 2496 tokens to be processed as a single batch.

However, vLLM auto-calculates max_num_batched_tokens = 2048 for any GPU with less than 70GB memory. Since 2496 > 2048, vLLM cannot fit even a single multimodal item into the batch budget. This causes the engine to either reject requests or fail during initialization depending on the version.

This effectively blocks Gemma-4 deployment on all cost-effective GPU instances (A100 40GB, L4 24GB, A10G 24GB) without manually overriding max_num_batched_tokens.

Instances confirmed working (auto-calculated value exceeds 2496):

• p5en.48xlarge (H200, 141GB per GPU)
• g7e.2xlarge (RTX PRO 6000, 96GB per GPU)

Instances confirmed failing (auto-calculated value = 2048):

• p4d.24xlarge (A100, 40GB per GPU)
• g6.12xlarge (L4, 24GB per GPU)
• g5.12xlarge (A10G, 24GB per GPU)

Workaround:

Explicitly set --max-num-batched-tokens to any value greater than 2496:

vllm serve google/gemma-4-31b-it --max-num-batched-tokens 4096

Suggested fix:

vLLM should detect that the model's max_multimodal_token_per_item exceeds the auto-calculated max_num_batched_tokens and either automatically raise it to accommodate at least one multimodal item, or emit a clear error message explaining the conflict and the workaround.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.