Your current environment

• vLLM: 0.1.dev1+gd56e95223 (custom container built from this commit of main)
• Hardware: 2× NVIDIA RTX 3090 (SM 8.6, 24 GB each), tensor parallel across both
• Model: cyankiwi/gemma-4-31B-it-AWQ-4bit
  • Quantization (from the model's config.json → quantization_config): quant_method: compressed-tensors, format: pack-quantized, num_bits: 4, group_size: 32, observer: mse, strategy: group, symmetric: true, type: int.
  • Architecture (from the model's config.json → text_config):
    • num_hidden_layers: 60
    • num_attention_heads: 32
    • num_key_value_heads: 16
    • head_dim: 256
    • global_head_dim: 512
    • sliding_window: 1024
    • layer_types: 60 entries, of which 50 are sliding_attention and 10 are full_attention (verified by counting the array)

vllm serve arguments

/models/cyankiwi/gemma-4-31B-it-AWQ-4bit
  --tensor-parallel-size 2
  --max-num-seqs 1
  --gpu-memory-utilization 0.96
  --max-model-len 131072
  --reasoning-parser gemma4
  --enable-auto-tool-choice
  --tool-call-parser gemma4
  --enable-prefix-caching
  --limit-mm-per-prompt '{"image": 0, "audio": 0}'
  --async-scheduling

kv_cache_dtype is left at its default value of auto, which resolves to BF16 in this configuration.

Observed log output (verbatim)

INFO [config.py:99]  Gemma4 model has heterogeneous head dimensions (head_dim=256, global_head_dim=512). Forcing TRITON_ATTN backend to prevent mixed-backend numerical divergence.
INFO [cuda.py:302]   Using AttentionBackendEnum.TRITON_ATTN backend.
INFO [gpu_model_runner.py:4820]  Model loading took 10.46 GiB memory and 2.480848 seconds
INFO [gpu_worker.py:436]  Available KV cache memory: 11.54 GiB
INFO [kv_cache_utils.py:1319] GPU KV cache size: 25,200 tokens
INFO [kv_cache_utils.py:1324] Maximum concurrency for 131,072 tokens per request: 1.87x

After loading the INT4 weights, each TP rank reports 11.54 GiB available for KV cache, and the resulting total GPU KV cache size is 25,200 tokens at max_model_len=131072, gpu_memory_utilization=0.96, and BF16 KV.

The model declares sliding_window: 1024 and 50 of its 60 layers are marked sliding_attention in layer_types.

Questions for maintainers

1. Is vLLM's Gemma 4 implementation expected to exploit the sliding_attention layer type and the 1024-token sliding_window when sizing the KV cache? If yes, is the 25,200-token result consistent with that exploitation on this hardware/config, or would a larger value be expected?

2. Is there a config flag I should be setting to enable sliding-window KV sharing for Gemma 4 that I am missing? (For reference, llama.cpp on the same model exposes --swa-full to toggle related behavior, and its default sizing for this architecture produces a substantially larger effective context on comparable memory.)

3. Separately, when I tried to set --kv-cache-dtype fp8_e5m2 on this INT4 checkpoint, the server raised ValueError("fp8_e5m2 kv-cache is not supported with fp8 checkpoints.") from vllm/model_executor/layers/attention/attention.py around lines 162–170. The relevant code on the running container is:

   quant_method = (
       quant_config.get_quant_method(layer, prefix=prefix) if quant_config else None
   )
   
   # See [Note: Register q/k/v/prob scales in state dict]
   if should_load_quant_weights(quant_method):
       assert isinstance(quant_method, BaseKVCacheMethod)
       # TODO (mgoin): kv cache dtype should be specified in the FP8
       # checkpoint config and become the "auto" behavior
       if layer.kv_cache_dtype == "fp8_e5m2":
           raise ValueError("fp8_e5m2 kv-cache is not supported with fp8 checkpoints.")

   with should_load_quant_weights defined as:

   def should_load_quant_weights(quant_method: QuantizeMethodBase | None) -> bool:
       """Returns whether the quantization method should load quantized weights."""
       return quant_method is not None and not isinstance(
           quant_method, UnquantizedLinearMethod
       )

   The gate that fires before the fp8_e5m2 error is should_load_quant_weights(quant_method), which returns True for any quant_method other than UnquantizedLinearMethod. For this compressed-tensors INT4 checkpoint that gate is True, even though the checkpoint is not fp8. The error message references "fp8 checkpoints" but the gate does not specifically check for an fp8 checkpoint. Is this intended for all quantized checkpoints, or is the error message describing a narrower condition than the actual check?

4. For completeness: --kv-cache-dtype fp8_e4m3 on the same configuration fails during Inductor codegen with

   triton.compiler.errors.CompilationError: at 1:0:
   def triton_red_fused__to_copy_add_cat_clamp_index_select_mul_reciprocal_rms_norm_split_split_with_sizes_sub_unsqueeze_view_1(...):
   ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")

   This appears to be a Triton / SM 8.6 hardware limitation for Inductor-fused kernels and not a vLLM-side check; I am not reporting it as a vLLM bug, but noting it because it means fp8_e5m2 is the only fp8 variant that could potentially be used on this hardware.

What this report is asking for

Guidance on whether the observed 25,200-token KV cache size for Gemma 4 31B at this configuration is expected on vLLM's current main, and clarification on the fp8_e5m2 gate above (intended scope, and/or message wording).

I am not asserting that vLLM is incorrect — I am reporting the exact observations and asking whether they match the current design.