vllm - 💡(How to fix) Fix [Bug]: Gemma 4 26b-a4b AWQ INT4 is broken in 0.19.1, but working in latest gemma4 tag on Dockerhub [1 comments, 2 participants]

Loading safetensors checkpoint shards: 0% Completed | 0/4 [00:00<?, ?it/s] (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] EngineCore failed to start. (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] Traceback (most recent call last): (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1082, in run_engine_core (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 848, in init (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] super().init( (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 114, in init (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] self.model_executor = executor_class(vllm_config) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 103, in init (EngineCore pid=272) Process EngineCore: (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] self._init_executor() (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 52, in _init_executor (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] self.driver_worker.load_model() (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 323, in load_model (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] self.model_runner.load_model(load_dummy_weights=load_dummy_weights) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 4751, in load_model (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] self.model = model_loader.load_model( (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] self.load_weights(model, model_config) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return func(*args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 381, in load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] loaded_weights = model.load_weights(self.get_all_weights(model_config, model)) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4_mm.py", line 1355, in load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return original_load_weights(self, weights, *args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 355, in load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] autoloaded_weights = set(self._load_module("", self.module, weights)) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 302, in _load_module (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] yield from self._load_module( (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 275, in _load_module (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] loaded_params = module_load_weights(weights) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py", line 1613, in load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return loader.load_weights(_weight_iterator()) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] return original_load_weights(self, weights, *args, **kwargs) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 355, in load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] autoloaded_weights = set(self._load_module("", self.module, weights)) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 302, in _load_module (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] yield from self._load_module( (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 275, in _load_module (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] loaded_params = module_load_weights(weights) (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/gemma4.py", line 1388, in load_weights (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] param = params_dict[name] (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] ~~~~~~~~~~~^^^^^^ (EngineCore pid=272) ERROR 04-18 14:25:37 [core.py:1108] KeyError: 'layers.0.moe.experts.0.down_proj_packed'

TL;DR

The issue is likely due to a version mismatch or incompatibility between the Docker image versions vllm/vllm-openai:v0.19.1 and vllm/vllm-openai:gemma4, causing a KeyError when loading model weights.

Guidance

• Verify that the model weights are compatible with the v0.19.1 version of the Docker image.
• Check the documentation for any specific requirements or compatibility issues with the gemma-4-26B-A4B-it-AWQ-4bit model.
• Consider using the same version of the Docker image that was used to train the model, in this case, vllm/vllm-openai:gemma4.
• If possible, try to update to a newer version of the Docker image that supports the gemma-4-26B-A4B-it-AWQ-4bit model.

Example

No code example is provided as the issue seems to be related to version compatibility rather than code syntax.

Notes

The error message indicates a KeyError when loading model weights, which suggests that the model weights are not compatible with the version of the Docker image being used. Without more information about the model weights and the Docker image versions, it is difficult to provide a more specific solution.

Recommendation

Apply workaround: Use the vllm/vllm-openai:gemma4 Docker image, which is known to work with the gemma-4-26B-A4B-it-AWQ-4bit model, instead of vllm/vllm-openai:v0.19.1. This should resolve the KeyError issue and allow the model to load correctly.