vllm - 💡(How to fix) Fix [Bug]: vLLM attempts to download Hugging Face cache file during inference despite local model path (Gemma 4) [1 participants]

TL;DR

Set the HF_HUB_OFFLINE=1 environment variable and ensure the local model path is correctly configured to prevent vLLM from attempting to download files from Hugging Face.

Guidance

• Verify that the HF_HUB_OFFLINE=1 environment variable is set before running vllm serve to prevent online requests.
• Check the local model path /mnt/models/google/gemma-4-26b-it for completeness and correctness, ensuring all required files are present.
• Confirm that the model version used for downloading (google/gemma-4-31B-it) matches the version used for serving (google/gemma-4-26b-it), as version discrepancies might cause issues.
• Review the vLLM documentation and configuration options to ensure that offline mode is properly supported and configured for the specific model being used.

Example

No specific code snippet is provided due to the nature of the issue, but setting the environment variable can be done using:

export HF_HUB_OFFLINE=1

before running the vllm serve command.

Notes

The issue seems to be related to the specific model version and the configuration of vLLM in an air-gapped environment. Ensuring that all dependencies and configurations are correctly set up for offline use is crucial.

Recommendation

Apply workaround: Set HF_HUB_OFFLINE=1 and verify the local model path to prevent online downloads, as this directly addresses the reported behavior of vLLM attempting to download files from Hugging Face despite being in an air-gapped environment.