ollama - 💡(How to fix) Fix Qwen3.5 GUFF [1 participants]

When trying to download the model hf.co/unsloth/Qwen3.5-4B-GGUF:Q8_0, I receive the error message: Error: 500 Internal Server Error: unable to load model. This happens with all Qwen3.5 models, even though Hugging Face indicates that these models are supported by Ollama.

Fix Plan

The fix involves updating the Ollama version and modifying the model download script.

• Update Ollama to the latest version using pip:

pip install --upgrade ollama

• Modify the model download script to handle the Qwen3.5 models:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "hf.co/unsloth/Qwen3.5-4B-GGUF:Q8_0"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Test the model
input_ids = tokenizer("Hello, world!", return_tensors="pt").input_ids
output = model.generate(input_ids)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Verification

Verify that the model downloads and runs successfully by checking the output of the test script.

Extra Tips

• Ensure that the GPU has sufficient memory to run the model.
• If issues persist, try downloading the model using the Hugging Face API directly:

from huggingface_hub import Repository

repo = Repository(local_dir="./", repo_id=model_name)
repo.git_pull()