ollama - 💡(How to fix) Fix gemma4:e4b only offloads 2.8 GiB to ROCm GPU despite 7.5 GiB available [3 comments, 2 participants]

TL;DR

The issue might be resolved by investigating and adjusting the memory allocation settings for the Ollama model, as the current configuration only offloads a fraction of the model weights to the GPU.

Guidance

• Review the Ollama documentation to understand how memory allocation is handled, especially for large models, to see if there are specific settings or configurations that can be adjusted to optimize GPU usage.
• Investigate the difference in behavior between the problematic model and other models (like qwen3.5:9b, qwen3-vl) that work fine on this GPU, focusing on model size, complexity, and any specific settings used for those models.
• Consider experimenting with different values for OLLAMA_NUM_GPU and other environment variables related to GPU usage, even though the provided attempts did not yield results, as there might be other combinations or settings that could influence the memory allocation.
• Look into the ROCm library settings and version (currently using version implied by HSA_OVERRIDE_GFX_VERSION=10.3.0) to ensure compatibility and optimal performance with the AMD Radeon RX 6600 XT.

Example

No specific code example can be provided without more details on the model or the exact commands used to run Ollama. However, reviewing the commands or scripts that load and run the models, especially focusing on any parameters related to GPU allocation, could provide insights.

Notes

The issue seems specific to the combination of the model being used and the Ollama version (0.21.0) on the specified hardware. The fact that other models work correctly suggests a potential issue with how this particular model interacts with Ollama's memory management.

Recommendation

Apply workaround: Given the model-specific nature of the issue and the lack of success with environment variable adjustments, the next step would involve deeper investigation into model-specific settings or optimizations that could improve GPU utilization, rather than upgrading Ollama, as there's no clear indication a newer version would resolve this specific issue.