ollama - 💡(How to fix) Fix Silent CPU fallback after driver update: CUDA forward compatibility error causes 100% CPU usage with no user warning [1 participants]

Expected behavior: ollama should either refuse to load the model with a clear error, or emit a visible WARNING log when falling back to CPU due to a CUDA initialization failure.

TL;DR

Reboot the system to ensure the updated NVIDIA driver's kernel module is loaded, potentially resolving the CUDA initialization failure causing ollama to fall back to CPU-only inference.

Guidance

• The "Driver/library version mismatch" error from nvidia-smi suggests that the kernel module from the previous driver version is still loaded, which could be causing the CUDA initialization failure.
• Rebooting the system should load the updated kernel module, potentially fixing the issue.
• Verify that the NVIDIA driver update was successful and that the NVML library version matches the driver version after rebooting.
• Check the ollama logs for any WARNING messages or errors related to CUDA initialization after rebooting to ensure the issue is resolved.

Notes

The issue seems to be related to the NVIDIA driver update and the kernel module not being updated. Rebooting the system should fix the issue, but if the problem persists, further investigation into the CUDA initialization failure may be necessary.

Recommendation

Apply workaround: Reboot the system to load the updated NVIDIA driver's kernel module, as this is likely to resolve the CUDA initialization failure and prevent ollama from falling back to CPU-only inference.