The result is "Error: 500 Internal Server Error: Post "http://127.0.0.1:40927/completion": EOF" ollama serve repoted MLX error: Failed to compile kernel: nvrtc: error: failed to open libnvrtc-builtins.so.13.0.

Fix Plan

The fix involves updating the CUDA version compatibility and ensuring the correct library paths are set.

Steps to Fix

• Update the nvrtc library to match the CUDA version (13.1.80) by installing the correct nvrtc package.
• Set the LD_LIBRARY_PATH environment variable to include the path to libnvrtc-builtins.so.13.0.

Example Code

# Install the correct nvrtc package
sudo apt-get install nvidia-cuda-toolkit

# Set the LD_LIBRARY_PATH environment variable
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-13.1/lib64

# Verify the library path
ldconfig -v | grep libnvrtc-builtins.so.13.0

Verification

Run the ollama command again to verify that the error is resolved:

ollama run x/z-image-turbo:bf16 "a cat holding a sign that says hello world"

If the issue persists, check the ollama logs for any other error messages.

Extra Tips

• Ensure that the CUDA version is compatible with the ollama version.
• Verify that the libnvrtc-builtins.so.13.0 library exists in the specified path.
• If using a Docker container, ensure that the container has the correct CUDA version and library paths set.