ollama - 💡(How to fix) Fix Metal backend crash on Apple M5 - bfloat/half type mismatch in MPPTensorOpsMatMul2d (v0.21.2) [1 participants]

From ~/.ollama/logs/server.log:

ggml_metal_init: the device does not have a precompiled Metal library - this is unexpected ggml_metal_init: will try to compile it on the fly ggml_metal_library_init: error: Error Domain=MTLLibraryErrorDomain Code=3 static_assert failed due to requirement '__tensor_ops_detail::__is_same_v<bfloat, half>' "Input types must match cooperative tensor types" ggml_metal_init: error: failed to initialize the Metal library ggml_backend_metal_device_init: error: failed to allocate context llama_init_from_model: failed to initialize the context: failed to initialize Metal backend panic: unable to create llama context

The server reports the GPU as MTLGPUFamilyApple10 (1010) which is M5-specific. The embedded Metal library does not have a precompiled library for this GPU family, and on-the-fly compilation fails due to a bfloat16/half type mismatch in the MetalPerformancePrimitives framework.

TL;DR

Setting the environment variable GGML_METAL_BF16_DISABLE=1 before starting the Ollama server allows models to load and run, albeit with CPU fallback and no GPU acceleration.

Guidance

• The error is caused by a bfloat16/half type mismatch in the MetalPerformancePrimitives framework, which prevents Metal shader compilation on Apple M5 hardware.
• To verify the workaround, set GGML_METAL_BF16_DISABLE=1 and restart the Ollama server, then check if models can be loaded and run successfully.
• The workaround can be applied by running the command GGML_METAL_BF16_DISABLE=1 ollama serve to start the server with CPU fallback.
• Note that this workaround disables GPU acceleration, so performance may be impacted.

Example

GGML_METAL_BF16_DISABLE=1 ollama serve

Notes

The provided workaround is a temporary solution until a precompiled Metal library for MTLGPUFamilyApple10 (M5) is available or the Metal compilation fallback issue is resolved.

Recommendation

Apply the workaround by setting GGML_METAL_BF16_DISABLE=1 to enable CPU fallback and allow models to run, as this is the only available solution until the underlying issue is addressed.