PR #15755: metal: harden for ggml initialization failures

• Repository: ollama/ollama
• Author: dhiltgen
• State: open | merged: False
• Link: https://github.com/ollama/ollama/pull/15755

Description (problem / solution / changelog)

ggml_metal_device_init performs a probe to verify the tensor API compiles. On some systems this passes, even though kernel coverage isn't complete, which results in a later crash when compiling the real kernels. This change adds a single retry if any of the error strings match this failure mode to disable the tensor API. It also hardens an error case in the Go initDevices to detect device initialization failures and panic instead of crashing later on a nil array entry.

Fixes #15734

On my test system the probe test disables the feature, so the crash behavior isn't seen. To simulate the bug, I temporarily bypassed the probe so the API was enabled, and verified the crash, then the retry kicked in properly and got models running. While running that repro, I uncovered some other rough edges and hardened those as well.

• GPU discovery used to hammer the /info API on failure cases, and cause multiple dummy loads concurrently, which broke the backend, and lead to a 30s hang/timeout instead of failing fast. This timeout has been a long running problem, and I now understand why, and this fixes it by synchronizing the dummy load inside the runner, so multiple /info calls queue up and return the device list after the initial load finishes (or fails once). This fix will likely help many users with unsupported AMD GPUs.
• StatusWriter was getting concurrent writes from go routine copies, leading to weird interleaving of the stdout/stderr data - switched to a common cmd.Stdout/Stderr object which triggers os/exec to serialize writes.
• StatusWriter was only capturing the last error which sometimes was a generic log message while the more pertinent error was detected earlier and overwritten - switched to an accumulation approach for all the matching patterns.

Changed files

• discover/runner.go (modified, +44/-24)
• llm/server.go (modified, +126/-40)
• llm/server_wait_test.go (added, +31/-0)
• llm/status.go (modified, +56/-6)
• llm/status_test.go (added, +44/-0)
• ml/backend/ggml/ggml.go (modified, +11/-3)
• runner/ollamarunner/runner.go (modified, +44/-14)