PR #16071: create: route quantize through MLX worker thread

• Repository: ollama/ollama
• Author: ooki1jp
• State: open | merged: False
• Link: https://github.com/ollama/ollama/pull/16071

Description (problem / solution / changelog)

Fixes #16070.

Problem

Since v0.23.1, ollama create --experimental --quantize <any> panics with mlx: There is no Stream(gpu, 1) in current thread on macOS Apple Silicon. v0.23.0 worked correctly with the same Modelfile and same input safetensors.

The root cause is that #15845 tightened MLX's per-thread state requirement: every CGO entry point now runs on a runtime.LockOSThread()-locked OS thread, and MLX expects that thread to have been initialized with SetDefaultDeviceGPU (which is what registers Stream(gpu, *) for the current thread).

The inference path was migrated to satisfy this invariant via a long-lived mlxthread.Thread worker started in x/mlxrunner/server.go. The create-quantize path, however, still calls mlx.Eval / mlx.Quantize / mlx.Contiguous directly from whatever goroutine the caller happens to use:

$ grep -rn "mlxthread" x/mlxrunner/
x/mlxrunner/server.go:37: worker, err := mlxthread.Start("mlxrunner", ...)
x/mlxrunner/runner.go:44: mlxThread     *mlxthread.Thread

$ grep -rn "mlxthread" x/create/
(no matches)

The first MLX call from such a goroutine locks a fresh OS thread, that thread has no MLX state, and mlx_eval panics inside transforms.cpp:73.

Fix

This PR adds the same pattern in x/create/client: a package-level mlxthread.Thread, lazily started, whose init callback runs mlx.CheckInit + mlx.SetDefaultDeviceGPU. The two public entry points that call MLX — quantizeTensor and quantizePackedGroup — are now thin wrappers that dispatch the real work via runOnMLXWorker.

Internal helpers (loadAndQuantizeArray, stackAndQuantizeExpertGroup, decodeSourceFP8Tensor) are unchanged because they always run inside one of the two entry points and therefore on the worker thread.

Diff is two files:

• x/create/client/mlxworker.go (new, ~60 lines): worker singleton + runOnMLXWorker helper.
• x/create/client/quantize.go (~35 lines added): wrap quantizeTensor / quantizePackedGroup; rename their bodies to *OnMLXThread.

Verification

Tested on M3 Air 16 GB (the host that hit the regression).

Inputs: google/gemma-4-E4B-it (target, 15 GB BF16 safetensors) and google/gemma-4-E4B-it-assistant (drafter, 152 MB).

Other notes

• gofmt clean, go vet clean.
• Existing client package tests still pass (go test ./x/create/client/...).
• I considered a smaller fix (adding SetDefaultDeviceGPU inside mlxCheck), but that conflicts with the existing architectural decision in #15845 to centralize MLX work on a dedicated worker — applying the same pattern here keeps x/create/client consistent with x/mlxrunner.
• The worker is lazily initialized so there is no startup-time cost for non-quantize create paths.

Changed files

• x/create/client/mlxworker.go (added, +57/-0)
• x/create/client/quantize.go (modified, +35/-0)

PR #14969: Create safetensors models through a hybrid local/remote pipeline.

• Repository: ollama/ollama
• Author: dhiltgen
• State: open | merged: False
• Link: https://github.com/ollama/ollama/pull/14969

Description (problem / solution / changelog)

Local OLLAMA_HOST uses the optimized local create path by default, while non-local hosts use the API upload/create flow. OLLAMA_CREATE_REMOTE can force the remote path for testing, and OLLAMA_CREATE_SERVER_QUANTIZE can force remote quantization to happen on the server.

Remote safetensors creation now reuses x/create.CreateSafetensorsModel as the single traversal/import path. This keeps remote behavior aligned with local handling for architecture-specific transforms, source FP8 and prequantized tensors, skipped companion tensors, expert grouping, and per-tensor quantization decisions.

Fix server-side safetensors assembly to preserve Modelfile parameters and FileType metadata, and handle packed tensor groups during server-side quantization without dropping tensors. Packed groups are quantized as multi-tensor blobs instead of attempting to quantize the group name as a single tensor.

Improve local packed-group creation by streaming from file-backed TensorData for unquantized groups instead of extracting whole tensor byte slices.

Reduce GGUF creation memory by passing through BF16 tensors when no conversion is needed and by making large tensor writes exclusive while still allowing smaller tensor writes to run concurrently.

Add focused tests for remote request metadata, server/client quantization selection, packed quantized tensor preservation, local packed streaming, remote upload/create failures, and GGUF write concurrency.

Hardens MLX thread handling to enforce bound thread requirement.

Test results when creating from gemma-4-31b-it

Fixes #16070

Changed files

• api/client.go (modified, +13/-0)
• api/types.go (modified, +11/-0)
• cmd/cmd.go (modified, +54/-5)
• cmd/cmd_test.go (modified, +27/-0)
• convert/reader_safetensors.go (modified, +1/-0)
• discover/gpu_info_darwin.m (modified, +12/-4)
• docs/api.md (modified, +39/-1)
• fs/ggml/gguf.go (modified, +217/-7)
• fs/ggml/gguf_test.go (modified, +599/-0)
• integration/create_test.go (modified, +239/-81)
• progress/bar.go (modified, +4/-0)
• server/create.go (modified, +396/-1)
• server/create_test.go (modified, +85/-0)
• server/images.go (modified, +92/-54)
• server/images_test.go (modified, +20/-0)
• server/quantization.go (modified, +31/-1)
• server/routes_create_test.go (modified, +551/-119)
• server/sched_test.go (modified, +4/-4)
• x/create/capabilities.go (added, +147/-0)
• x/create/client/create.go (modified, +35/-169)
• x/create/client/create_test.go (modified, +237/-15)
• x/create/client/quantize.go (modified, +691/-176)
• x/create/client/quantize_test.go (modified, +160/-3)
• x/create/client/remote.go (added, +650/-0)
• x/create/client/remote_test.go (added, +1024/-0)
• x/create/create.go (modified, +475/-84)
• x/create/create_test.go (modified, +2/-2)
• x/create/gemma4.go (modified, +4/-22)
• x/create/quantization_planner_test.go (added, +235/-0)
• x/create/qwen35.go (modified, +6/-0)
• x/internal/mlxtest/mlxtest.go (added, +87/-0)
• x/mlxrunner/cache/cache_test.go (modified, +2/-3)
• x/mlxrunner/cache/rotating_attention_test.go (modified, +125/-97)
• x/mlxrunner/mlx/array_test.go (modified, +27/-27)
• x/mlxrunner/mlx/compile_test.go (modified, +123/-123)
• x/mlxrunner/mlx/io.go (modified, +1/-0)
• x/mlxrunner/mlx/memory.go (modified, +5/-0)
• x/mlxrunner/mlx/mlx.go (modified, +34/-0)
• x/mlxrunner/mlx/stream.go (modified, +48/-24)
• x/mlxrunner/mlx/thread_test.go (modified, +25/-0)
• x/mlxrunner/model/embedding_test.go (modified, +2/-3)
• x/mlxrunner/sample/sample_test.go (modified, +2/-3)
• x/mlxrunner/server.go (modified, +1/-0)
• x/models/gemma4/gemma4_moe_test.go (modified, +31/-19)
• x/models/gemma4/gemma4_test.go (modified, +3/-3)
• x/models/laguna/laguna_test.go (modified, +2/-3)
• x/models/nn/nn_test.go (modified, +2/-3)
• x/models/nn/sdpa_test.go (modified, +5/-4)
• x/models/qwen3_5/qwen3_5_test.go (modified, +2/-3)
• x/safetensors/extractor.go (modified, +156/-4)
• x/safetensors/extractor_test.go (modified, +113/-0)