ollama - 💡(How to fix) Fix Qwen3.5:9b concurrent call BUG [12 comments, 5 participants]

level=WARN source=sched.go:450 msg="model architecture does not currently support parallel requests" architecture=qwen35

Fix Plan

To address the issue of Ollama failing to handle concurrent requests for the qwen3.5 architecture on an ARM-based NVIDIA DGX Spark, we need to:

• Update the Ollama version to the latest, as the current version (0.17.6) may have known issues with parallelism on specific architectures.
• Modify the ggml_backend_sched_reserve function to properly handle the qwen3.5 architecture.
• Ensure that the CUDA runner is correctly configured for the Linux-ARM64 platform.

Code Changes

Here are the steps to modify the ggml_backend_sched_reserve function:

// In ggml/backend/ggml.go
func (c *Context) Reserve(graph *Graph) error {
    // Add a check for the qwen3.5 architecture
    if graph.Architecture == "qwen35" {
        // Handle the reservation for this architecture
        // This may involve setting specific parameters or using a different reservation strategy
        return c.reserveQwen35Graph(graph)
    }
    // ... rest of the function remains the same
}

func (c *Context) reserveQwen35Graph(graph *Graph) error {
    // Implementation for reserving the qwen3.5 graph
    // This may involve setting the parallelism level to 1 or using a different scheduling strategy
    // For example:
    c.parallelism = 1
    // ... rest of the implementation
}

Configuration Changes

Update the OLLAMA_NUM_PARALLEL environment variable to a value that is supported by the qwen3.5 architecture.

Temporary Workaround

If updating the Ollama version or modifying the code is not feasible, a temporary workaround could be to set the OLLAMA_NUM_PARALLEL environment variable to 1, which may prevent the crash but will also disable parallelism.

Verification

To verify that the fix worked, run the Ollama server with the modified code and configuration, and test concurrent requests for the qwen3.5 architecture. Check the server logs for any warnings or errors related to parallelism or architecture support.