ollama - 💡(How to fix) Fix qwen2.5vl:7b GGML_ASSERT(a->ne[2] * 4 == b->ne[0]) - 500 Internal Server Error on vision inference [1 participants]

During batch vision inference (PDF page-by-page extraction with Qwen2.5-VL), Ollama randomly returns 500 Internal Server Error. The server log shows: The error is not deterministic — the same image succeeds on retry or after an Ollama restart. During sustained batch processing (~10-15 sequential requests), the error rate is approximately 30-40%.

Error Pattern

• Error rate increases with sustained usage
• Splitting images into smaller halves (1288x839) reduces error frequency
• Splitting the image into 2 halves before sending reduces error rate

TL;DR

Splitting images into smaller halves before sending them to the Ollama service reduces the error rate of intermittent 500 Internal Server Errors.

Guidance

• Verify that the error is related to the image size by testing with smaller images and observing if the error rate decreases.
• Experiment with adjusting the OLLAMA_NUM_PARALLEL environment variable to see if it affects the error rate, as the current setting of 1 may not be optimal.
• Consider implementing a retry mechanism with keep_alive in the client to handle intermittent errors, as restarting the Ollama service temporarily clears the issue.
• Investigate the GGML_ASSERT error in the server log to understand the root cause of the issue, which may be related to the GPU or model configuration.

Example

No specific code example is provided, but the workaround of splitting images into smaller halves can be implemented in the client-side code.

Notes

The issue seems to be related to the image size and the Ollama service's ability to handle large images. The fact that splitting images into smaller halves reduces the error rate suggests that the service may be experiencing memory or processing limitations.

Recommendation

Apply the workaround of splitting images into smaller halves before sending them to the Ollama service, as it has been shown to reduce the error rate. This workaround can be implemented until a more permanent fix is found.