ollama - 💡(How to fix) Fix TurboQuant-MoE: SOTA 8.5x KV-cache compression with Residual Correction [1 participants]

TL;DR

The author of the TurboQuant-MoE repository is offering to contribute their residual correction logic to improve the accuracy of the TurboQuant integration in the Ollama backend.

Guidance

• Review the discussion in PR 15090 to understand the context and requirements for the TurboQuant integration.
• Evaluate the performance benefits of the proposed residual correction logic, particularly for 3-bit and 4-bit formats.
• Consider the potential impact of integrating the new logic on the overall quality and performance of the Ollama backend.
• Discuss the contribution with the author to determine the best approach for incorporating the residual correction logic.

Notes

The feasibility of the contribution depends on the compatibility of the TurboQuant-MoE repository's implementation with the Ollama backend.

Recommendation

Apply workaround: Collaborate with the author to integrate the residual correction logic, as it may improve the accuracy and performance of the TurboQuant integration.