ollama - 💡(How to fix) Fix Flash Attention gating for deepseek2 appears controlled by GGUF metadata [1 participants]

TL;DR

Aligning GGUF metadata fields for the deepseek2 architecture may enable Flash Attention support without modifying model weights.

Guidance

• Investigate the GGUF metadata fields used by models that support Flash Attention (e.g., glmmoelite arch) to identify the specific keys controlling FA eligibility.
• Compare the GGUF metadata fields between deepseek2 and glmmoelite architectures to determine the differences that might be blocking FA support.
• Check the attention type, head dimensions, and KV layout used in deepseek2 models to see if they meet the constraints required for Flash Attention.
• Verify if updating the GGUF metadata fields for deepseek2 models to match those of supported architectures enables Flash Attention without requiring changes to model weights.

Notes

The solution relies on identifying the correct GGUF metadata fields and understanding the constraints for Flash Attention support, which may require further investigation and experimentation.

Recommendation

Apply workaround: Update the GGUF metadata fields for deepseek2 models to match those of supported architectures, as this may enable Flash Attention support without modifying model weights.