ollama - 💡(How to fix) Fix [Feature Request] Support weightless RMSNorm (for FlashNorm weight folding trick) [1 participants]

TL;DR

Modify the RMSNorm implementation to support the FlashNorm variant by removing norm weights.

Guidance

• Review the RMSNorm implementation and identify where norm weights are being used.
• Consider adding a conditional check to skip norm weight multiplication when no norm weights are provided.
• Study the example models (Llama, Qwen, SMolLM) on Hugging Face to understand how the weight folding trick is applied.
• Evaluate the potential performance benefits of using FlashNorm in your specific use case.

Example

# Pseudocode example of modified RMSNorm implementation
if norm_weights is None:
    # Skip norm weight multiplication
    normalized_input = input / rms_norm
else:
    # Original RMSNorm implementation with norm weights
    normalized_input = input / (rms_norm * norm_weights)

Notes

The implementation details may vary depending on the specific RMSNorm implementation being used. It's essential to carefully review and test the modified implementation to ensure correctness and performance.

Recommendation

Apply workaround: Modify the RMSNorm implementation to support FlashNorm by removing norm weights, as this change can reduce inference overhead without affecting accuracy.