vllm - 💡(How to fix) Fix [RFC]: Pre-Hopper-usable 4-bit KV cache support [1 participants]

TL;DR

The issue can be addressed by adding support for 4-bit KV cache to make long-context local use more practical for pre-Hopper GPUs.

Guidance

• Investigate the feasibility of implementing plain 4-bit KV cache support for local gemma4 use, considering its potential benefits for other models as well.
• Reach out to the vLLM team to determine their interest in supporting 4-bit KV cache for pre-Hopper GPUs.
• Evaluate the potential impact of 4-bit KV cache support on the performance and usability of long-context local use.
• Consider the compatibility of 4-bit KV cache support with existing models and hardware configurations.

Example

No specific code snippet is provided due to the lack of technical details in the issue.

Notes

The issue lacks concrete technical information, making it challenging to provide a definitive solution. The guidance provided is based on the context and motivation stated in the issue.

Recommendation

Apply workaround: Implementing 4-bit KV cache support for local gemma4 use as a potential workaround to improve long-context local use on pre-Hopper GPUs. This is recommended because it addresses the core concern of making long-context local use more practical for local users.