vllm - 💡(How to fix) Fix [Performance]: Prefix cache hit lower on vLLM than on other inference stacks [1 participants]

Fix Plan

The fix involves optimizing prefix caching for high concurrency workloads.

Steps to Fix

• Update vllm serve command to include --prefix-cache-size and --prefix-cache-eviction-policy flags.
• Implement a custom prefix cache eviction policy to handle high concurrency.

Example Code

VLLM_USE_FLASHINFER_MOE_FP4=1 \
vllm serve nvidia/DeepSeek-V3.2-NVFP4 \
    --tensor-parallel-size 8 \
    --quantization modelopt_fp4 \
    --host 0.0.0.0 \
    --port 30000 \
    --max-model-len 32768 \
    --enable-prefix-caching \
    --prefix-cache-size 10000 \
    --prefix-cache-eviction-policy lru

Alternatively, you can implement a custom cache using a library like cachetools in Python:

from cachetools import LRUCache

cache = LRUCache(maxsize=10000)

def get_prefix_cache(key):
    if key in cache:
        return cache[key]
    else:
        # Calculate prefix cache value
        value = calculate_prefix_cache(key)
        cache[key] = value
        return value

Verification

Verify the fix by monitoring the prefix cache metric and checking for improved performance. You can use tools like prometheus and grafana to monitor metrics.

Extra Tips

• Monitor cache hit ratio and adjust --prefix-cache-size accordingly.
• Consider using a more advanced caching library like redis or memcached for larger workloads.