litellm - 💡(How to fix) Fix Conflict with **K8s Gateway API Inference Extension**

What is the problem?

We are using K8s Gateway API Inference Extension (with Envoy Gateway) for LLM traffic. Its core capability is:

• Select the final inference Pod (vLLM/SGLang) based on KV cache affinity, GPU memory, queue depth at the gateway layer.
• It routes requests directly to the optimal Pod IP:port, with no further routing expected.

However, if we place LiteLLM behind this gateway:

1. Gateway already picks the best Pod (KV-cache-aware, low load).
2. LiteLLM re-routes the request again (round-robin/least-busy over all backends).
3. The gateway’s KV-cache-based optimal selection is completely lost / overwritten.

This creates a fundamental conflict in the same request path:

• Gateway: “I’ve already chosen the final Pod.”
• LiteLLM: “I’ll ignore that and choose again.”

Are there any suggestions to handle it?

Thanks in advance.