ollama - 💡(How to fix) Fix [v0.30.0-rc] Memory Degradation: 25GB KV Cache stuck on single-concurrency RAG stack unless upstream TCP connection is physically reset

What is the issue?

What is the issue?

In the new llama.cpp native runner architecture, there is an issue with memory/VRAM reclamation during single-user, long-context scenarios (128K window).

The ~25GB KV Cache is not released when a session ends or a "New Conversation" is triggered in the upstream app (Dify/RAGFlow). Instead, it remains locked in VRAM (stuck at ~80GB total allocation).

Crucially, the exact moment the upstream Reranker service (managed by Xinference) is restarted—which forces a physical TCP connection closure—Ollama immediately and cleanly flushes the 25GB KV Cache, dropping the VRAM back down to the model's static weight level (~55GB).

This indicates that the new llama.cpp runner heavily relies on absolute TCP connection teardown to flush the context cache, causing unexpected memory degradation when upstream API gateways/microservices keep the HTTP connection alive.

Steps to reproduce

1. Spin up a full RAG stack via Docker: Dify + RAGFlow + Xinference (Reranker) + Ollama (v0.30.0 Pre-release).
2. Load Qwen3.6:27B-BF16 and set the context window to 128K.
3. Initiate a massive RAG query that utilizes the full 128K context. Total VRAM allocation hits ~80GB (54GB static weights + 25GB KV Cache) on the Blackwell GPU.
4. Click "New Conversation" in Dify/RAGFlow to end the session. The 25GB KV Cache is unpredictably retained in VRAM and does not release after several minutes.
5. Manually run docker restart <xinference-reranker-container>.
6. Observation: The moment the connection resets, Ollama's VRAM drops instantly back to ~55GB.

Expected behavior

Ollama should have an internal timeout or explicit API endpoint to flush the KV Cache of an inactive session, rather than permanently locking up to 25GB of VRAM just because an upstream proxy/reranker keeps a TCP connection open or lingering.

Environment

• OS: Windows 11 (Docker Desktop via WSL2 backend)
• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB GDDR7)
• Ollama Version: v0.30.0-rcX (Please fill in your exact rc version tag here)
• Model: Qwen3.6:27B-BF16
• Upstream components: Dify, RAGFlow, Xinference (hosting the Reranker)

Relevant log output

OS

No response

GPU

No response

CPU

No response

Ollama version

No response