Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [Bug]: sleep mode not releasing GPU memory [5 comments, 5 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#37860•Fetched 2026-04-08 01:17:27
View on GitHub
Comments
5
Participants
5
Timeline
12
Reactions
0
Author
uuddlrlrbaba-letsgouuddlrlrbaba-letsgo
Participants
CMLKevinCMLKevininfernixinfernixnjhillnjhillrishapsrishapsuuddlrlrbaba-letsgouuddlrlrbaba-letsgo
Timeline (top)
commented ×5mentioned ×3subscribed ×3labeled ×1

Code Example

(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [block_pool.py:452] Successfully reset prefix cache
(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [cumem.py:213] CuMemAllocator: sleep freed 23.27 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 23.27 GiB is discarded directly.
(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [gpu_worker.py:140] Sleep mode freed 24.69 GiB memory, 16.27 GiB memory is still in use.
(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [abstract.py:312] It took 0.060528 seconds to fall asleep.
(APIServer pid=1331) INFO:     10.130.9.10:58068 - "POST /sleep?level=1 HTTP/1.1" 200 OK
RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

in docker image vllm/vllm-openai:v0.16.0

============================== System Info

OS : Ubuntu 22.04.5 LTS (x86_64) GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0 Clang version : Could not collect CMake version : Could not collect Libc version : glibc-2.35

============================== PyTorch Info

PyTorch version : 2.9.1+cu129 Is debug build : False CUDA used to build PyTorch : 12.9 ROCM used to build PyTorch : N/A

============================== vLLM Info

ROCM Version : Could not collect vLLM Version : 0.16.0 vLLM Build Flags: CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled

</details>

🐛 Describe the bug

(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [block_pool.py:452] Successfully reset prefix cache
(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [cumem.py:213] CuMemAllocator: sleep freed 23.27 GiB memory in total, of which 0.00 GiB is backed up in CPU and the rest 23.27 GiB is discarded directly.
(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [gpu_worker.py:140] Sleep mode freed 24.69 GiB memory, 16.27 GiB memory is still in use.
(EngineCore_DP0 pid=1531) INFO 03-23 07:20:41 [abstract.py:312] It took 0.060528 seconds to fall asleep.
(APIServer pid=1331) INFO:     10.130.9.10:58068 - "POST /sleep?level=1 HTTP/1.1" 200 OK

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

The fix involves optimizing memory allocation and deallocation in the CuMemAllocator and gpu_worker.py modules.

  • Step 1: Update CuMemAllocator
    • Modify the sleep method in cumem.py to release memory more efficiently:

def sleep(self): # Release memory allocated by CuMemAllocator self.release_memory() # Log the amount of memory freed logger.info(f"CuMemAllocator: sleep freed {self.get_freed_memory()} GiB memory in total")

*   **Step 2: Optimize `gpu_worker.py`**
    *   Update the `sleep_mode` method in `gpu_worker.py` to reduce memory usage:
    ```python
def sleep_mode(self):
    # Release memory allocated by gpu_worker
    self.release_gpu_memory()
    # Log the amount of memory freed and still in use
    logger.info(f"Sleep mode freed {self.get_freed_gpu_memory()} GiB memory, {self.get_used_gpu_memory()} GiB memory is still in use")
  • Step 3: Implement Memory Profiling
    • Add memory profiling to identify memory leaks and optimize memory usage:

import tracemalloc

def profile_memory(func): def wrapper(*args, kwargs): tracemalloc.start() result = func(*args, kwargs) current, peak = tracemalloc.get_traced_memory() logger.info(f"Memory usage: {current / 106} MiB, Peak memory usage: {peak / 106} MiB") tracemalloc.stop() return result return wrapper


### Verification
To verify the fix, monitor the memory usage and logs after applying the changes. Check for any memory leaks or issues with memory allocation and deallocation.

### Extra Tips
*   Regularly review and optimize memory usage to prevent regressions.
*   Use memory profiling tools to identify and fix memory-related issues.
*   Consider implementing automated tests to detect memory leaks and issues.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#cache issue#memory leak#API versioning#request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

vllm - 💡(How to fix) Fix [Bug]: sleep mode not releasing GPU memory [5 comments, 5 participants]