PR #38779: [Transformers v5] Add engine_core.shutdown() to LLMEngine.del

• Repository: vllm-project/vllm
• Author: Cursx
• State: closed | merged: False
• Link: https://github.com/vllm-project/vllm/pull/38779

Description (problem / solution / changelog)

Purpose

fix https://github.com/vllm-project/vllm/issues/38384

huggingface_hub >= 1.0.0 switched the HTTP backend from requests to httpx. httpx creates non-daemon threads, which causes threading._shutdown() to hang when the Python process shuts down.These changes could lead to some unexpected issues in the CI pipeline.

issue https://github.com/vllm-project/vllm/issues/38384 mentioned that "This test seems to be reliably failing in CI when v5 is installed, but I have not been able to reproduce it locally." This might be related to a specific combination of CI environment conditions (huggingface-hub >= 1.0.0 + no cache + Docker networking), which causes the httpx threads to stay alive when the child process exits.

update：The current version of huggingface-hub in the main branch CI is 0.36.2. Meanwhile, transformers v5 requires huggingface_hub >= 1.0.0, an update that involves a transition in the HTTP backend.

Test Result

CI test result in https://github.com/vllm-project/vllm/pull/30566

link https://buildkite.com/vllm/ci/builds/59212/steps/canvas?jid=019d484e-3c50-4ece-a743-6073d55b5eb5：

[2026-04-01T09:46:39Z] (EngineCore pid=663) INFO 04-01 09:46:39 [core.py:1210] Shutdown initiated (timeout=0) [2026-04-01T09:46:39Z] (EngineCore pid=663) INFO 04-01 09:46:39 [core.py:1233] Shutdown complete [2026-04-01T09:46:40Z] gpu memory used/total (GiB): 0=0.45/22.49; [2026-04-01T09:46:40Z] Done waiting for free GPU memory on devices devices=[0] (threshold='2.0 GiB') dur_s=0.00 [2026-04-01T09:46:40Z] PASSED

The GPU memory wasn't released at all after del llm—it remained stuck at 20.65 GiB right up until the timeout. The clearance threshold is 2.0 GiB, but the memory was pinned at 20.65 GiB.：

[2026-04-01T09:48:18Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:23Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:28Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:33Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:38Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:43Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:48Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:53Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:58Z] gpu memory used/total (GiB): 0=20.65/22.49; 1=20.65/22.49; [2026-04-01T09:48:59Z] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++

update: Notice this detail 👆: if the child process had actually been SIGKILLed, the OS should have reclaimed the GPU memory within a few seconds. Instead, the logs show it pinned at 20.65 GiB for a full 2 minutes. This strongly implies that the child process is still alive, rather than "killed but memory not released." It seems a GC delay might be preventing the SIGTERM from being dispatched. But here is the question: would this cause the test to fail reliably in CI while remaining un-reproducible locally?🤔

I took a look at the code related to Worker. I initially suspected a GC delay, but test_vllm_gc_ed confirms there are no circular references — del llm triggers cleanup immediately. The 2-minute memory hold is better explained by orphaned Worker processes: once EngineCore is SIGKILLed, its Worker children become orphans outside kill_process_tree's reach, and they continue holding GPU memory indefinitely.

So, a more complete picture of the situation looks like this: EngineCore likely hangs at threading._shutdown() due to httpx background threads from huggingface_hub. The parent process's 5-second join timeout then expires, and it SIGKILLs EngineCore. At that point, _ensure_worker_termination() never runs, so Worker processes become orphaned and continue holding ~20 GiB of GPU memory indefinitely.

Adding os._exit() after engine_core.shutdown() bypasses threading._shutdown(), allowing EngineCore to exit cleanly within the parent's timeout. This preserves the entire shutdown chain: engine_core.shutdown() → model_executor.shutdown() → _ensure_worker_termination() gets to run, which SIGTERMs/SIGKILLs Workers while they are still child processes of EngineCore, so the OS properly reclaims their GPU memory.

cc @hmellor

update: weakref.finalize relies on the reference count reaching zero or garbage collection (GC). However, the global httpx client in huggingface_hub >= 1.0.0 may indirectly hold references to MPClient or its higher-level objects through mechanisms such as callbacks or caching. In a CI environment (no cache → triggers HTTP download → httpx client is created), this reference chain prevents the reference count from dropping to zero, causing weakref.finalize to never trigger.

Changed files

• vllm/v1/engine/llm_engine.py (modified, +2/-0)

PR #4084: Fix reference cycle in hf_raise_for_status delaying object destruction

• Repository: huggingface/huggingface_hub
• Author: yg7445
• State: closed | merged: False
• Link: https://github.com/huggingface/huggingface_hub/pull/4084

Description (problem / solution / changelog)

Summary

Commit 098091fe ("#3889") changed hf_raise_for_status() from inline raises to storing exceptions in local variables before raising:

# Before (v1.5.0) — no cycle
raise _format(RemoteEntryNotFoundError, message, response) from e

# After (v1.6.0) — creates cycle
entry_err = _format(RemoteEntryNotFoundError, message, response)
entry_err.repo_type = repo_type
entry_err.repo_id = repo_id
raise entry_err from e

This creates a CPython reference cycle:

1. entry_err.__cause__ → e (the original HTTPStatusError)
2. e.__traceback__ → traceback → tb_frame → hf_raise_for_status frame
3. hf_raise_for_status frame → f_locals['entry_err'] → back to (1)

The cycle prevents the exception from being freed by refcounting when except blocks exit. The cyclic GC will eventually collect it, but the delay is long enough to cause real problems. When this exception propagates through callers (e.g. transformers.cached_files → LLM.__init__), the traceback chain holds a reference to self in the caller's frame, preventing deterministic cleanup.

In vLLM, this means del llm doesn't immediately trigger the weakref.finalize that sends SIGTERM to the EngineCore subprocess, so GPU memory isn't released until the cyclic GC eventually runs. Bisected to v1.6.0 — v1.5.0 works fine. Related: vllm-project/vllm#38384.

Fix

Move repo_type/repo_id/bucket_id assignment into helper functions (_format_with_repo_info, _format_with_bucket_info) so the exception object is never stored as a local variable in hf_raise_for_status's frame. This preserves the functionality added in #3889 while avoiding the reference cycle.

Test plan

• Added unit tests for _format_with_repo_info and _format_with_bucket_info
• Verified all existing hf_raise_for_status tests still pass
• Verified vLLM's tests/v1/shutdown/test_delete.py passes (8/8) with this fix

[!NOTE] Low Risk Low risk: refactors how Hub HTTP exceptions are constructed/raised while preserving error types and attached metadata; main risk is subtle differences in exception object lifetimes or attributes in edge cases.

Overview Fixes hf_raise_for_status to avoid creating reference cycles when enriching raised HTTP exceptions with repo/bucket metadata.

This introduces _format_with_repo_info and _format_with_bucket_info helpers that set repo_type/repo_id/bucket_id on the error without keeping the exception in local variables, and updates the relevant raise paths to use them. Adds focused unit tests covering both helpers.

^{Reviewed by Cursor Bugbot for commit e49bb83f22d0e259a203da656370d6e388d7a1c9. Bugbot is set up for automated code reviews on this repo. Configure here.}

Changed files

• src/huggingface_hub/utils/_http.py (modified, +38/-19)
• tests/test_utils_http.py (modified, +56/-1)

PR #4092: Fix reference cycle in hf_raise_for_status causing delayed object destruction

• Repository: huggingface/huggingface_hub
• Author: Wauplin
• State: closed | merged: True
• Link: https://github.com/huggingface/huggingface_hub/pull/4092

Description (problem / solution / changelog)

Why? PR equivalent to https://github.com/huggingface/huggingface_hub/pull/4084 but slightly cleaner. Should solved vLLM garbage collection problem (https://github.com/vllm-project/vllm/issues/38384). It seems that in https://github.com/huggingface/huggingface_hub/pull/3889 we've introduced a reference cycle issue in hf_raise_for_status, making it impossible to free-up memory correctly.

I have tested the regression test introduced in this PR and it does fail on main

Kudos to @yg7445 for investigating + suggesting the solution. This was not an easy bug to spot!

Summary

Fixes a CPython reference cycle in hf_raise_for_status() that prevents deterministic exception cleanup.

When exceptions are stored in local variables before raise ... from e, a reference cycle forms: err.__cause__ → e → e.__traceback__ → frame → f_locals['err'] → back to start. This delays garbage collection and causes real issues downstream (e.g., vLLM GPU memory not released until cyclic GC runs). See vllm-project/vllm#38384.

Alternative approach to #4084: instead of introducing two new helper functions, this extends the existing _format() with **attrs so attributes (like repo_type, repo_id, bucket_id) are set inside _format and the exception is never stored in the caller's frame.

Changes

• Extended _format() to accept **attrs keyword arguments, set as attributes on the error
• Converted all 5 call sites in hf_raise_for_status from local-variable pattern to inline raise _format(...) from e
• Added regression test using weakref to verify no reference cycle exists

🤖 Generated with Claude Code

[!NOTE] ^{Cursor Bugbot is generating a summary for commit 8ba3871177887a96e5e1d19c013f24b082d006d0. Configure here.}

Changed files

• src/huggingface_hub/utils/_http.py (modified, +14/-21)
• tests/test_utils_http.py (modified, +37/-0)

PR #39695: Introduce De-dup/Similarity-Check in CI Workflow for PR/Issue

• Repository: vllm-project/vllm
• Author: panpan0000
• State: open | merged: False
• Link: https://github.com/vllm-project/vllm/pull/39695

Description (problem / solution / changelog)

Co-Author: Trae + GPT5.3-Codex

Purpose

Example to explain https://github.com/vllm-project/vllm/issues/39694

Example Algorithm:

• Scoring: 0.75 * text_similarity + 0.25 * file_overlap .
• Threshold used for report: 0.75 .
• Using Github Action CI Cache to temp save the Github API result cache for recent 1000 PR/500 issue..etc

Test Plan

Using 1000 recent PR to test the similarity check :

High-similarity pairs ( >=0.75 ): 26

Test Result

PR Similarity

• Repo: vllm-project/vllm
• PR count: 1000
• Candidate pairs: 17375
• High-similarity pairs (>= 0.75): 26

Similar Issues:

• Repo: vllm-project/vllm
• Issue count: 500
• Candidate pairs: 9909
• High-similarity pairs (>= 0.75): 12

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Changed files

• .github/workflows/detect-duplicate-issues.yml (added, +64/-0)
• .github/workflows/detect-duplicate-prs.yml (added, +55/-0)
• .github/workflows/scripts/detect_duplicate_issues.py (added, +453/-0)
• .github/workflows/scripts/detect_duplicate_prs.py (added, +317/-0)