Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [Bug]: _find_range_for_shape in hotpath [1 pull requests, 2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#37222•Fetched 2026-04-08 00:48:41
View on GitHub
Comments
2
Participants
2
Timeline
16
Reactions
0
Author
zou3519zou3519
Participants
KrxGuKrxGuzou3519zou3519
Timeline (top)
referenced ×8commented ×2cross-referenced ×2labeled ×2

Fix Action

Fixed

PR fix notes

PR #37281: [Bugfix][Perf]: avoid Range allocation and dict hashing in _find_range_for_shape

Description (problem / solution / changelog)

Fixes #37222

_find_range_for_shape sits in the hotpath of PiecewiseBackend.__call__ - it runs on every forward pass when cudagraphs are off or the shape is outside captured sizes (raised in #37222).

Problem: every call allocates a Range object, scans a list, and hashes it twice through the range_entries dict.

Fix: precompute two structures once in __init__ after range_entries is finalized:

  • _compile_size_to_entry: dict[int, RangeEntry] - O(1) int lookup for exact sizes, no Range construction
  • _compile_ranges_and_entries: list[tuple[Range, RangeEntry]] - pre-zipped so range hits skip the second range_entries[range] hash

Numbers (500k iterations, 16 compile sizes): <img width="433" height="70" alt="image" src="https://github.com/user-attachments/assets/d0aaa489-3502-4d15-aab2-6aac047e1a38" /> Semantics preserved exactly - same None guard, same range ordering, same first-match behavior.

Changed files

  • tests/compile/test_piecewise_dispatch.py (added, +176/-0)
  • vllm/compilation/backends.py (modified, +7/-0)
  • vllm/compilation/piecewise_backend.py (modified, +60/-10)
RAW_BUFFERClick to expand / collapse

Your current environment

main

🐛 Describe the bug

when cudagraphs off or beyond max cudagraph size, this function is in the hotpath. Most notably, there's a hash.

We should be able to avoid this.

https://github.com/vllm-project/vllm/blob/714c6e0eab76a4fb1394089d848ecfe46408b9c9/vllm/compilation/piecewise_backend.py#L356-L357

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To avoid the hash calculation when CuDAGs are off or beyond the maximum size, we can add a conditional check before performing the hash operation.

Code Changes

# piecewise_backend.py

# ...

def _get_hash(self, cudagraphs):
    if not self.use_cudagraphs or len(cudagraphs) > self.max_cudagraph_size:
        # Return a default or cached hash value, or raise an exception
        return None  # or raise ValueError("CuDAGs are off or exceeded maximum size")
    # Existing hash calculation code here
    return hash(cudagraphs)

# ...

Alternatively, consider using a caching mechanism to store the hash values for recently accessed CuDAGs, reducing the need for repeated hash calculations.

Verification

To verify the fix, test the _get_hash function with different inputs, including cases where CuDAGs are off or exceed the maximum size. Ensure that the function returns the expected result without performing unnecessary hash calculations.

Extra Tips

  • Consider adding logging or monitoring to track the frequency of CuDAGs being off or exceeding the maximum size, helping identify potential performance bottlenecks.
  • Review the use_cudagraphs and max_cudagraph_size configuration settings to ensure they are properly set and updated in your application.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#docker error#permission error#memory optimization#batch processing#GPU compatibility

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

vllm - ✅(Solved) Fix [Bug]: _find_range_for_shape in hotpath [1 pull requests, 2 comments, 2 participants]