Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix Even when all guards are dropped dynamo dispatch is +4% higher latency on vLLM!! [2 comments, 2 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#179056•Fetched 2026-04-08 02:21:49
View on GitHub
Comments
2
Participants
2
Timeline
111
Reactions
0
Author
laithsakkalaithsakka
Participants
anijain2305anijain2305laithsakkalaithsakka
Timeline (top)
mentioned ×52subscribed ×52labeled ×5commented ×2

Root Cause

when aot compile =1

this is NOT issue in compile because anyway we skip dynamo dispatch

Fix Action

Fix / Workaround

when aot compile =0

command1

cd /home/lsakka/vllm && HF_HUB_OFFLINE=1 VLLM_USE_AOT_COMPILE=0 VLLM_USE_BYTECODE_HOOK=1 CUDA_VISIBLE_DEVICES=0 /home/lsakka/.conda/envs/fresh-test/bin/python -m vllm.entrypoints.cli.main bench latency --model facebook/opt-125m --batch-size 1 --input-len 128 --output-len 8 --num-iters 1000 --num-iters-warmup 10 --compilation-config '{"cudagraph_mode": "none"}' --output-json /tmp/latency_hook_1.json

comand2

 cd /home/lsakka/vllm && HF_HUB_OFFLINE=1 VLLM_USE_AOT_COMPILE=0 VLLM_USE_BYTECODE_HOOK=0 CUDA_VISIBLE_DEVICES=0 /home/lsakka/.conda/envs/fresh-test/bin/pyth
on -m vllm.entrypoints.cli.main bench latency --model facebook/opt-125m --batch-size 1 --input-len 128 --output-le
n 8 --num-iters 1000 --num-iters-warmup 10 --compilation-config '{"cudagraph_mode": "none"}' --output-json /tmp/la
tency_hook_0.json

result is hook =1 (skip dynamo dispatch)

when aot compile =1

this is NOT issue in compile because anyway we skip dynamo dispatch

Code Example

cd /home/lsakka/vllm && HF_HUB_OFFLINE=1 VLLM_USE_AOT_COMPILE=0 VLLM_USE_BYTECODE_HOOK=1 CUDA_VISIBLE_DEVICES=0 /home/lsakka/.conda/envs/fresh-test/bin/python -m vllm.entrypoints.cli.main bench latency --model facebook/opt-125m --batch-size 1 --input-len 128 --output-len 8 --num-iters 1000 --num-iters-warmup 10 --compilation-config '{"cudagraph_mode": "none"}' --output-json /tmp/latency_hook_1.json

---

cd /home/lsakka/vllm && HF_HUB_OFFLINE=1 VLLM_USE_AOT_COMPILE=0 VLLM_USE_BYTECODE_HOOK=0 CUDA_VISIBLE_DEVICES=0 /home/lsakka/.conda/envs/fresh-test/bin/pyth
on -m vllm.entrypoints.cli.main bench latency --model facebook/opt-125m --batch-size 1 --input-len 128 --output-le
n 8 --num-iters 1000 --num-iters-warmup 10 --compilation-config '{"cudagraph_mode": "none"}' --output-json /tmp/la
tency_hook_0.json
RAW_BUFFERClick to expand / collapse

when aot compile =0

command1

cd /home/lsakka/vllm && HF_HUB_OFFLINE=1 VLLM_USE_AOT_COMPILE=0 VLLM_USE_BYTECODE_HOOK=1 CUDA_VISIBLE_DEVICES=0 /home/lsakka/.conda/envs/fresh-test/bin/python -m vllm.entrypoints.cli.main bench latency --model facebook/opt-125m --batch-size 1 --input-len 128 --output-len 8 --num-iters 1000 --num-iters-warmup 10 --compilation-config '{"cudagraph_mode": "none"}' --output-json /tmp/latency_hook_1.json

comand2

 cd /home/lsakka/vllm && HF_HUB_OFFLINE=1 VLLM_USE_AOT_COMPILE=0 VLLM_USE_BYTECODE_HOOK=0 CUDA_VISIBLE_DEVICES=0 /home/lsakka/.conda/envs/fresh-test/bin/pyth
on -m vllm.entrypoints.cli.main bench latency --model facebook/opt-125m --batch-size 1 --input-len 128 --output-le
n 8 --num-iters 1000 --num-iters-warmup 10 --compilation-config '{"cudagraph_mode": "none"}' --output-json /tmp/la
tency_hook_0.json

result is hook =1 (skip dynamo dispatch)

<img width="1222" height="419" alt="Image" src="https://github.com/user-attachments/assets/290b76c6-ca12-460c-b709-09fd5e4bf90e" />

when aot compile =1

this is NOT issue in compile because anyway we skip dynamo dispatch

<img width="1005" height="387" alt="Image" src="https://github.com/user-attachments/assets/4125711a-eb4f-4793-8d48-dc4b2f71016b" /> <img width="1022" height="370" alt="Image" src="https://github.com/user-attachments/assets/bb13944b-b52f-4d2e-bcad-4e1140bf4b63" />

cc @jerryzh168 @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @amjames @Lucaskabela @jataylo @azahed98

extent analysis

TL;DR

The issue seems to be related to the performance difference when using AOT compilation, and setting VLLM_USE_AOT_COMPILE=1 may resolve the issue.

Guidance

  • Compare the performance of the two commands with VLLM_USE_AOT_COMPILE=0 and VLLM_USE_AOT_COMPILE=1 to see if the issue is indeed related to AOT compilation.
  • Investigate why setting VLLM_USE_BYTECODE_HOOK=1 seems to skip Dynamo dispatch, and how this affects performance.
  • Check the documentation for VLLM_USE_AOT_COMPILE and VLLM_USE_BYTECODE_HOOK to understand their intended behavior and potential interactions.
  • Verify that the performance difference is not due to other factors, such as differences in input data or environment variables.

Notes

The issue description lacks specific details about the expected behavior and the performance metrics being measured, making it difficult to provide a more targeted solution.

Recommendation

Apply workaround by setting VLLM_USE_AOT_COMPILE=1 to see if it resolves the performance issue, as the provided images suggest that AOT compilation may not be the cause of the problem.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#vector store#embedding generation#cache error#pipeline error#runtime error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING