vllm - 💡(How to fix) Fix [Performance]: Async scheduling on AMD hardware

Proposal to improve performance

No response

Report of performance regression

It looks like on AMD hardware (I'm using MI350X), schedule can't properly overlap with GPU computation. Explanation from CodeX:

On CUDA, graph launch behaves like an enqueue, so vLLM overlaps CPU scheduling with GPU graph execution. On MI350X ROCm, hipGraphLaunch appears to hold the calling Python thread for nearly the full graph execution. As a result, async scheduling is enabled but cannot create useful overlap in TP=1 uni; scheduling runs after the graph/sample boundary and creates a GPU idle bubble before the next forward.

Is this a known issue? Looks like it's ROCm runtime limitation.

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.