vllm - 💡(How to fix) Fix [CI] test_no_sync_with_spec_decode[eagle3-llama]: unexpected GPU-CPU sync [1 participants]

FAILED test_async_spec_decode.py::test_no_sync_with_spec_decode[eagle3-llama] AssertionError: Unexpected GPU-CPU sync: seq_lens_cpu lazy init triggered 2 times. See stack traces above. assert 2 == 0

TL;DR

The most likely fix involves addressing the GPU-CPU synchronization issue caused by seq_lens_cpu lazy initialization, potentially by revisiting the implementation of async scheduling in speculative decoding.

Guidance

• Review the changes introduced in PR #39206 to understand the added assertion and its implications on the test.
• Investigate the seq_lens_cpu lazy initialization in the context of speculative decoding and async scheduling to identify potential synchronization points.
• Consider revisiting the implementation of async scheduling to minimize or eliminate GPU-CPU synchronization, as discussed in issue #29134.
• Analyze the stack traces provided in the error details to gain a deeper understanding of the synchronization issue.

Example

No specific code snippet can be provided without further context, but reviewing the implementation of seq_lens_cpu lazy initialization and its interaction with async scheduling may help identify the root cause.

Notes

The provided information suggests that the issue is related to a known limitation (issue #29134) that hasn't been resolved yet. The test is correctly detecting a real synchronization issue, and addressing this will require a deeper understanding of the speculative decoding and async scheduling implementation.

Recommendation

Apply workaround: Revisit the implementation of async scheduling in speculative decoding to minimize or eliminate GPU-CPU synchronization, as this is a known limitation that needs to be addressed to resolve the issue.