vllm - 💡(How to fix) Fix [CI Failure]: mi300_1: V1 Core + KV + Metrics [2 comments, 1 participants]

ERROR 04-29 07:29:19 [worker.py:2112] Traceback (most recent call last): ERROR 04-29 07:29:19 [worker.py:2112] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/nixl/worker.py", line 2096, in _read_blocks ERROR 04-29 07:29:19 [worker.py:2112] handle = self.nixl_wrapper.make_prepped_xfer( ERROR 04-29 07:29:19 [worker.py:2112] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 04-29 07:29:19 [worker.py:2112] File "/vllm-workspace/tests/v1/kv_connector/unit/test_nixl_connector.py", line 1957, in make_prepped_xferERROR 04-29 07:29:19 [worker.py:2112] raise RuntimeError("BAD STATUS") ERROR 04-29 07:29:19 [worker.py:2112] RuntimeError: BAD STATUS

TL;DR

Investigate the RuntimeError: BAD STATUS exception in the make_prepped_xfer method of test_nixl_connector.py to resolve the failing test.

Guidance

• Review the worker.py and test_nixl_connector.py files to understand the context of the make_prepped_xfer method and the RuntimeError exception.
• Check the status handling in the make_prepped_xfer method to ensure it correctly handles different status codes.
• Verify the test environment and hardware status, particularly for mi300_1, to rule out any external factors contributing to the failure.
• Analyze the test history and break frequency to identify any patterns or correlations with other changes or events.

Notes

The issue seems to be related to a specific test case and hardware configuration, so the solution may require a deep understanding of the test code and the underlying system.

Recommendation

Apply workaround: Investigate and fix the status handling in the make_prepped_xfer method, as it is likely the root cause of the RuntimeError exception.