vllm - 💡(How to fix) Fix [Bug]: MoRIIO does not support heterogenous TP [1 comments, 1 participants]

TL;DR

Update the MoRI-IO connector to iterate over the remote TP size instead of the local TP size when sending notifications.

Guidance

• Update vllm-router to store the TP size for each instance in its service registry and pass it along in the kv_transfer_params.
• Modify the connector to use the remote TP size for notification iteration, ensuring all ranks are correctly notified.
• Verify the fix by testing with different TP sizes for the D and P instances to ensure notifications are sent correctly.
• Review the registration messages to confirm that TP sizes are being sent and stored as expected.

Example

No code snippet is provided as the issue does not contain specific code references.

Notes

This fix assumes that the TP sizes are correctly sent in the registration messages and that updating the vllm-router and connector as described will resolve the notification issue.

Recommendation

Apply workaround: Update the MoRI-IO connector and vllm-router as described to ensure correct notification of remote ranks.