TL;DR

Address the training-inference consistency issues by fixing logprobs/logits surface consistency and supporting R3 routing replay.

Guidance

• Review and address the issues linked to training-inference consistency, such as vllm#39701 and vllm#37737.
• Investigate the rollout performance and efficiency improvements, including vllm#40244 and verl#6117.
• Consider stabilizing the RL framework serving contract as mentioned in verl#5737.

Example

No specific code snippet is provided due to the lack of detailed technical information in the issue.

Notes

The provided issue appears to be a roadmap or a list of tasks for the vLLM RL project, rather than a specific problem to be solved. Therefore, the guidance is focused on addressing the individual issues linked in the roadmap.

Recommendation

Apply workaround: Address the individual issues linked in the roadmap, starting with the training-inference consistency and rollout performance improvements, to gradually stabilize and improve the vLLM RL project.

FAIL-SAFE: Given the nature of the issue, which is more of a roadmap than a specific problem, the provided guidance is focused on addressing the individual tasks and issues linked in the roadmap.