vllm - 💡(How to fix) Fix _update_request_as_session does not update max_tokens from StreamingUpdate [2 comments, 2 participants]

Bug Description

In _update_request_as_session (vllm/v1/core/sched/scheduler.py), the method updates sampling_params and arrival_time from the StreamingUpdate, but does not update session.max_tokens.

The StreamingUpdate dataclass carries a max_tokens field (vllm/v1/request.py, line 41), but this value is silently discarded.

Affected Code

# scheduler.py, _update_request_as_session
session.arrival_time = update.arrival_time
session.sampling_params = update.sampling_params
# max_tokens is not updated from update.max_tokens

Impact

In streaming sessions where subsequent chunks specify a different max_tokens, the request always uses the initial chunk's value. The stop condition in check_stop() (request.num_output_tokens >= request.max_tokens) uses a stale value, causing:

• Too few tokens generated if the user increases max_tokens in a subsequent chunk
• Too many tokens generated if the user decreases it

Fix

Add session.max_tokens = update.max_tokens after the sampling_params update.