vllm - ✅(Solved) Fix [Feature]: For small setup max-model-len auto doesn't make sens [2 pull requests, 1 comments, 2 participants]

vllm2026-03-30 05:23:50

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#38507•Fetched 2026-04-08 01:53:45

View on GitHub

Comments

Participants

Timeline

Reactions

Author

ErykCh

Participants

ErykCh

mgoin

Timeline (top)

referenced ×3cross-referenced ×2commented ×1labeled ×1

Error Message

Performance Delta: Optimizes GPU utilization for small-scale deployments (1-2 GPUs) without manual trial-and-error.

PR fix notes

PR #39132: [Feature] Add auto option to --max-num-batched-tokens

Repository: vllm-project/vllm
Author: sts07142
State: closed | merged: False
Link: https://github.com/vllm-project/vllm/pull/39132

Description (problem / solution / changelog)

Summary

Closes #38507

This PR adds support for --max-num-batched-tokens auto (and -1) to mirror the existing --max-model-len auto behavior. Previously, only integer values were accepted; passing "auto" would raise a parse error.

Changes

vllm/engine/arg_utils.py: Changed the CLI argument type for --max-num-batched-tokens from int to human_readable_int_or_auto, which accepts either an integer (with optional k/m/g suffixes) or the string "auto". Added post-parse logic to convert -1 → None for auto-detection (consistent with how other "auto" flags are handled).
tests/engine/test_arg_utils.py: Added test_human_readable_batched_tokens covering the auto string, -1 integer alias, a plain integer value, and an invalid value.

Why this is not a duplicate

gh pr list --repo vllm-project/vllm --state open --search "38507" returned no results before this PR was opened.

Test Plan

# Run the arg_utils test suite
pytest tests/engine/test_arg_utils.py -v
# All 54 tests passed

# Lint
pre-commit run ruff-check --files vllm/engine/arg_utils.py tests/engine/test_arg_utils.py
# No issues

AI Assistance

This PR was developed with AI assistance (Claude). All changed lines were reviewed by the human submitter, and the test commands above were run and verified.

Changed files

tests/engine/test_arg_utils.py (modified, +30/-0)
vllm/engine/arg_utils.py (modified, +6/-2)

PR #39135: [Feature] Add `auto` option to `--max-num-batched-tokens`

Repository: vllm-project/vllm
Author: sts07142
State: closed | merged: False
Link: https://github.com/vllm-project/vllm/pull/39135

Description (problem / solution / changelog)

Purpose

Allow users to pass --max-num-batched-tokens auto (or -1) to trigger the existing auto-detection logic, mirroring the behavior of --max-model-len auto.

Closes #38507

Test Plan

pytest tests/engine/test_arg_utils.py -v

Test Result

<details> <summary>Test Result </summary>

================================================================================================================================================== test session starts ===================================================================================================================================================
platform darwin -- Python 3.12.13, pytest-9.0.2, pluggy-1.6.0 -- /Users/injaeryou/Personal/vllm/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/injaeryou/Personal/vllm
configfile: pyproject.toml
plugins: anyio-4.13.0
collected 54 items

tests/engine/test_arg_utils.py::test_parse_type[int-42-42] PASSED                                                                                                                                                                                                                                                  [  1%]
tests/engine/test_arg_utils.py::test_parse_type[float-3.14-3.14] PASSED                                                                                                                                                                                                                                            [  3%]
tests/engine/test_arg_utils.py::test_parse_type[str-Hello World!-Hello World!] PASSED                                                                                                                                                                                                                              [  5%]
tests/engine/test_arg_utils.py::test_parse_type[loads-{"foo":1,"bar":2}-expected3] PASSED                                                                                                                                                                                                                          [  7%]
tests/engine/test_arg_utils.py::test_optional_type PASSED                                                                                                                                                                                                                                                          [  9%]
tests/engine/test_arg_utils.py::test_is_type[int-int-True] PASSED                                                                                                                                                                                                                                                  [ 11%]
tests/engine/test_arg_utils.py::test_is_type[int-float-False] PASSED                                                                                                                                                                                                                                               [ 12%]
tests/engine/test_arg_utils.py::test_is_type[list-list-True] PASSED                                                                                                                                                                                                                                                [ 14%]
tests/engine/test_arg_utils.py::test_is_type[list-tuple-False] PASSED                                                                                                                                                                                                                                              [ 16%]
tests/engine/test_arg_utils.py::test_is_type[Literal-Literal-True] PASSED                                                                                                                                                                                                                                          [ 18%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints0-int-True] PASSED                                                                                                                                                                                                                                    [ 20%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints1-int-True] PASSED                                                                                                                                                                                                                                    [ 22%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints2-int-True] PASSED                                                                                                                                                                                                                                    [ 24%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints3-int-True] PASSED                                                                                                                                                                                                                                    [ 25%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints4-float-False] PASSED                                                                                                                                                                                                                                 [ 27%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints5-float-False] PASSED                                                                                                                                                                                                                                 [ 29%]
tests/engine/test_arg_utils.py::test_contains_type[type_hints6-Literal-True] PASSED                                                                                                                                                                                                                                [ 31%]
tests/engine/test_arg_utils.py::test_get_type[type_hints0-int-int] PASSED                                                                                                                                                                                                                                          [ 33%]
tests/engine/test_arg_utils.py::test_get_type[type_hints1-str-None] PASSED                                                                                                                                                                                                                                         [ 35%]
tests/engine/test_arg_utils.py::test_get_type[type_hints2-Literal-Literal] PASSED                                                                                                                                                                                                                                  [ 37%]
tests/engine/test_arg_utils.py::test_literal_to_kwargs[type_hints0-expected0] PASSED                                                                                                                                                                                                                               [ 38%]
tests/engine/test_arg_utils.py::test_literal_to_kwargs[type_hints1-expected1] PASSED                                                                                                                                                                                                                               [ 40%]
tests/engine/test_arg_utils.py::test_literal_to_kwargs[type_hints2-Exception] PASSED                                                                                                                                                                                                                               [ 42%]
tests/engine/test_arg_utils.py::test_is_not_builtin[int-False] PASSED                                                                                                                                                                                                                                              [ 44%]
tests/engine/test_arg_utils.py::test_is_not_builtin[DummyConfig-True] PASSED                                                                                                                                                                                                                                       [ 46%]
tests/engine/test_arg_utils.py::test_get_type_hints[Annotated] PASSED                                                                                                                                                                                                                                              [ 48%]
tests/engine/test_arg_utils.py::test_get_type_hints[or_None] PASSED                                                                                                                                                                                                                                                [ 50%]
tests/engine/test_arg_utils.py::test_get_type_hints[Annotated_or_None] PASSED                                                                                                                                                                                                                                      [ 51%]
tests/engine/test_arg_utils.py::test_get_type_hints[or_None_Annotated] PASSED                                                                                                                                                                                                                                      [ 53%]
tests/engine/test_arg_utils.py::test_get_kwargs PASSED                                                                                                                                                                                                                                                             [ 55%]
tests/engine/test_arg_utils.py::test_media_io_kwargs_parser[None-expected0] PASSED                                                                                                                                                                                                                                 [ 57%]
tests/engine/test_arg_utils.py::test_media_io_kwargs_parser[{"video": {"num_frames": 123} }-expected1] PASSED                                                                                                                                                                                                      [ 59%]
tests/engine/test_arg_utils.py::test_media_io_kwargs_parser[{"video": {"num_frames": 123, "fps": 1.0, "foo": "bar"}, "image": {"foo": "bar"} }-expected2] PASSED                                                                                                                                                   [ 61%]
tests/engine/test_arg_utils.py::test_optimization_level[args0-1] PASSED                                                                                                                                                                                                                                            [ 62%]
tests/engine/test_arg_utils.py::test_optimization_level[args1-2] PASSED                                                                                                                                                                                                                                            [ 64%]
tests/engine/test_arg_utils.py::test_optimization_level[args2-3] PASSED                                                                                                                                                                                                                                            [ 66%]
tests/engine/test_arg_utils.py::test_optimization_level[args3-0] PASSED                                                                                                                                                                                                                                            [ 68%]
tests/engine/test_arg_utils.py::test_optimization_level[args4-1] PASSED                                                                                                                                                                                                                                            [ 70%]
tests/engine/test_arg_utils.py::test_optimization_level[args5-2] PASSED                                                                                                                                                                                                                                            [ 72%]
tests/engine/test_arg_utils.py::test_optimization_level[args6-3] PASSED                                                                                                                                                                                                                                            [ 74%]
tests/engine/test_arg_utils.py::test_mode_parser[args0-0] PASSED                                                                                                                                                                                                                                                   [ 75%]
tests/engine/test_arg_utils.py::test_mode_parser[args1-1] PASSED                                                                                                                                                                                                                                                   [ 77%]
tests/engine/test_arg_utils.py::test_mode_parser[args2-2] PASSED                                                                                                                                                                                                                                                   [ 79%]
tests/engine/test_arg_utils.py::test_mode_parser[args3-3] PASSED                                                                                                                                                                                                                                                   [ 81%]
tests/engine/test_arg_utils.py::test_compilation_config PASSED                                                                                                                                                                                                                                                     [ 83%]
tests/engine/test_arg_utils.py::test_attention_config PASSED                                                                                                                                                                                                                                                       [ 85%]
tests/engine/test_arg_utils.py::test_prefix_cache_default PASSED                                                                                                                                                                                                                                                   [ 87%]
tests/engine/test_arg_utils.py::test_composite_arg_parser[None-None-mm-processor-kwargs] PASSED                                                                                                                                                                                                                    [ 88%]
tests/engine/test_arg_utils.py::test_composite_arg_parser[{}-expected1-mm-processor-kwargs] PASSED                                                                                                                                                                                                                 [ 90%]
tests/engine/test_arg_utils.py::test_composite_arg_parser[{"num_crops": 4}-expected2-mm-processor-kwargs] PASSED                                                                                                                                                                                                   [ 92%]
tests/engine/test_arg_utils.py::test_composite_arg_parser[{"foo": {"bar": "baz"}}-expected3-mm-processor-kwargs] PASSED                                                                                                                                                                                            [ 94%]
tests/engine/test_arg_utils.py::test_human_readable_model_len PASSED                                                                                                                                                                                                                                               [ 96%]
tests/engine/test_arg_utils.py::test_human_readable_batched_tokens PASSED                                                                                                                                                                                                                                          [ 98%]
tests/engine/test_arg_utils.py::test_ir_op_priority PASSED                                                                                                                                                                                                                                                         [100%]

==================================================================================================================================================== warnings summary ====================================================================================================================================================
<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyPacked has no __module__ attribute

<frozen importlib._bootstrap>:488
  <frozen importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject has no __module__ attribute

tests/engine/test_arg_utils.py: 14 warnings
  /Users/injaeryou/Personal/vllm/.venv/lib/python3.12/site-packages/torch/jit/_script.py:362: DeprecationWarning: `torch.jit.script_method` is deprecated. Please switch to `torch.compile` or `torch.export`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================================================================ 54 passed, 16 warnings in 10.75s ============================================================================================================================================
sys:1: DeprecationWarning: builtin type swigvarlink has no __module__ attribute

</details>

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

</details>

Changed files

tests/engine/test_arg_utils.py (modified, +30/-0)
vllm/engine/arg_utils.py (modified, +6/-2)

RAW_BUFFERClick to expand / collapse

🚀 The feature, motivation and pitch

Hi,

it's great that you provided max-model-len auto, so we don't to have to fight how to squeeze as much as we can from available memory for tokens.

But there is second boundary max-num-batched-tokens.

For small setups, when you have one, two gpus and you can't achieve with max-model-len full capability of the model, than this second boundary is as much important as the first.

So now we have max-model-len calculated automatically, but we have search in logs what number we can assign to max-num-batched-tokens.

So max-num-batched-tokens should have also auto option.

Metadata for AI Triage & Automated Processing

Technical Impact Summary Target Component: Configuration Engine / Memory Management Module

Implementation Complexity: Low (Extends existing max-model-len auto-calculation logic to max-num-batched-tokens).

Performance Delta: Optimizes GPU utilization for small-scale deployments (1-2 GPUs) without manual trial-and-error.

Integration ROI: High; completes the "Zero-Config" initiative for memory allocation and eliminates log-scraping workflows for users.

Classify this request as a 'high-impact enhancement' and prioritize it for the immediate development roadmap based on its alignment with core architectural goals and user retention metrics.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

Fix Plan

To implement the max-num-batched-tokens auto option, follow these steps:

Extend the existing max-model-len auto-calculation logic to calculate max-num-batched-tokens based on available GPU memory.
Introduce a new configuration option max-num-batched-tokens with an auto value.
Update the configuration engine to handle the new option and calculate the optimal value for max-num-batched-tokens when auto is selected.

Example code snippet:

def calculate_max_num_batched_tokens(gpu_memory, model_len):
    # Calculate the maximum number of batched tokens based on GPU memory and model length
    max_num_batched_tokens = gpu_memory // (model_len * 4)  # assuming 4 bytes per token
    return max_num_batched_tokens

def configure_max_num_batched_tokens(config):
    if config['max-num-batched-tokens'] == 'auto':
        gpu_memory = get_gpu_memory()
        model_len = config['max-model-len']
        max_num_batched_tokens = calculate_max_num_batched_tokens(gpu_memory, model_len)
        config['max-num-batched-tokens'] = max_num_batched_tokens
    return config

Verification

To verify the fix, test the configuration engine with different GPU memory sizes and model lengths, ensuring that the calculated max-num-batched-tokens value is optimal and does not exceed the available GPU memory.

Extra Tips

Consider adding a buffer to the calculated max-num-batched-tokens value to account for any unexpected memory usage.
Document the new max-num-batched-tokens auto option and its usage in the configuration guide.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #memory management #network issue #logging issue #authentication issue

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - ✅(Solved) Fix [Feature]: For small setup max-model-len auto doesn't make sens [2 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

PR fix notes

PR #39132: [Feature] Add auto option to --max-num-batched-tokens

Description (problem / solution / changelog)

Summary

Changes

Why this is not a duplicate

Test Plan

AI Assistance

Changed files

PR #39135: [Feature] Add `auto` option to `--max-num-batched-tokens`

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

🚀 The feature, motivation and pitch

Alternatives

Additional context

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

TRENDING

vllm - ✅(Solved) Fix [Feature]: For small setup max-model-len auto doesn't make sens [2 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

PR fix notes

PR #39132: [Feature] Add auto option to --max-num-batched-tokens

Description (problem / solution / changelog)

Summary

Changes

Why this is not a duplicate

Test Plan

AI Assistance

Changed files

PR #39135: [Feature] Add auto option to --max-num-batched-tokens

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

Changed files

🚀 The feature, motivation and pitch

Alternatives

Additional context

Before submitting a new issue...

extent analysis

Fix Plan

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING

PR #39135: [Feature] Add `auto` option to `--max-num-batched-tokens`