🚀 The feature, motivation and pitch

vLLM already supports --max-model-len auto, which is very useful because vLLM can determine the maximum context length that fits the current memory budget after loading the model and profiling memory usage.

I would like to request combining this auto behavior with a user-defined upper bound.

Example:

vllm serve <model> \
  --max-model-len auto \
  --max-model-len-cap 131072

Semantics:

fitted_len = maximum context length vLLM can fit in the configured memory budget effective_max_model_len = min(fitted_len, max_model_len_cap)

Motivation

auto is great when the goal is “use as much context as possible”. However, on shared or restricted hardware, this can be undesirable. If a model supports a very large context length, e.g. 1M tokens, auto may consume the available vLLM memory budget for KV cache, leaving less room for other local applications or parallel serving processes.

Simply reducing --gpu-memory-utilization or setting a fixed memory budget does not fully solve this. The useful context length depends on model weight size, architecture, KV cache dtype, parallelism, and other runtime details. A small model can consume the same memory budget as a larger model simply by fitting more context.

Pitch

Use the best context length that fits, but never exceed the deployment/product limit.

This would be useful for local serving, small-business internal deployments, workstations, and shared GPUs.

Since vLLM already has the auto-fit logic and updates the effective max model length internally, this may be a relatively small code change: apply an optional cap after the auto-fit value is computed, before finalizing the engine configuration.

Alternatives

Example: Desired context is 128k, but it might not always fit completely and the model has a much larger max context limit.

Fixed --max-model-len vllm serve <model> --max-model-len 131072

This works only if the chosen value fits. If it does not fit, startup fails instead of gracefully falling back to a smaller fitting value.

Plain --max-model-len auto

This is robust, but it may select a context length larger than the deployment wants to expose, causing unnecessary KV-cache memory usage.

--gpu-memory-utilization or --kv-cache-memory-bytes

These help limit memory, but they do not express the actual intent: automatically fit the context length, while capping the served maximum context.

External gateway limit

A proxy can reject requests above 128k tokens, but vLLM may still reserve KV cache for a larger auto-selected context length. This gives the right API behavior, but not the desired memory behavior.

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.