vllm - 💡(How to fix) Fix [Bug]: vLLM v1 with prefix caching: first request differs from subsequent identical requests at temperature=0 [1 participants]

Your current environment

Waiting on our institution cluster to get back to me, someone submitted a ton of jobs blocking this simply script, will update immediately when it's returned, this bug should be able to reproduced without the enviroment, since it bugs out on startup :)

🐛 Describe the bug

This bug can be simply circumvented by just having a few warm up rounds, but I still report it here for the record 😶‍🌫️

Brief Env:

vLLM version: 0.19.0 Model: Qwen/Qwen3-8B GPU: NVIDIA H100 80GB (SXM) OS: Linux (RHEL 8) CUDA: 12.x PyTorch: (matches vllm 0.19 requirements)

Summary

With --enable-prefix-caching (the v1 default for supported models), the same /v1/completions request sent multiple times sequentially to the same vLLM server returns 2 distinct outputs at temperature=0:

• Run 1 on a freshly started server returns output A.
• Runs 2..N return output B ≠ A, but stable across runs.

Restarting the server returns the first request to A. Disabling prefix caching with --no-enable-prefix-caching makes the output deterministic across all runs.

This is a correctness bug, since the documented behavior at temperature=0 is deterministic decoding.

Server (default v1 config; prefix caching is auto-enabled for Qwen3-8B)

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen3-8B --port 8002 \
    --max-model-len 4096 --gpu-memory-utilization 0.4

Note: in vLLM v1, enable_prefix_caching defaults to model_config.is_prefix_caching_supported (True for Qwen3-8B). Removing --enable-prefix-caching from the cmdline does NOT disable it; only --no-enable-prefix-caching does.

Reproducer script

repro_for_issue.py repro_for_issue.py sends the same prompt 5 times sequentially with temperature=0, logprobs=5, stream=False, and reports unique-output count:

python repro_for_issue.py --base-url http://localhost:8002 --prompt-len 32

Observed Output (default config, prefix caching ON)

runs:       5, prompt_seed: 'repro-issue-001'
temperature: 0, logprobs: 5

  run 1/5:  \n\nOkay, let me try to figure out what's going on here. So, there's th
  run 2/5:  \nThe next completion should be deterministic because temperature is z
  run 3/5:  \nThe next completion should be deterministic because temperature is z
  run 4/5:  \nThe next completion should be deterministic because temperature is z
  run 5/5:  \nThe next completion should be deterministic because temperature is z

unique outputs: 2 / 5
First divergence at character offset 2 (output token 0):
  variant 1 token = ' \n'
  variant 2 token = ' \n\n'

Re-running the script on the same server (no restart) returns unique = 1 (variant 1 only). Restart returns the pattern.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.