vllm - 💡(How to fix) Fix [Bug]: vLLM inference produces garbled output on long context for SFT'd Qwen3-14B (short context is normal) [1 participants]

Your current environment

Your output of `python collect_env.py` here

🐛 Describe the bug

1. System Info 1.1 training env

• llamafactory-cli version 0.9.5.dev0
• transformers Version: 5.2.0
• Python 3.12.12
• torch 2.8.0+cu128
• torchaudio 2.8.0+cu128
• torchdata 0.11.0
• torchvision 0.23.0+cu128 1.2 Inference env
• vllm docker：vllm/vllm-openai:v0.18.0
• vllm docker transformers Version: 4.57.6

2、Reproduce step 2.1 After SFT of the Qwen3-14B model, the parameters for merging and exporting are configured as follows: model_name_or_path: ~/work/basemodel/Qwen3-14B adapter_name_or_path: saves/qwen3-14b-g3.0/train template: qwen3 trust_remote_code: true export export_dir: saves/qwen3-14b-g3.0/export export_size: 5 export_device: auto export_legacy_format: false

2.2 The following error occurs when loading the exported model in vLLM for the first time: Upon starting the merged model on vLLM 0.18.0, an error is raised indicating that a specific parameter in tokenizer_config.json is expected to be a dict {}, but a list [] was provided: "extra_special_tokens": [ "<|im_start|>", "<|im_end|>", "<|object_ref_start|>", "<|object_ref_end|>", "<|box_start|>", "<|box_end|>", "<|quad_start|>", "<|quad_end|>", "<|vision_start|>", "<|vision_end|>", "<|vision_pad|>", "<|image_pad|>", "<|video_pad|>" ],

2.3 I referenced the original Qwen3-14B tokenizer_config.json and changed extra_special_tokens to additional_special_tokens. After this change, the model runs successfully in vLLM.

2.4 After modifying tokenizer_config.json, I ran inference with vLLM (using parameter: --max-model-len 30720). The model returns garbled text for long inputs, but works correctly for short inputs.

2.5 Inference with transformers 4.57.1 on the same long text also returns garbled text.

2.6 However, inference with transformers 5.2.0 on the same long text works correctly.

Question: How can I resolve the garbled text issue for long context?"

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.