vllm - 💡(How to fix) Fix [Bug]: Accuracy issue running Model Runner V2 with Qwen3.5 [1 comments, 2 participants]

Fix Plan

The fix involves setting the VLLM_USE_V2_MODEL_RUNNER environment variable to 0 to ensure compatibility with the model runner.

Steps to Fix

• Set the environment variable VLLM_USE_V2_MODEL_RUNNER to 0 before running the vllm serve command:

export VLLM_USE_V2_MODEL_RUNNER=0

• Run the vllm serve command with the desired model and parameters:

vllm serve "$MODEL" --tensor-parallel-size 2 --data-parallel-size 2 --enable-expert-parallel --port 9256

• Run the lm_eval command with the desired tasks and model arguments:

lm_eval --model local-completions --model_args "base_url=http://127.0.0.1:9256/v1/completions,model=$MODEL,num_concurrent=1024" --tasks gsm8k

Verification

Verify that the fix worked by checking the output of the lm_eval command. The exact match metric values should be similar to those obtained when VLLM_USE_V2_MODEL_RUNNER is set to 0.

Extra Tips

• Make sure to set the VLLM_USE_V2_MODEL_RUNNER environment variable before running the vllm serve command to ensure compatibility with the model runner.
• If you encounter any issues, try resetting the environment variable to its default value and re-running the commands.