vllm - 💡(How to fix) Fix [Usage]: Improve error handling when task='classify' is used with decoder-only models (e.g., Qwen, Llama, Mistral) [2 comments, 2 participants]

Your current environment

Title:
Improve error handling when task='classify' is used with decoder-only models (e.g., Qwen, Llama, Mistral)

Body:

🐞 Problem Description

When initializing an LLM instance with task='classify' on a decoder-only causal language model (such as Qwen/Qwen2.5-0.5B-Instruct, meta-llama/Llama-3-8B, or mistralai/Mistral-7B-v0.1), vLLM fails during model loading with a cryptic error:

ValueError: Following weights were not initialized from checkpoint: {'score.weight'}
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

This occurs because:

• The task='classify' option triggers the use of a classification head (score layer).
• However, standard Causal LM checkpoints (like those from Qwen, Llama, Mistral) do not contain a score.weight parameter, as they are not trained for sequence classification.
• vLLM attempts to construct a model with this missing head and only fails at weight-loading time.

🔁 Reproduction Code

from vllm import LLM, SamplingParams
prompts = ['hello']
sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=1024, stop=[])
llm = LLM(model='Qwen/Qwen2.5-0.5B-Instruct', enable_lora=True, task='classify')
outputs = llm.generate(prompts, sampling_params)

Note: Same issue occurs with meta-llama/Llama-3-8B, mistralai/Mistral-7B-v0.1, etc.

✅ Expected Behavior

vLLM should validate compatibility between task and model architecture early (e.g., in LLM.__init__ or engine args parsing) and raise a clear, actionable error such as:

ValueError: The 'classify' task is only supported for encoder or encoder-decoder models 
(e.g., BERT, DeBERTa, T5). The model 'Qwen/Qwen2.5-0.5B-Instruct' uses a decoder-only 
architecture (Qwen2ForCausalLM) and does not include a classification head. 
Consider using prompt-based classification instead, or switch to a model trained for classification.

💡 Suggested Fix

Check the model’s architecture before enabling classification mode:

# Pseudocode
if task == "classify" and "CausalLM" in model_config.architectures:
    raise ValueError("task='classify' is not supported for decoder-only models...")

Additionally, the documentation for the task parameter should explicitly list supported model types.

📌 Context

• vLLM version: 0.12.0
• Model: Qwen/Qwen2.5-0.5B-Instruct (but affects all Causal LMs)
• Use case: User mistakenly assumed task='classify' works for any model; better guardrails would improve UX.

Feel free to adjust the tone or add more details, but this version clearly explains the problem, provides reproducible code, proposes a solution, and aligns with vLLM’s goal of being user-friendly and robust.

Let me know if you'd like a Chinese version or help submitting it!

How would you like to use vllm

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.