vllm - 💡(How to fix) Fix [Bug]: Qwen3.5-9B-AWQ on ROCm/vLLM 0.19.0 can get stuck generating endless "!" inside JSON schema output [4 comments, 3 participants]

vLLM server version: 0.19.0

OS:
PRETTY_NAME="Ubuntu 24.04.4 LTS"
VERSION="24.04.4 LTS (Noble Numbat)"

Kernel:
Linux saturnix-AB350M-Pro4 6.17.0-20-generic #20~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 19 01:28:37 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Python:
Python 3.12.3

GPU:
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 [Radeon RX 7900 XT/7900 XTX/7900 GRE/7900M] (rev cc)

ROCm note from host:
ROCk module is loaded
Unable to open /dev/kfd read-write: No such file or directory
saturnix is member of video group

Model:
QuantTrio/Qwen3.5-9B-AWQ

Served model name:
qwen3.5:9b

Docker image:
vllm/vllm-openai-rocm:latest

Server launch command:
sudo docker run --rm --network host \
  --device /dev/kfd --device /dev/dri \
  --group-add render \
  --ipc=host --shm-size 16G \
  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-rocm:latest \
  --model QuantTrio/Qwen3.5-9B-AWQ \
  --served-model-name qwen3.5:9b \
  --quantization awq \
  --host 127.0.0.1 \
  --port 11434 \
  --max-model-len 16384 \
  --enable-prefix-caching \
  --max-num-seqs 2 \
  --gpu-memory-utilization 0.95 \
  --language-model-only

---

{
  "keywords": [
    ["little blue truck", "little blue truck"],
    ["spring", "spring"],
    ["easter", "easter"],
    ["baby animals", "baby animals"],
    ["interactive book", "interactive book"],
    ["flap book", "flap book"],
    ["children's book", "children's book"],
    ["farm animals", "farm animals"]
  ],
  "topics": [
    ["little blue truck", "little blue truck"],
    ["spring", "spring"],
    ["easter", "easter"],
    ["farm animals", "farm animals"],
    ["baby animals", "baby animals"],
    ["interactive reading", "interactive reading!!!!!!!!!!!!!!!!!!!!!!!!!

---

from openai import OpenAI

SYSTEM_PROMPT = """You are a book metadata classifier. Return a precise JSON object with:
- keywords: 4-8 keywords, each as [original_language, english]
- topics: 4-8 topics, each as [original_language, english]
- externally_promoted: boolean
- famous_author: boolean
- is_fiction: boolean
- low_content: boolean
- confidence: integer 1-10
- notes: short English explanation, max 200 chars
"""

USER_PROMPT = """<book>
Title: Little Blue Truck's Springtime: An Easter And Springtime Book For Kids – An Interactive Adventure with Baby Animals
Author: Schertle, Alice
Publisher: Clarion Books
Market: amazon.com
Description: Celebrate the beauty of springtime with the #1 New York Times best-selling Little Blue Truck!
Beep! Beep! Little Blue Truck is out for a ride with his good friend Toad. The sun is shining and the flowers are blooming—it's a beautiful spring day! Who will they see along the way?
Open the flaps to meet all of the sweet baby animals just born on the farm. Peep! Peep!
</book>
<web_search_results>
Use the following as evidence for famous_author and externally_promoted.
Author search results:
[en.wikipedia.org] Alice Schertle (born 1941) is an American poet, teacher, and author from Los Angeles. She is known as the author of numerous children's books, most notably the New York Times best-selling Little Blue Truck series.
[aliceschertle.com] Alice Schertle is an award-winning poet whose books for children include All You Need for a Snowman and the New York Times bestselling Good Night, Little Blue Truck.
[littlebluetruckbooks.com] Alice Schertle is a poet and author of many well-loved books for children, including the beloved, #1 New York Times best-selling Little Blue Truck series.
[encyclopedia.com] Schertle, Alice, born April 7, 1941, in Los Angeles, CA...
[poetryfoundation.org] Children's poet Alice Schertle was born in Los Angeles...
Publisher search results:
[harpercollins.com] Are you a Clarion Books fan? Sign up now for Clarion Books alerts...
[instagram.com] Clarion Books (@clarionbooks) on Instagram...
[facebook.com] Clarion Books. Learn more about the latest releases from Clarion Books...
[aalbc.com] Browse books published by Clarion Books, an imprint of HarperCollins.
</web_search_results>"""

SCHEMA = {
    "type": "object",
    "properties": {
        "keywords": {
            "type": "array",
            "items": {"type": "array", "items": {"type": "string"}},
        },
        "topics": {
            "type": "array",
            "items": {"type": "array", "items": {"type": "string"}},
        },
        "externally_promoted": {"type": "boolean"},
        "famous_author": {"type": "boolean"},
        "is_fiction": {"type": "boolean"},
        "low_content": {"type": "boolean"},
        "confidence": {"type": "integer"},
        "notes": {"type": "string", "maxLength": 200},
    },
    "required": ["keywords", "topics", "is_fiction", "confidence"],
}

client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="dummy")

stream = client.chat.completions.create(
    model="qwen3.5:9b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_PROMPT},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "BookTagResult",
            "schema": SCHEMA,
        },
    },
    temperature=0.7,
    top_p=0.8,
    stream=True,
)

for chunk in stream:
    if not chunk.choices:
        continue
    delta = chunk.choices[0].delta
    text = delta.content or ""
    if text:
        print(text, end="", flush=True)