transformers - 💡(How to fix) Fix chat/completions API fail on Qwen3.5-0.8B for streaming inference [2 comments, 3 participants]

ERROR: Exception in ASGI application Traceback (most recent call last): File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/uvicorn/protocols/http/httptools_impl.py", line 420, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ self.scope, self.receive, self.send ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/applications.py", line 1163, in call await super().call(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/applications.py", line 90, in call await self.middleware_stack(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 191, in call with recv_stream, send_stream, collapse_excgroups(): ~~~~~~~~~~~~~~~~~~^^ File "/opt/homebrew/Cellar/[email protected]/3.14.3_1/Frameworks/Python.framework/Versions/3.14/lib/python3.14/contextlib.py", line 162, in exit self.gen.throw(value) ~~~~~~~~~~~~~~^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_utils.py", line 87, in collapse_excgroups raise exc File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 193, in call response = await self.dispatch_func(request, call_next) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/server.py", line 83, in request_id_middleware response = await call_next(request) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 168, in call_next raise app_exc from app_exc.cause or app_exc.context File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/base.py", line 144, in coro await self.app(scope, receive_or_disconnect, send_no_error) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/middleware/exceptions.py", line 63, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/routing.py", line 660, in call await self.middleware_stack(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/routing.py", line 680, in app await route.handle(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 134, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 120, in app response = await f(request) ^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 674, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<3 lines>... ) ^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/fastapi/routing.py", line 328, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/server.py", line 91, in chat_completions return await chat_handler.handle_request(body, request.state.request_id) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/chat_completion.py", line 138, in handle_request return self._streaming( ~~~~~~~~~~~~~~~^ request_id, ^^^^^^^^^^^ ...<6 lines>... tool_format=tool_format, ^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/chat_completion.py", line 174, in _streaming queue, streamer = gen_manager.generate_streaming(model, processor, inputs, gen_config, request_id=request_id) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/wei.zhang/Developer/git/Agentic/FlexServ/venvs/transformers/lib/python3.14/site-packages/transformers/cli/serving/utils.py", line 565, in generate_streaming streamer = DirectStreamer(processor._tokenizer, loop, queue, skip_special_tokens=True) ^^^^^^^^^^^^^^^^^^^^ AttributeError: 'Qwen3VLProcessor' object has no attribute '_tokenizer'. Did you mean: 'tokenizer'?

TL;DR

The issue is likely due to a change in the Qwen3VLProcessor class, and setting the stream option to False might be a temporary workaround.

Guidance

• The error message indicates that the Qwen3VLProcessor object has no attribute _tokenizer, suggesting a change in the class structure.
• The issue only occurs when the stream option is set to True, implying that the streaming functionality is not compatible with the current version of the Qwen3VLProcessor class.
• To verify the issue, try setting the stream option to False and see if the error persists.
• If the issue is indeed related to the Qwen3VLProcessor class, updating the class to use the correct attribute name (tokenizer instead of _tokenizer) might resolve the issue.

Example

No code example is provided as the issue is related to a specific class and its internal structure, which is not fully understood from the given information.

Notes

The issue seems to be related to a change in the transformers library, specifically in the Qwen3VLProcessor class. The error message suggests that the class structure has changed, and the _tokenizer attribute is no longer available. Without more information about the class and its intended behavior, it is difficult to provide a definitive solution.

Recommendation

Apply workaround: set the stream option to False to avoid the error, as the issue seems to be related to the streaming functionality. This might not be a permanent solution, but it can help to identify the root cause of the issue.