ollama - 💡(How to fix) Fix Format output for qwen 3.5 35b model does not count thinking tokens as eval [2 comments, 2 participants]

Fix Plan

To fix the issue with the qwen 3.5 35b model not counting thinking tokens as eval, but adding them as prompt_eval, we need to adjust the format parameter in the client_llm.chat request.

Code Changes

Here's an example of how to modify the request:

response = client_llm.chat(
    model="qwen3.5:35b", 
    messages=messages,
    context_length=40000,
    top_p=0.95,
    top_k=20,
    temperature=1,
    repeat_penalty=1.5,
    max_tokens=6000,
    stream=True,
    format={"output_type": "text", "include_thinking_tokens": True}, 
)

Alternatively, if you are using the ProcessedTranscription.model_json_schema() function to generate the format, you can modify it to include the include_thinking_tokens field:

class ProcessedTranscription:
    @staticmethod
    def model_json_schema():
        return {
            "output_type": "text",
            "include_thinking_tokens": True,  # Add this line
        }

Verification

To verify that the fix worked, run the same request multiple times and check the prompt_eval values. They should now be consistent and include the thinking tokens as eval.

Extra Tips

Make sure to update the ProcessedTranscription.model_json_schema() function or the format parameter in all places where the qwen 3.5 35b model is used to ensure consistency.