ollama - 💡(How to fix) Fix gemma4-64k chat-template special tokens (<|tool_call|>, <|"|>, <|channel|>) leak into OpenAI-compat tool-calling output [1 participants]

TL;DR

The issue can be potentially resolved by upgrading to a version where the fix for similar issues (#15241, #15315) is confirmed to be stable, or by exploring alternative model families as a workaround.

Guidance

• Review the release notes for versions after 0.20.6 to confirm if the fix for issues #15241 and #15315 is stable and applicable to the current scenario.
• Test the behavior with different model families to identify if the issue is model-specific or a broader compatibility problem with the Ollama OpenAI-compat endpoint.
• Investigate the tool schema configuration for /v1/chat/completions to ensure it aligns with the expected format for tool calls, potentially adjusting the schema to better match the model's output.
• Consider filing a new issue or updating the existing ones (#15241, #15315) with the current version (0.21.1) and detailed reproduction steps for further assistance.

Example

No specific code example can be provided without more details on the tool schema and model configuration. However, ensuring that the tool calls are properly formatted and that the model is correctly configured to handle these calls is crucial.

Notes

The provided information suggests that the issue might be related to how the model handles tool calls and special tokens, but without direct access to the model's configuration or the exact tool schema used, it's challenging to provide a definitive fix. The issue seems to persist despite previous fixes, indicating a potential regression or a specific scenario not covered by those fixes.

Recommendation

Apply workaround: Since the issue persists in version 0.21.1 and previous fixes do not seem to apply, swapping to a different model family appears to be the most reliable temporary solution until a more permanent fix can be identified and implemented.