Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [Bug]: Complex tool json schemas not dereferenced before passing to chat templates [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#39108•Fetched 2026-04-08 03:01:54
View on GitHub
Comments
0
Participants
1
Timeline
6
Reactions
2
Author
bbrowningbbrowning
Participants
bbrowningbbrowning
Timeline (top)
cross-referenced ×2referenced ×2labeled ×1subscribed ×1

Fix Action

Fixed

PR fix notes

PR #39147: fix: dereference $ref in tool schemas before passing to chat templates

Description (problem / solution / changelog)

Fixes #39108

Problem

Tool schemas with complex JSON features like $ref, $defs, and anyOf are not dereferenced before being passed to chat templates. Jinja templates cannot resolve $ref references, so models don't see the referenced properties. This causes models to fail when tools use shared definitions.

Fix

Use jsonref.replace_refs() to flatten all tool schemas before passing them to apply_chat_template(). This dereferences all $ref and resolves allOf/anyOf inline.

Example

Before:

{"properties": {"address": {"$ref": "#/$defs/address"}}}

After:

{"properties": {"address": {"type": "object", "properties": {"street": {...}, "city": {...}}}}

Changes

  • vllm/renderers/hf.py: 10 lines added in safe_apply_chat_template

Testing

  • Syntax validation passes
  • jsonref is already a dependency of vllm (used elsewhere)

Changed files

  • vllm/renderers/hf.py (modified, +17/-0)
RAW_BUFFERClick to expand / collapse

Your current environment

local testing from main

🐛 Describe the bug

When users are using complex JSON schemas in their tool definitions with features like anyOf, $ref, $def, and so on we're not deferencing those $ref nor are we flattening any of the anyOf things into lists before passing to chat templates. The net result of this is most models do not actually see those referenced properties or anyOf types in their rendered chat templates, as the jinja templates can't easily dereference these things.

See https://github.com/vllm-project/vllm/issues/39089#issuecomment-4194272963 for an example where I noticed this as part of a separate user-reported bug.

Potential options here include using the jsonref library to flatten all function tool call properties before we pass them to tokenizers, such as within our OpenAIServingRender class and its render_chat method.

Likewise, we should probably flatten anyOf types to type lists, and exclude nullable there as that's already handled via the required field on function definition.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

Using the jsonref library to flatten JSON schemas with $ref and anyOf features may resolve the issue of referenced properties not being visible in rendered chat templates.

Guidance

  • Investigate the jsonref library as a potential solution to flatten JSON schemas before passing them to tokenizers.
  • Consider modifying the OpenAIServingRender class, specifically the render_chat method, to incorporate the jsonref library.
  • Flatten anyOf types to type lists, excluding nullable types as they are already handled via the required field.
  • Review the example provided in https://github.com/vllm-project/vllm/issues/39089#issuecomment-4194272963 to understand the issue's impact.

Example

No code snippet is provided due to the lack of specific implementation details in the issue.

Notes

The solution may require additional modifications to handle edge cases or specific requirements of the OpenAIServingRender class and the render_chat method.

Recommendation

Apply a workaround using the jsonref library to flatten JSON schemas, as it appears to be a promising solution to the issue at hand.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#integration issue#index setup#retrieval issue#search optimization#API routing

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

vllm - ✅(Solved) Fix [Bug]: Complex tool json schemas not dereferenced before passing to chat templates [1 pull requests, 1 participants]