Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Compat Matrix] Slice 1: Tracer bullet — one feature, one provider, end-to-end [1 pull requests, 2 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#26477•Fetched 2026-04-26 05:06:52
View on GitHub
Comments
2
Participants
1
Timeline
13
Reactions
0
Author
mateo-berrimateo-berri
Participants
mateo-berrimateo-berri
Timeline (top)
cross-referenced ×7labeled ×3commented ×2closed ×1

Error Message

  • The component renders an error placeholder if schema_version does not match what it supports.

Fix Action

Fixed

PR fix notes

PR #26491: [WIP] feat(tests): Claude Code Compatibility Matrix v0 (PRD #26476)

Description (problem / solution / changelog)

Relevant issues

Implements the v0 of the Claude Code Compatibility Matrix.

  • Parent PRD: #26476
  • Slice 1 (tracer bullet): #26477
  • Slice 2 (4 provider columns for basic_messaging_non_streaming): #26478
  • Slice 3 (PR gate in CircleCI): #26479
  • Slice 4 (daily cron VM publishes matrix to docs): #26480
  • Slice 5 (full v0 row set: 6 features × 5 providers): #26481

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
    • Note: tests for this feature live under tests/claude_code/_driver_unit_tests/, tests/claude_code/_builder_unit_tests/, tests/claude_code/_publisher_unit_tests/, and tests/claude_code/_pr_gate_unit_tests/ — these are deep-module unit tests for the new helpers (Claude Code CLI Driver, Matrix JSON Builder, Publisher, PR-Gate Version Resolver) per the PRD's "Testing Decisions" section. They follow the same mocked-subprocess / golden-file patterns established in tests/test_litellm/.
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run Link:

  • CI run for the last commit Link:

  • Merge / cherry-pick CI run Links:

Screenshots / Proof of Fix

This PR ships the end-to-end v0 of the Claude Code Compatibility Matrix as defined in PRD #26476. Verification of the pipeline:

1. Test scaffolding (slices 1, 2, 5). New layout under tests/claude_code/<feature>/test_<provider>.py. The compat_result pytest fixture captures tagged-union outcomes (pass / fail / not_applicable / not_tested); a conftest.py hook merges per-test results into a structured compat-results.json artifact. Six features × five providers = 30 cells, each exercised against three Claude tiers (Haiku 4.5 / Sonnet 4.6 / Opus 4.7), all-must-pass aggregation per cell.

2. Claude Code CLI Driver + Matrix JSON Builder. Two deep helper modules (tests/claude_code/cli_driver.py, tests/claude_code/matrix_builder.py) wrapping subprocess + parsing and pure-function JSON construction respectively. Unit-tested against mocked subprocess (driver) and golden fixtures (builder) — see _driver_unit_tests/ and _builder_unit_tests/.

3. PR Gate (slice 3). New CircleCI job claude_code_compat_pr_gate boots the proxy from the PR's code, installs the claude CLI at the version returned by the new PR-gate version resolver (newest published >= 3 days ago, queried at run-time from the npm registry), and runs the full tests/claude_code/ suite. Red status blocks merge.

4. Daily Cron Publisher (slice 4). New GitHub Actions workflow .github/workflows/claude_code_compat_matrix.yml runs on three triggers (daily cron, release.published filtered to v*-stable, workflow_dispatch). Resolves the latest stable LiteLLM release via the GitHub Releases API, pulls the corresponding ghcr.io image, installs the latest Claude Code CLI, runs the test suite, builds the matrix JSON, and direct-pushes it to BerriAI/litellm-docs. Cross-repo authentication uses a GitHub App scoped to contents: write on the docs repo only; the select_files_to_commit allowlist enforces "only compatibility-matrix.json is ever pushed" since GitHub Apps cannot scope tokens to a single file path.

5. Sample matrix output. tests/claude_code/sample_compatibility-matrix.json shows the expected v1 schema shape that the docs site's <CompatibilityMatrix /> React component will consume.

Secret scan. Verified no committed secrets:

  • All real credentials are loaded via os.environ.get(...) or ${{ secrets.* }}.
  • Test fixtures use obvious placeholders (sk-test, sk-abc, "k", ghs_xxx).
  • sk-1234 and sk-cron-matrix are dev master keys used only inside ephemeral test/cron containers (consistent with existing CI conventions in .circleci/config.yml).
  • pathrise-convert-1606954137718 is the standard GCP test project ID already used throughout the LiteLLM test suite (a project ID is not a credential).
  • .gitignore excludes the CI-output files (compat-results.json, compatibility-matrix.json).
  • Workflow uses SHA-pinned actions, permissions: contents: read, and persist-credentials: false on checkout.

Type

🆕 New Feature 🚄 Infrastructure ✅ Test

Changes

  • tests/claude_code/manifest.yaml — single source of truth for the matrix's row order and provider column order.
  • tests/claude_code/<feature>/test_<provider>.py — 30 per-(feature, provider) test files, one feature directory each for basic_messaging_non_streaming, basic_messaging_streaming, tool_use, prompt_caching_5m, vision, extended_thinking.
  • tests/claude_code/conftest.py — compat_result fixture and pytest_runtest_logreport hook that emits the structured compat-results.json artifact.
  • tests/claude_code/cli_driver.py — Claude Code CLI Driver (deep module wrapping subprocess + stream-JSON parsing).
  • tests/claude_code/matrix_builder.py — pure-function builder that turns the per-test results artifact into the published compatibility-matrix.json per the v1 schema.
  • tests/claude_code/resolver.py — Latest Stable LiteLLM Resolver (queries the GitHub Releases API for newest v*-stable).
  • tests/claude_code/pr_gate_version_resolver.py — Claude Code PR-Gate Version Resolver (queries npm for newest version published >= 3 days ago).
  • tests/claude_code/publisher.py — daily-cron publisher orchestrator: resolves versions, runs the test suite, builds JSON, direct-pushes to the docs repo. Includes the select_files_to_commit allowlist enforcement.
  • tests/claude_code/test_config.yaml — proxy routing config for the PR gate, mapping aliases to upstream models per provider.
  • tests/claude_code/_*_unit_tests/ — unit tests for the four deep modules.
  • .github/workflows/claude_code_compat_matrix.yml — daily cron workflow.
  • .circleci/config.yml — new claude_code_compat_pr_gate job wired into the existing main-branches workflow.
  • .gitignore — exclude CI-output files (compat-results.json, compatibility-matrix.json).

Out of scope for this PR (per PRD's "Deferred to v1+"): the docs-side React <CompatibilityMatrix /> component, MDX page at docs/tutorials/claude-code-compatibility, Slack regression alerts, operational guardrails (deadman alerts, staleness banner), additional features beyond the v0 row set, PR-comment diff commenter, click-to-modal cell deep-dive, and a written ADR artifact.

Changed files

  • .circleci/config.yml (modified, +96/-0)
  • .github/workflows/claude_code_compat_matrix.yml (added, +127/-0)
  • .gitignore (modified, +6/-1)
  • tests/claude_code/__init__.py (added, +0/-0)
  • tests/claude_code/_builder_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/expected_matrix.json (added, +38/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/manifest.yaml (added, +9/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/results.json (added, +41/-0)
  • tests/claude_code/_builder_unit_tests/test_matrix_builder.py (added, +282/-0)
  • tests/claude_code/_builder_unit_tests/test_v0_layout.py (added, +116/-0)
  • tests/claude_code/_driver_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_driver_unit_tests/test_cli_driver.py (added, +339/-0)
  • tests/claude_code/_driver_unit_tests/test_compat_result.py (added, +70/-0)
  • tests/claude_code/_pr_gate_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py (added, +131/-0)
  • tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py (added, +139/-0)
  • tests/claude_code/_publisher_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_publisher_unit_tests/test_publisher.py (added, +99/-0)
  • tests/claude_code/_publisher_unit_tests/test_resolver.py (added, +112/-0)
  • tests/claude_code/basic_messaging_non_streaming/__init__.py (added, +0/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_anthropic.py (added, +103/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_azure.py (added, +107/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py (added, +96/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py (added, +96/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py (added, +96/-0)
  • tests/claude_code/basic_messaging_streaming/__init__.py (added, +0/-0)
  • tests/claude_code/basic_messaging_streaming/test_anthropic.py (added, +106/-0)
  • tests/claude_code/basic_messaging_streaming/test_azure.py (added, +103/-0)
  • tests/claude_code/basic_messaging_streaming/test_bedrock_converse.py (added, +99/-0)
  • tests/claude_code/basic_messaging_streaming/test_bedrock_invoke.py (added, +99/-0)
  • tests/claude_code/basic_messaging_streaming/test_vertex_ai.py (added, +99/-0)
  • tests/claude_code/cli_driver.py (added, +261/-0)
  • tests/claude_code/conftest.py (added, +164/-0)
  • tests/claude_code/extended_thinking/__init__.py (added, +0/-0)
  • tests/claude_code/extended_thinking/test_anthropic.py (added, +116/-0)
  • tests/claude_code/extended_thinking/test_azure.py (added, +118/-0)
  • tests/claude_code/extended_thinking/test_bedrock_converse.py (added, +110/-0)
  • tests/claude_code/extended_thinking/test_bedrock_invoke.py (added, +110/-0)
  • tests/claude_code/extended_thinking/test_vertex_ai.py (added, +110/-0)
  • tests/claude_code/manifest.yaml (added, +36/-0)
  • tests/claude_code/matrix_builder.py (added, +179/-0)
  • tests/claude_code/pr_gate_version_resolver.py (added, +148/-0)
  • tests/claude_code/prompt_caching_5m/__init__.py (added, +0/-0)
  • tests/claude_code/prompt_caching_5m/test_anthropic.py (added, +113/-0)
  • tests/claude_code/prompt_caching_5m/test_azure.py (added, +111/-0)
  • tests/claude_code/prompt_caching_5m/test_bedrock_converse.py (added, +104/-0)
  • tests/claude_code/prompt_caching_5m/test_bedrock_invoke.py (added, +104/-0)
  • tests/claude_code/prompt_caching_5m/test_vertex_ai.py (added, +104/-0)
  • tests/claude_code/publisher.py (added, +388/-0)
  • tests/claude_code/resolver.py (added, +91/-0)
  • tests/claude_code/sample_compatibility-matrix.json (added, +141/-0)
  • tests/claude_code/test_config.yaml (added, +103/-0)
  • tests/claude_code/tool_use/__init__.py (added, +0/-0)
  • tests/claude_code/tool_use/test_anthropic.py (added, +114/-0)
  • tests/claude_code/tool_use/test_azure.py (added, +113/-0)
  • tests/claude_code/tool_use/test_bedrock_converse.py (added, +109/-0)
  • tests/claude_code/tool_use/test_bedrock_invoke.py (added, +109/-0)
  • tests/claude_code/tool_use/test_vertex_ai.py (added, +109/-0)
  • tests/claude_code/vision/__init__.py (added, +0/-0)
  • tests/claude_code/vision/test_anthropic.py (added, +99/-0)
  • tests/claude_code/vision/test_azure.py (added, +100/-0)
  • tests/claude_code/vision/test_bedrock_converse.py (added, +95/-0)
  • tests/claude_code/vision/test_bedrock_invoke.py (added, +95/-0)
  • tests/claude_code/vision/test_vertex_ai.py (added, +95/-0)
RAW_BUFFERClick to expand / collapse

Parent PRD

#26476

What to build

The first vertical slice. Build the thinnest possible end-to-end path through every layer for a single (feature, provider) cell, so that visiting the compatibility matrix page renders a real green cell sourced from a real test.

Pick basic_messaging_non_streaming on Anthropic as the cell. Build:

  • The tests/claude_code/ directory structure with manifest.yaml containing one feature entry.
  • One per-provider test file that drives the real claude CLI in headless mode and reports its outcome via the new compat_result pytest fixture.
  • The compat_result fixture and supporting conftest.py plumbing that captures tagged-union outcomes and emits the structured pytest-results artifact described in the PRD's "Test code organization" section.
  • The Claude Code CLI Driver helper module (only as deep as this one test requires; subsequent slices will exercise more of its surface).
  • The Matrix JSON Builder module (only as deep as required to handle one feature, one provider).
  • A compatibility-matrix.json checked into the docs repo by hand for now (the publishing pipeline is a later slice).
  • The MDX page at docs/tutorials/claude-code-compatibility and the <CompatibilityMatrix /> React component that renders the JSON: colored cells with hover tooltips, version + last-updated banner with relative/absolute timestamp logic in the user's local timezone, schema-version check.

This is the architectural review point. Once approved, subsequent slices fill in providers, features, CI, and the publishing pipeline mechanically.

Acceptance criteria

  • tests/claude_code/manifest.yaml exists with a single feature entry mapping basic_messaging_non_streaming to its display name.
  • tests/claude_code/basic_messaging_non_streaming/test_anthropic.py runs, drives the real claude CLI against a running LiteLLM proxy, and reports a result via the compat_result fixture.
  • The compat_result fixture accepts the four tagged-union variants from the PRD (pass, fail, not_applicable, not_tested) and the conftest hook emits a structured results artifact next to JUnit XML.
  • The Claude Code CLI Driver module exposes a single entry point that handles subprocess invocation, stream-JSON parsing for the non-streaming case, and structured result return.
  • The Matrix JSON Builder module produces a compatibility-matrix.json conforming to the schema in the PRD's "JSON schema (v1)" section, with schema_version set to "1".
  • A hand-authored compatibility-matrix.json is checked into the docs repo containing the one cell at pass.
  • An MDX page at docs/tutorials/claude-code-compatibility renders the <CompatibilityMatrix /> component.
  • The component imports the JSON at build time (no runtime network calls) and renders the cell with a green background and check icon.
  • The component renders the version banner with relative time when recent and absolute time (in the user's local timezone, explicitly labeled in the tooltip) when older, plus the LiteLLM and Claude Code version strings.
  • The component renders an error placeholder if schema_version does not match what it supports.
  • On a narrow viewport the matrix scrolls horizontally rather than reflowing.

Blocked by

None - can start immediately.

User stories addressed

Reference by number from the parent PRD:

  • User story 1
  • User story 2
  • User story 4
  • User story 5
  • User story 18
  • User story 19
  • User story 21
  • User story 24
  • User story 25
  • User story 26
  • User story 27
  • User story 28
  • User story 29
  • User story 30

extent analysis

TL;DR

Implement the compat_result fixture and supporting conftest.py plumbing to capture tagged-union outcomes and emit the structured pytest-results artifact.

Guidance

  • Review the PRD's "Test code organization" section to understand the required structure of the pytest-results artifact.
  • Implement the compat_result fixture to accept the four tagged-union variants (pass, fail, not_applicable, not_tested) and emit the structured results artifact.
  • Ensure the conftest.py hook is properly configured to capture the outcomes and emit the artifact next to JUnit XML.
  • Verify that the compatibility-matrix.json produced by the Matrix JSON Builder module conforms to the schema in the PRD's "JSON schema (v1)" section.

Notes

The implementation details of the compat_result fixture and conftest.py plumbing are not provided, so the above guidance is based on the acceptance criteria and PRD references.

Recommendation

Apply workaround by manually implementing the compat_result fixture and conftest.py plumbing according to the PRD's specifications, as the issue lacks enough information to provide a direct solution.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#request error#file not found#serialization error#model compatibility#GPU setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING