Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Compat Matrix] Slice 3: Wire PR gate in CircleCI [1 pull requests, 2 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#26479•Fetched 2026-04-26 05:06:48
View on GitHub
Comments
2
Participants
1
Timeline
11
Reactions
0
Author
mateo-berrimateo-berri
Participants
mateo-berrimateo-berri
Timeline (top)
cross-referenced ×5labeled ×3commented ×2closed ×1

Fix Action

Fixed

PR fix notes

PR #26491: [WIP] feat(tests): Claude Code Compatibility Matrix v0 (PRD #26476)

Description (problem / solution / changelog)

Relevant issues

Implements the v0 of the Claude Code Compatibility Matrix.

  • Parent PRD: #26476
  • Slice 1 (tracer bullet): #26477
  • Slice 2 (4 provider columns for basic_messaging_non_streaming): #26478
  • Slice 3 (PR gate in CircleCI): #26479
  • Slice 4 (daily cron VM publishes matrix to docs): #26480
  • Slice 5 (full v0 row set: 6 features × 5 providers): #26481

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
    • Note: tests for this feature live under tests/claude_code/_driver_unit_tests/, tests/claude_code/_builder_unit_tests/, tests/claude_code/_publisher_unit_tests/, and tests/claude_code/_pr_gate_unit_tests/ — these are deep-module unit tests for the new helpers (Claude Code CLI Driver, Matrix JSON Builder, Publisher, PR-Gate Version Resolver) per the PRD's "Testing Decisions" section. They follow the same mocked-subprocess / golden-file patterns established in tests/test_litellm/.
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run Link:

  • CI run for the last commit Link:

  • Merge / cherry-pick CI run Links:

Screenshots / Proof of Fix

This PR ships the end-to-end v0 of the Claude Code Compatibility Matrix as defined in PRD #26476. Verification of the pipeline:

1. Test scaffolding (slices 1, 2, 5). New layout under tests/claude_code/<feature>/test_<provider>.py. The compat_result pytest fixture captures tagged-union outcomes (pass / fail / not_applicable / not_tested); a conftest.py hook merges per-test results into a structured compat-results.json artifact. Six features × five providers = 30 cells, each exercised against three Claude tiers (Haiku 4.5 / Sonnet 4.6 / Opus 4.7), all-must-pass aggregation per cell.

2. Claude Code CLI Driver + Matrix JSON Builder. Two deep helper modules (tests/claude_code/cli_driver.py, tests/claude_code/matrix_builder.py) wrapping subprocess + parsing and pure-function JSON construction respectively. Unit-tested against mocked subprocess (driver) and golden fixtures (builder) — see _driver_unit_tests/ and _builder_unit_tests/.

3. PR Gate (slice 3). New CircleCI job claude_code_compat_pr_gate boots the proxy from the PR's code, installs the claude CLI at the version returned by the new PR-gate version resolver (newest published >= 3 days ago, queried at run-time from the npm registry), and runs the full tests/claude_code/ suite. Red status blocks merge.

4. Daily Cron Publisher (slice 4). New GitHub Actions workflow .github/workflows/claude_code_compat_matrix.yml runs on three triggers (daily cron, release.published filtered to v*-stable, workflow_dispatch). Resolves the latest stable LiteLLM release via the GitHub Releases API, pulls the corresponding ghcr.io image, installs the latest Claude Code CLI, runs the test suite, builds the matrix JSON, and direct-pushes it to BerriAI/litellm-docs. Cross-repo authentication uses a GitHub App scoped to contents: write on the docs repo only; the select_files_to_commit allowlist enforces "only compatibility-matrix.json is ever pushed" since GitHub Apps cannot scope tokens to a single file path.

5. Sample matrix output. tests/claude_code/sample_compatibility-matrix.json shows the expected v1 schema shape that the docs site's <CompatibilityMatrix /> React component will consume.

Secret scan. Verified no committed secrets:

  • All real credentials are loaded via os.environ.get(...) or ${{ secrets.* }}.
  • Test fixtures use obvious placeholders (sk-test, sk-abc, "k", ghs_xxx).
  • sk-1234 and sk-cron-matrix are dev master keys used only inside ephemeral test/cron containers (consistent with existing CI conventions in .circleci/config.yml).
  • pathrise-convert-1606954137718 is the standard GCP test project ID already used throughout the LiteLLM test suite (a project ID is not a credential).
  • .gitignore excludes the CI-output files (compat-results.json, compatibility-matrix.json).
  • Workflow uses SHA-pinned actions, permissions: contents: read, and persist-credentials: false on checkout.

Type

🆕 New Feature 🚄 Infrastructure ✅ Test

Changes

  • tests/claude_code/manifest.yaml — single source of truth for the matrix's row order and provider column order.
  • tests/claude_code/<feature>/test_<provider>.py — 30 per-(feature, provider) test files, one feature directory each for basic_messaging_non_streaming, basic_messaging_streaming, tool_use, prompt_caching_5m, vision, extended_thinking.
  • tests/claude_code/conftest.py — compat_result fixture and pytest_runtest_logreport hook that emits the structured compat-results.json artifact.
  • tests/claude_code/cli_driver.py — Claude Code CLI Driver (deep module wrapping subprocess + stream-JSON parsing).
  • tests/claude_code/matrix_builder.py — pure-function builder that turns the per-test results artifact into the published compatibility-matrix.json per the v1 schema.
  • tests/claude_code/resolver.py — Latest Stable LiteLLM Resolver (queries the GitHub Releases API for newest v*-stable).
  • tests/claude_code/pr_gate_version_resolver.py — Claude Code PR-Gate Version Resolver (queries npm for newest version published >= 3 days ago).
  • tests/claude_code/publisher.py — daily-cron publisher orchestrator: resolves versions, runs the test suite, builds JSON, direct-pushes to the docs repo. Includes the select_files_to_commit allowlist enforcement.
  • tests/claude_code/test_config.yaml — proxy routing config for the PR gate, mapping aliases to upstream models per provider.
  • tests/claude_code/_*_unit_tests/ — unit tests for the four deep modules.
  • .github/workflows/claude_code_compat_matrix.yml — daily cron workflow.
  • .circleci/config.yml — new claude_code_compat_pr_gate job wired into the existing main-branches workflow.
  • .gitignore — exclude CI-output files (compat-results.json, compatibility-matrix.json).

Out of scope for this PR (per PRD's "Deferred to v1+"): the docs-side React <CompatibilityMatrix /> component, MDX page at docs/tutorials/claude-code-compatibility, Slack regression alerts, operational guardrails (deadman alerts, staleness banner), additional features beyond the v0 row set, PR-comment diff commenter, click-to-modal cell deep-dive, and a written ADR artifact.

Changed files

  • .circleci/config.yml (modified, +96/-0)
  • .github/workflows/claude_code_compat_matrix.yml (added, +127/-0)
  • .gitignore (modified, +6/-1)
  • tests/claude_code/__init__.py (added, +0/-0)
  • tests/claude_code/_builder_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/expected_matrix.json (added, +38/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/manifest.yaml (added, +9/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/results.json (added, +41/-0)
  • tests/claude_code/_builder_unit_tests/test_matrix_builder.py (added, +282/-0)
  • tests/claude_code/_builder_unit_tests/test_v0_layout.py (added, +116/-0)
  • tests/claude_code/_driver_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_driver_unit_tests/test_cli_driver.py (added, +339/-0)
  • tests/claude_code/_driver_unit_tests/test_compat_result.py (added, +70/-0)
  • tests/claude_code/_pr_gate_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py (added, +131/-0)
  • tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py (added, +139/-0)
  • tests/claude_code/_publisher_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_publisher_unit_tests/test_publisher.py (added, +99/-0)
  • tests/claude_code/_publisher_unit_tests/test_resolver.py (added, +112/-0)
  • tests/claude_code/basic_messaging_non_streaming/__init__.py (added, +0/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_anthropic.py (added, +103/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_azure.py (added, +107/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py (added, +96/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py (added, +96/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py (added, +96/-0)
  • tests/claude_code/basic_messaging_streaming/__init__.py (added, +0/-0)
  • tests/claude_code/basic_messaging_streaming/test_anthropic.py (added, +106/-0)
  • tests/claude_code/basic_messaging_streaming/test_azure.py (added, +103/-0)
  • tests/claude_code/basic_messaging_streaming/test_bedrock_converse.py (added, +99/-0)
  • tests/claude_code/basic_messaging_streaming/test_bedrock_invoke.py (added, +99/-0)
  • tests/claude_code/basic_messaging_streaming/test_vertex_ai.py (added, +99/-0)
  • tests/claude_code/cli_driver.py (added, +261/-0)
  • tests/claude_code/conftest.py (added, +164/-0)
  • tests/claude_code/extended_thinking/__init__.py (added, +0/-0)
  • tests/claude_code/extended_thinking/test_anthropic.py (added, +116/-0)
  • tests/claude_code/extended_thinking/test_azure.py (added, +118/-0)
  • tests/claude_code/extended_thinking/test_bedrock_converse.py (added, +110/-0)
  • tests/claude_code/extended_thinking/test_bedrock_invoke.py (added, +110/-0)
  • tests/claude_code/extended_thinking/test_vertex_ai.py (added, +110/-0)
  • tests/claude_code/manifest.yaml (added, +36/-0)
  • tests/claude_code/matrix_builder.py (added, +179/-0)
  • tests/claude_code/pr_gate_version_resolver.py (added, +148/-0)
  • tests/claude_code/prompt_caching_5m/__init__.py (added, +0/-0)
  • tests/claude_code/prompt_caching_5m/test_anthropic.py (added, +113/-0)
  • tests/claude_code/prompt_caching_5m/test_azure.py (added, +111/-0)
  • tests/claude_code/prompt_caching_5m/test_bedrock_converse.py (added, +104/-0)
  • tests/claude_code/prompt_caching_5m/test_bedrock_invoke.py (added, +104/-0)
  • tests/claude_code/prompt_caching_5m/test_vertex_ai.py (added, +104/-0)
  • tests/claude_code/publisher.py (added, +388/-0)
  • tests/claude_code/resolver.py (added, +91/-0)
  • tests/claude_code/sample_compatibility-matrix.json (added, +141/-0)
  • tests/claude_code/test_config.yaml (added, +103/-0)
  • tests/claude_code/tool_use/__init__.py (added, +0/-0)
  • tests/claude_code/tool_use/test_anthropic.py (added, +114/-0)
  • tests/claude_code/tool_use/test_azure.py (added, +113/-0)
  • tests/claude_code/tool_use/test_bedrock_converse.py (added, +109/-0)
  • tests/claude_code/tool_use/test_bedrock_invoke.py (added, +109/-0)
  • tests/claude_code/tool_use/test_vertex_ai.py (added, +109/-0)
  • tests/claude_code/vision/__init__.py (added, +0/-0)
  • tests/claude_code/vision/test_anthropic.py (added, +99/-0)
  • tests/claude_code/vision/test_azure.py (added, +100/-0)
  • tests/claude_code/vision/test_bedrock_converse.py (added, +95/-0)
  • tests/claude_code/vision/test_bedrock_invoke.py (added, +95/-0)
  • tests/claude_code/vision/test_vertex_ai.py (added, +95/-0)
RAW_BUFFERClick to expand / collapse

Parent PRD

#26476

What to build

Wire the compatibility tests into CircleCI as a pre-merge gate. The PR gate runs the full tests/claude_code/ suite on every PR and blocks merge when any test fails.

This slice introduces the Claude Code PR-Gate Version Resolver (described in the PRD's "Version resolvers" section): at CI-run time, it queries the npm registry for @anthropic-ai/claude-code and selects the newest version whose publish timestamp is at least 3 days old. The Claude Code CLI is installed at that version inside the CI environment.

The CI job boots a LiteLLM proxy from the PR's code, runs the compatibility test suite against it, and reports pass/fail as a standard CircleCI status check. No custom PR comment in v0 — just the standard required-status-check pattern.

Acceptance criteria

  • A new CircleCI job runs on every PR into the staging branch and exercises the compatibility test suite at tests/claude_code/.
  • The job installs the Claude Code CLI at a version computed at run time from the npm registry, selecting the newest version published at least 3 days ago.
  • The job boots a LiteLLM proxy from the PR's code and runs the tests against it.
  • The job's status check is configured as a required check on the staging branch so that a red status blocks merge.
  • When a test in tests/claude_code/ is intentionally broken in a PR, the gate goes red and merge is blocked.
  • The selected Claude Code version is logged in the CI output so engineers can see which version produced a result.
  • No regression to the existing proxy_e2e_anthropic_messages_tests job behavior.

Blocked by

  • Blocked by #26478

User stories addressed

Reference by number from the parent PRD:

  • User story 10
  • User story 11
  • User story 12

extent analysis

TL;DR

Implement a CircleCI job that runs the compatibility test suite with a dynamically selected version of the Claude Code CLI, based on the newest version published at least 3 days ago, to block merges on failed tests.

Guidance

  • Configure a new CircleCI job to run on every PR into the staging branch, exercising the compatibility test suite at tests/claude_code/.
  • Use the npm registry to select the newest version of @anthropic-ai/claude-code with a publish timestamp at least 3 days old, and install the Claude Code CLI at that version.
  • Boot a LiteLLM proxy from the PR's code and run the tests against it, reporting pass/fail as a standard CircleCI status check.
  • Configure the job's status check as a required check on the staging branch to block merge on failed tests.

Notes

This implementation relies on the CircleCI environment and the npm registry, and assumes that the compatibility test suite is correctly configured to run against the LiteLLM proxy.

Recommendation

Apply workaround by implementing the CircleCI job as described, to ensure that merges are blocked when compatibility tests fail.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#ISR setup#authentication setup#request error#file not found#serialization error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING