Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Compat Matrix] Slice 4: Daily cron VM publishes the matrix to docs [1 pull requests, 1 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#26480•Fetched 2026-04-26 05:06:47
View on GitHub
Comments
1
Participants
1
Timeline
9
Reactions
0
Author
mateo-berrimateo-berri
Participants
mateo-berrimateo-berri
Timeline (top)
cross-referenced ×5labeled ×2closed ×1commented ×1

Fix Action

Fix / Workaround

  • Provisioning the isolated VM in the LiteLLM repo's infrastructure (kept outside the main CI environment per the PRD's "Two CI environments" section).

  • Creating the GitHub App installed on the docs repo only, scoped to contents: write, and configuring the VM with its installation credentials.

  • Implementing the Latest Stable LiteLLM Resolver described in the PRD's "Version resolvers" section: queries the GitHub Releases API for BerriAI/litellm and returns the newest tag matching v*-stable.

  • Implementing the matrix publishing script that: resolves the latest stable LiteLLM tag, pulls the corresponding Docker image, installs the latest Claude Code CLI, runs the full compatibility test suite, invokes the Matrix JSON Builder to produce compatibility-matrix.json, and direct-pushes the file to the docs repo's main branch.

  • Configuring three triggers: GitHub Release published (auto-fires on new stable releases), daily cron, and manual dispatch.

  • An isolated VM is provisioned and runs a daily cron job for the matrix pipeline.

  • A GitHub App scoped to contents: write on the docs repo only is created, installed, and its credentials provided to the VM via secret management.

  • The Latest Stable LiteLLM Resolver returns the newest tag matching v*-stable from the GitHub Releases API.

  • The publishing script pulls the Docker image for the resolved stable tag, runs it as the proxy, installs the latest Claude Code CLI, runs tests/claude_code/ against the proxy, and produces compatibility-matrix.json via the Matrix JSON Builder.

  • The script direct-pushes only compatibility-matrix.json to the docs repo's main branch using the GitHub App token; the script does not write any other files.

  • The pipeline is triggered by all three trigger sources: published *-stable releases, the daily cron, and manual dispatch.

  • After this slice lands, the JSON in the docs repo is regenerated by the cron rather than hand-authored.

  • The docs page banner correctly shows the LiteLLM version, Claude Code version, and generated_at timestamp produced by the pipeline.

  • The hand-authored matrix JSON from earlier slices is removed and replaced by the auto-generated output.

PR fix notes

PR #26491: [WIP] feat(tests): Claude Code Compatibility Matrix v0 (PRD #26476)

Description (problem / solution / changelog)

Relevant issues

Implements the v0 of the Claude Code Compatibility Matrix.

  • Parent PRD: #26476
  • Slice 1 (tracer bullet): #26477
  • Slice 2 (4 provider columns for basic_messaging_non_streaming): #26478
  • Slice 3 (PR gate in CircleCI): #26479
  • Slice 4 (daily cron VM publishes matrix to docs): #26480
  • Slice 5 (full v0 row set: 6 features × 5 providers): #26481

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
    • Note: tests for this feature live under tests/claude_code/_driver_unit_tests/, tests/claude_code/_builder_unit_tests/, tests/claude_code/_publisher_unit_tests/, and tests/claude_code/_pr_gate_unit_tests/ — these are deep-module unit tests for the new helpers (Claude Code CLI Driver, Matrix JSON Builder, Publisher, PR-Gate Version Resolver) per the PRD's "Testing Decisions" section. They follow the same mocked-subprocess / golden-file patterns established in tests/test_litellm/.
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.

  • Branch creation CI run Link:

  • CI run for the last commit Link:

  • Merge / cherry-pick CI run Links:

Screenshots / Proof of Fix

This PR ships the end-to-end v0 of the Claude Code Compatibility Matrix as defined in PRD #26476. Verification of the pipeline:

1. Test scaffolding (slices 1, 2, 5). New layout under tests/claude_code/<feature>/test_<provider>.py. The compat_result pytest fixture captures tagged-union outcomes (pass / fail / not_applicable / not_tested); a conftest.py hook merges per-test results into a structured compat-results.json artifact. Six features × five providers = 30 cells, each exercised against three Claude tiers (Haiku 4.5 / Sonnet 4.6 / Opus 4.7), all-must-pass aggregation per cell.

2. Claude Code CLI Driver + Matrix JSON Builder. Two deep helper modules (tests/claude_code/cli_driver.py, tests/claude_code/matrix_builder.py) wrapping subprocess + parsing and pure-function JSON construction respectively. Unit-tested against mocked subprocess (driver) and golden fixtures (builder) — see _driver_unit_tests/ and _builder_unit_tests/.

3. PR Gate (slice 3). New CircleCI job claude_code_compat_pr_gate boots the proxy from the PR's code, installs the claude CLI at the version returned by the new PR-gate version resolver (newest published >= 3 days ago, queried at run-time from the npm registry), and runs the full tests/claude_code/ suite. Red status blocks merge.

4. Daily Cron Publisher (slice 4). New GitHub Actions workflow .github/workflows/claude_code_compat_matrix.yml runs on three triggers (daily cron, release.published filtered to v*-stable, workflow_dispatch). Resolves the latest stable LiteLLM release via the GitHub Releases API, pulls the corresponding ghcr.io image, installs the latest Claude Code CLI, runs the test suite, builds the matrix JSON, and direct-pushes it to BerriAI/litellm-docs. Cross-repo authentication uses a GitHub App scoped to contents: write on the docs repo only; the select_files_to_commit allowlist enforces "only compatibility-matrix.json is ever pushed" since GitHub Apps cannot scope tokens to a single file path.

5. Sample matrix output. tests/claude_code/sample_compatibility-matrix.json shows the expected v1 schema shape that the docs site's <CompatibilityMatrix /> React component will consume.

Secret scan. Verified no committed secrets:

  • All real credentials are loaded via os.environ.get(...) or ${{ secrets.* }}.
  • Test fixtures use obvious placeholders (sk-test, sk-abc, "k", ghs_xxx).
  • sk-1234 and sk-cron-matrix are dev master keys used only inside ephemeral test/cron containers (consistent with existing CI conventions in .circleci/config.yml).
  • pathrise-convert-1606954137718 is the standard GCP test project ID already used throughout the LiteLLM test suite (a project ID is not a credential).
  • .gitignore excludes the CI-output files (compat-results.json, compatibility-matrix.json).
  • Workflow uses SHA-pinned actions, permissions: contents: read, and persist-credentials: false on checkout.

Type

🆕 New Feature 🚄 Infrastructure ✅ Test

Changes

  • tests/claude_code/manifest.yaml — single source of truth for the matrix's row order and provider column order.
  • tests/claude_code/<feature>/test_<provider>.py — 30 per-(feature, provider) test files, one feature directory each for basic_messaging_non_streaming, basic_messaging_streaming, tool_use, prompt_caching_5m, vision, extended_thinking.
  • tests/claude_code/conftest.py — compat_result fixture and pytest_runtest_logreport hook that emits the structured compat-results.json artifact.
  • tests/claude_code/cli_driver.py — Claude Code CLI Driver (deep module wrapping subprocess + stream-JSON parsing).
  • tests/claude_code/matrix_builder.py — pure-function builder that turns the per-test results artifact into the published compatibility-matrix.json per the v1 schema.
  • tests/claude_code/resolver.py — Latest Stable LiteLLM Resolver (queries the GitHub Releases API for newest v*-stable).
  • tests/claude_code/pr_gate_version_resolver.py — Claude Code PR-Gate Version Resolver (queries npm for newest version published >= 3 days ago).
  • tests/claude_code/publisher.py — daily-cron publisher orchestrator: resolves versions, runs the test suite, builds JSON, direct-pushes to the docs repo. Includes the select_files_to_commit allowlist enforcement.
  • tests/claude_code/test_config.yaml — proxy routing config for the PR gate, mapping aliases to upstream models per provider.
  • tests/claude_code/_*_unit_tests/ — unit tests for the four deep modules.
  • .github/workflows/claude_code_compat_matrix.yml — daily cron workflow.
  • .circleci/config.yml — new claude_code_compat_pr_gate job wired into the existing main-branches workflow.
  • .gitignore — exclude CI-output files (compat-results.json, compatibility-matrix.json).

Out of scope for this PR (per PRD's "Deferred to v1+"): the docs-side React <CompatibilityMatrix /> component, MDX page at docs/tutorials/claude-code-compatibility, Slack regression alerts, operational guardrails (deadman alerts, staleness banner), additional features beyond the v0 row set, PR-comment diff commenter, click-to-modal cell deep-dive, and a written ADR artifact.

Changed files

  • .circleci/config.yml (modified, +96/-0)
  • .github/workflows/claude_code_compat_matrix.yml (added, +127/-0)
  • .gitignore (modified, +6/-1)
  • tests/claude_code/__init__.py (added, +0/-0)
  • tests/claude_code/_builder_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/expected_matrix.json (added, +38/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/manifest.yaml (added, +9/-0)
  • tests/claude_code/_builder_unit_tests/fixtures/results.json (added, +41/-0)
  • tests/claude_code/_builder_unit_tests/test_matrix_builder.py (added, +282/-0)
  • tests/claude_code/_builder_unit_tests/test_v0_layout.py (added, +116/-0)
  • tests/claude_code/_driver_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_driver_unit_tests/test_cli_driver.py (added, +339/-0)
  • tests/claude_code/_driver_unit_tests/test_compat_result.py (added, +70/-0)
  • tests/claude_code/_pr_gate_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_pr_gate_unit_tests/test_circleci_pr_gate_wiring.py (added, +131/-0)
  • tests/claude_code/_pr_gate_unit_tests/test_pr_gate_version_resolver.py (added, +139/-0)
  • tests/claude_code/_publisher_unit_tests/__init__.py (added, +0/-0)
  • tests/claude_code/_publisher_unit_tests/test_publisher.py (added, +99/-0)
  • tests/claude_code/_publisher_unit_tests/test_resolver.py (added, +112/-0)
  • tests/claude_code/basic_messaging_non_streaming/__init__.py (added, +0/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_anthropic.py (added, +103/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_azure.py (added, +107/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_bedrock_converse.py (added, +96/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_bedrock_invoke.py (added, +96/-0)
  • tests/claude_code/basic_messaging_non_streaming/test_vertex_ai.py (added, +96/-0)
  • tests/claude_code/basic_messaging_streaming/__init__.py (added, +0/-0)
  • tests/claude_code/basic_messaging_streaming/test_anthropic.py (added, +106/-0)
  • tests/claude_code/basic_messaging_streaming/test_azure.py (added, +103/-0)
  • tests/claude_code/basic_messaging_streaming/test_bedrock_converse.py (added, +99/-0)
  • tests/claude_code/basic_messaging_streaming/test_bedrock_invoke.py (added, +99/-0)
  • tests/claude_code/basic_messaging_streaming/test_vertex_ai.py (added, +99/-0)
  • tests/claude_code/cli_driver.py (added, +261/-0)
  • tests/claude_code/conftest.py (added, +164/-0)
  • tests/claude_code/extended_thinking/__init__.py (added, +0/-0)
  • tests/claude_code/extended_thinking/test_anthropic.py (added, +116/-0)
  • tests/claude_code/extended_thinking/test_azure.py (added, +118/-0)
  • tests/claude_code/extended_thinking/test_bedrock_converse.py (added, +110/-0)
  • tests/claude_code/extended_thinking/test_bedrock_invoke.py (added, +110/-0)
  • tests/claude_code/extended_thinking/test_vertex_ai.py (added, +110/-0)
  • tests/claude_code/manifest.yaml (added, +36/-0)
  • tests/claude_code/matrix_builder.py (added, +179/-0)
  • tests/claude_code/pr_gate_version_resolver.py (added, +148/-0)
  • tests/claude_code/prompt_caching_5m/__init__.py (added, +0/-0)
  • tests/claude_code/prompt_caching_5m/test_anthropic.py (added, +113/-0)
  • tests/claude_code/prompt_caching_5m/test_azure.py (added, +111/-0)
  • tests/claude_code/prompt_caching_5m/test_bedrock_converse.py (added, +104/-0)
  • tests/claude_code/prompt_caching_5m/test_bedrock_invoke.py (added, +104/-0)
  • tests/claude_code/prompt_caching_5m/test_vertex_ai.py (added, +104/-0)
  • tests/claude_code/publisher.py (added, +388/-0)
  • tests/claude_code/resolver.py (added, +91/-0)
  • tests/claude_code/sample_compatibility-matrix.json (added, +141/-0)
  • tests/claude_code/test_config.yaml (added, +103/-0)
  • tests/claude_code/tool_use/__init__.py (added, +0/-0)
  • tests/claude_code/tool_use/test_anthropic.py (added, +114/-0)
  • tests/claude_code/tool_use/test_azure.py (added, +113/-0)
  • tests/claude_code/tool_use/test_bedrock_converse.py (added, +109/-0)
  • tests/claude_code/tool_use/test_bedrock_invoke.py (added, +109/-0)
  • tests/claude_code/tool_use/test_vertex_ai.py (added, +109/-0)
  • tests/claude_code/vision/__init__.py (added, +0/-0)
  • tests/claude_code/vision/test_anthropic.py (added, +99/-0)
  • tests/claude_code/vision/test_azure.py (added, +100/-0)
  • tests/claude_code/vision/test_bedrock_converse.py (added, +95/-0)
  • tests/claude_code/vision/test_bedrock_invoke.py (added, +95/-0)
  • tests/claude_code/vision/test_vertex_ai.py (added, +95/-0)
RAW_BUFFERClick to expand / collapse

Parent PRD

#26476

What to build

Stand up the daily cron pipeline on an isolated VM that publishes the real compatibility-matrix.json to the docs repo. After this slice lands, the hand-authored matrix JSON in the docs repo from slices #26477 and #26478 is replaced by automatically-regenerated output, and the docs page begins reflecting real test runs against the latest stable LiteLLM release.

This slice covers, end-to-end:

  • Provisioning the isolated VM in the LiteLLM repo's infrastructure (kept outside the main CI environment per the PRD's "Two CI environments" section).
  • Creating the GitHub App installed on the docs repo only, scoped to contents: write, and configuring the VM with its installation credentials.
  • Implementing the Latest Stable LiteLLM Resolver described in the PRD's "Version resolvers" section: queries the GitHub Releases API for BerriAI/litellm and returns the newest tag matching v*-stable.
  • Implementing the matrix publishing script that: resolves the latest stable LiteLLM tag, pulls the corresponding Docker image, installs the latest Claude Code CLI, runs the full compatibility test suite, invokes the Matrix JSON Builder to produce compatibility-matrix.json, and direct-pushes the file to the docs repo's main branch.
  • Configuring three triggers: GitHub Release published (auto-fires on new stable releases), daily cron, and manual dispatch.

Acceptance criteria

  • An isolated VM is provisioned and runs a daily cron job for the matrix pipeline.
  • A GitHub App scoped to contents: write on the docs repo only is created, installed, and its credentials provided to the VM via secret management.
  • The Latest Stable LiteLLM Resolver returns the newest tag matching v*-stable from the GitHub Releases API.
  • The publishing script pulls the Docker image for the resolved stable tag, runs it as the proxy, installs the latest Claude Code CLI, runs tests/claude_code/ against the proxy, and produces compatibility-matrix.json via the Matrix JSON Builder.
  • The script direct-pushes only compatibility-matrix.json to the docs repo's main branch using the GitHub App token; the script does not write any other files.
  • The pipeline is triggered by all three trigger sources: published *-stable releases, the daily cron, and manual dispatch.
  • After this slice lands, the JSON in the docs repo is regenerated by the cron rather than hand-authored.
  • The docs page banner correctly shows the LiteLLM version, Claude Code version, and generated_at timestamp produced by the pipeline.
  • The hand-authored matrix JSON from earlier slices is removed and replaced by the auto-generated output.

Blocked by

  • Blocked by #26478

User stories addressed

Reference by number from the parent PRD:

  • User story 13
  • User story 14
  • User story 15
  • User story 16

extent analysis

TL;DR

Implement the daily cron pipeline on an isolated VM to publish the compatibility-matrix.json to the docs repo, ensuring all acceptance criteria are met.

Guidance

  • Verify that the isolated VM is provisioned correctly and the GitHub App is installed with the correct scope (contents: write) to enable writing to the docs repo.
  • Ensure the Latest Stable LiteLLM Resolver is correctly implemented to fetch the newest stable tag from the GitHub Releases API.
  • Test the matrix publishing script to confirm it resolves the latest stable LiteLLM tag, runs the compatibility test suite, and produces compatibility-matrix.json without writing any other files to the docs repo.
  • Configure the three triggers (GitHub Release published, daily cron, and manual dispatch) to ensure the pipeline runs as expected.

Example

No specific code snippet can be provided without further details, but ensuring the script uses the correct GitHub App token for authentication and that the compatibility-matrix.json is correctly generated and pushed to the docs repo is crucial.

Notes

The implementation details of the matrix publishing script, the Latest Stable LiteLLM Resolver, and the configuration of the GitHub App and triggers are critical for the success of this task. Ensuring all components work together seamlessly is essential.

Recommendation

Apply the workaround by manually testing each component of the pipeline (Resolver, script, and triggers) to identify and fix any issues before automating the process, as the full automation depends on the correct functioning of each part.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#installation#SSR setup#ISR setup#authentication setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Compat Matrix] Slice 4: Daily cron VM publishes the matrix to docs [1 pull requests, 1 comments, 1 participants]