litellm - 💡(How to fix) Fix [Compat Matrix] Slice 7: Golden-file tests for Matrix JSON Builder [1 participants]

Parent PRD

#26476

What to build

Add golden-file tests for the Matrix JSON Builder module introduced in slice #26477. Per the PRD's "Testing Decisions" section, the builder is a pure function from inputs (pytest results artifact, manifest, providers list, run metadata) to output JSON, and the right test shape is golden-file comparisons that fix the schema contract.

Tests feed fixture inputs to the builder and compare the produced JSON against a checked-in expected output. Any schema drift — intentional or accidental — surfaces as a diff in review. The tests should cover: the four status states, multi-feature ordering driven by the manifest, multi-provider ordering driven by the providers list, the not_tested fill-in for missing combinations, the per-cell aggregation across three models, and metadata propagation (versions, timestamp, schema_version).

This slice can be pursued in parallel with slices #26478, #26479, #26480, #26481, and #26482 once #26477 has merged.

Acceptance criteria

• Golden-file tests for the Matrix JSON Builder exist with a fixture input directory and a checked-in expected JSON output.
• Tests cover all four status states (pass, fail, not_applicable, not_tested).
• Tests assert that feature row order in the output matches the manifest order.
• Tests assert that provider column order in the output matches the providers list passed in.
• Tests assert that not_tested is filled in when no test result exists for a (feature, provider) combination and no not_applicable reason is declared.
• Tests assert per-cell aggregation: when any of the three models fails, the cell is fail with the failing model identified in the error.
• Tests assert metadata propagation: schema_version, generated_at, litellm_version, and claude_code_version appear in the output as supplied.
• Tests run with the rest of the unit test suite and pass on CI.

Blocked by

• Blocked by #26477

User stories addressed

Reference by number from the parent PRD:

• User story 27 (schema versioning)