litellm - ✅(Solved) Fix [Bug]: /v1/evals proxy returns 500 on update eval, list runs, and get run due to Pydantic validation mismatches [1 pull requests, 1 participants]

litellm2026-03-11 07:52:04

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

BerriAI/litellm#23329•Fetched 2026-04-08 00:37:29

View on GitHub

Comments

Participants

Timeline

Reactions

Author

Jishnuraj07

Participants

Jishnuraj07

Timeline (top)

labeled ×3cross-referenced ×1referenced ×1

Error Message

== OpenAI Evals API Test (LiteLLM 1.82.0) ==

Step 1: Creating eval... PASS - Created: eval_69b11c00f3488191aced8950552b0d4d Step 2: Listing evals... PASS - 200 (2 eval(s)) Step 3: Getting eval details... PASS - 200 Step 4: Updating eval metadata... FAIL - 500: {"error":{"message":"litellm.APIConnectionError: openai - {\n "error": {\n "message": "Invalid type for 'metadata.queue_time_seconds': expected a string, but got a decimal number instead."}}"}} Step 5: Creating eval run... PASS - Created: evalrun_69b11c0ad860819199e57ad13d863a4e status=queued Step 6: Listing eval runs (waiting 5s for run to process)... FAIL - 500: {"error":{"message":"litellm.APIConnectionError: 2 validation errors for ListRunsResponse\ndata.0.per_testing_criteria_results.0.testing_criteria_index\n Field required ..."}} Step 7: Getting run details... FAIL - 500: {"error":{"message":"litellm.APIConnectionError: 2 validation errors for Run\nper_testing_criteria_results.0.testing_criteria_index\n Field required ..."}} Step 8: Deleting eval run... PASS - 200 Step 9: Deleting eval... PASS - 200

Root Cause

Pydantic ListRunsResponse validation fails because PerTestingCriteriaResult requires testing_criteria_index and result_counts fields that OpenAI doesn't return.

Fix Action

Fixed

Fixed by PR: fix(evals): fix three /v1/evals proxy endpoints returning HTTP 500 (https://github.com/BerriAI/litellm/pull/24263)

PR fix notes

PR #24263: fix(evals): fix three /v1/evals proxy endpoints returning HTTP 500

Repository: BerriAI/litellm
Author: sourrris
State: open | merged: False
Link: https://github.com/BerriAI/litellm/pull/24263

Description (problem / solution / changelog)

Relevant issues

Fixes #23329

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible, it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 3/5 before requesting a maintainer review

Type

🐛 Bug Fix

Changes

Three /v1/evals proxy endpoints return HTTP 500 when called through the LiteLLM proxy:

Bug 1 — `POST /v1/evals/{eval_id}` (Update Eval) returns 500

Root cause: The proxy injects queue_time_seconds as a float into request metadata. OpenAI requires all metadata values to be strings, so the request is rejected with a 400 that surfaces as a 500 through the proxy.

Fix: Convert non-string metadata values to strings in transform_update_eval_request() in the transformation layer (OpenAI-specific constraint).

Bug 2 & 3 — `GET /v1/evals/{eval_id}/runs` and `GET /v1/evals/{eval_id}/runs/{run_id}` return 500

Root cause: PerTestingCriteriaResult Pydantic model has testing_criteria_index: int and result_counts: ResultCounts as required fields, but OpenAI's actual response omits these fields and returns failed, passed, sample_id directly. Pydantic validation fails on deserialization.

Fix: Make testing_criteria_index and result_counts Optional with None defaults, add model_config = {"extra": "allow"} to accept extra fields from OpenAI.

Files changed

litellm/llms/openai/evals/transformation.py — metadata string sanitization in transform_update_eval_request()
litellm/types/llms/openai_evals.py — make PerTestingCriteriaResult fields Optional, allow extra fields
tests/test_litellm/llms/openai/evals/test_openai_evals_transformation.py — 3 regression tests covering all 3 bugs

Testing

$ python -m pytest tests/test_litellm/llms/openai/evals/test_openai_evals_transformation.py -v
======================== 16 passed, 2 warnings in 0.30s ========================

Changed files

litellm/llms/openai/evals/transformation.py (modified, +9/-0)
litellm/types/llms/openai_evals.py (modified, +4/-2)
tests/test_litellm/llms/openai/evals/test_openai_evals_transformation.py (modified, +120/-0)

Code Example

== OpenAI Evals API Test (LiteLLM 1.82.0) ==

Step 1: Creating eval...
  PASS - Created: eval_69b11c00f3488191aced8950552b0d4d
Step 2: Listing evals...
  PASS - 200 (2 eval(s))
Step 3: Getting eval details...
  PASS - 200
Step 4: Updating eval metadata...
  FAIL - 500: {"error":{"message":"litellm.APIConnectionError: openai - {\n  \"error\": {\n    \"message\": \"Invalid type for 'metadata.queue_time_seconds': expected a string, but got a decimal number instead.\"}}"}}
Step 5: Creating eval run...
  PASS - Created: evalrun_69b11c0ad860819199e57ad13d863a4e status=queued
Step 6: Listing eval runs (waiting 5s for run to process)...
  FAIL - 500: {"error":{"message":"litellm.APIConnectionError: 2 validation errors for ListRunsResponse\ndata.0.per_testing_criteria_results.0.testing_criteria_index\n  Field required ..."}}
Step 7: Getting run details...
  FAIL - 500: {"error":{"message":"litellm.APIConnectionError: 2 validation errors for Run\nper_testing_criteria_results.0.testing_criteria_index\n  Field required ..."}}
Step 8: Deleting eval run...
  PASS - 200
Step 9: Deleting eval...
  PASS - 200

---

import requests
import time

BASE = "http://localhost:4000/v1"
HEADERS = {"Content-Type": "application/json", "Authorization": "Bearer sk-1234"}

print("== OpenAI Evals API Test (LiteLLM 1.82.0) ==\n")

# Step 1: Create eval
print("Step 1: Creating eval...")
r = requests.post(f"{BASE}/evals", headers=HEADERS, json={
    "name": "LiteLLM 1.82.0 Test Eval",
    "data_source_config": {
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "question": {"type": "string"},
                "expected_answer": {"type": "string"}
            },
            "required": ["question", "expected_answer"]
        }
    },
    "testing_criteria": [{
        "type": "score_model",
        "name": "Accuracy",
        "model": "gpt-4o-mini",
        "input": [{"role": "user", "content": "Is this correct? Q: {{item.question}} A: {{sample.output_text}} Expected: {{item.expected_answer}}"}],
        "pass_threshold": 0.5
    }]
})
if r.status_code == 200:
    eval_id = r.json()["id"]
    print(f"  PASS - Created: {eval_id}")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:200]}")
    exit(1)

# Step 2: List evals
print("Step 2: Listing evals...")
r = requests.get(f"{BASE}/evals", headers=HEADERS)
status = "PASS" if r.status_code == 200 else "FAIL"
count = len(r.json().get("data", [])) if r.status_code == 200 else 0
print(f"  {status} - {r.status_code} ({count} eval(s))")

# Step 3: Get eval
print("Step 3: Getting eval details...")
r = requests.get(f"{BASE}/evals/{eval_id}", headers=HEADERS)
status = "PASS" if r.status_code == 200 else "FAIL"
print(f"  {status} - {r.status_code}")

# Step 4: Update eval
print("Step 4: Updating eval metadata...")
r = requests.post(f"{BASE}/evals/{eval_id}", headers=HEADERS, json={"metadata": {"env": "test"}})
if r.status_code == 200:
    print("  PASS - Updated eval")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 5: Create run
print("Step 5: Creating eval run...")
r = requests.post(f"{BASE}/evals/{eval_id}/runs", headers=HEADERS, json={
    "name": "Test Run 1.82.0",
    "data_source": {
        "type": "jsonl",
        "source": {
            "type": "file_content",
            "content": [
                {"item": {"question": "What is 2+2?", "expected_answer": "4"},
                 "sample": {"output_text": "4", "model": "gpt-4o-mini"}},
                {"item": {"question": "Capital of France?", "expected_answer": "Paris"},
                 "sample": {"output_text": "Paris", "model": "gpt-4o-mini"}}
            ]
        }
    }
})
run_id = None
if r.status_code == 200:
    run_id = r.json()["id"]
    run_status = r.json().get("status", "unknown")
    print(f"  PASS - Created: {run_id} status={run_status}")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 6: List runs
print("Step 6: Listing eval runs (waiting 5s for run to process)...")
time.sleep(5)
r = requests.get(f"{BASE}/evals/{eval_id}/runs", headers=HEADERS)
if r.status_code == 200:
    runs = r.json().get("data", [])
    print(f"  PASS - {len(runs)} run(s)")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 7: Get run details
if run_id:
    print("Step 7: Getting run details...")
    r = requests.get(f"{BASE}/evals/{eval_id}/runs/{run_id}", headers=HEADERS)
    if r.status_code == 200:
        print(f"  PASS - status={r.json().get('status', '?')}")
    else:
        print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 8: Delete run
if run_id:
    print("Step 8: Deleting eval run...")
    r = requests.delete(f"{BASE}/evals/{eval_id}/runs/{run_id}", headers=HEADERS)
    status = "PASS" if r.status_code == 200 else "FAIL"
    print(f"  {status} - {r.status_code}")

# Step 9: Delete eval
print("Step 9: Deleting eval...")
r = requests.delete(f"{BASE}/evals/{eval_id}", headers=HEADERS)
status = "PASS" if r.status_code == 200 else "FAIL"
print(f"  {status} - {r.status_code}")

print("\n== Done ==")


### Relevant log output

RAW_BUFFERClick to expand / collapse

Check for existing issues

I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

The OpenAI Evals proxy endpoints (/v1/evals) return HTTP 500 on three operations due to Pydantic response model mismatches with OpenAI's actual API responses. Tested on litellm v1.82.0

== OpenAI Evals API Test (LiteLLM 1.82.0) ==

Step 1: Creating eval...
  PASS - Created: eval_69b11c00f3488191aced8950552b0d4d
Step 2: Listing evals...
  PASS - 200 (2 eval(s))
Step 3: Getting eval details...
  PASS - 200
Step 4: Updating eval metadata...
  FAIL - 500: {"error":{"message":"litellm.APIConnectionError: openai - {\n  \"error\": {\n    \"message\": \"Invalid type for 'metadata.queue_time_seconds': expected a string, but got a decimal number instead.\"}}"}}
Step 5: Creating eval run...
  PASS - Created: evalrun_69b11c0ad860819199e57ad13d863a4e status=queued
Step 6: Listing eval runs (waiting 5s for run to process)...
  FAIL - 500: {"error":{"message":"litellm.APIConnectionError: 2 validation errors for ListRunsResponse\ndata.0.per_testing_criteria_results.0.testing_criteria_index\n  Field required ..."}}
Step 7: Getting run details...
  FAIL - 500: {"error":{"message":"litellm.APIConnectionError: 2 validation errors for Run\nper_testing_criteria_results.0.testing_criteria_index\n  Field required ..."}}
Step 8: Deleting eval run...
  PASS - 200
Step 9: Deleting eval...
  PASS - 200

Steps to Reproduce

python test code to run /evals through proxy

import requests
import time

BASE = "http://localhost:4000/v1"
HEADERS = {"Content-Type": "application/json", "Authorization": "Bearer sk-1234"}

print("== OpenAI Evals API Test (LiteLLM 1.82.0) ==\n")

# Step 1: Create eval
print("Step 1: Creating eval...")
r = requests.post(f"{BASE}/evals", headers=HEADERS, json={
    "name": "LiteLLM 1.82.0 Test Eval",
    "data_source_config": {
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "question": {"type": "string"},
                "expected_answer": {"type": "string"}
            },
            "required": ["question", "expected_answer"]
        }
    },
    "testing_criteria": [{
        "type": "score_model",
        "name": "Accuracy",
        "model": "gpt-4o-mini",
        "input": [{"role": "user", "content": "Is this correct? Q: {{item.question}} A: {{sample.output_text}} Expected: {{item.expected_answer}}"}],
        "pass_threshold": 0.5
    }]
})
if r.status_code == 200:
    eval_id = r.json()["id"]
    print(f"  PASS - Created: {eval_id}")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:200]}")
    exit(1)

# Step 2: List evals
print("Step 2: Listing evals...")
r = requests.get(f"{BASE}/evals", headers=HEADERS)
status = "PASS" if r.status_code == 200 else "FAIL"
count = len(r.json().get("data", [])) if r.status_code == 200 else 0
print(f"  {status} - {r.status_code} ({count} eval(s))")

# Step 3: Get eval
print("Step 3: Getting eval details...")
r = requests.get(f"{BASE}/evals/{eval_id}", headers=HEADERS)
status = "PASS" if r.status_code == 200 else "FAIL"
print(f"  {status} - {r.status_code}")

# Step 4: Update eval
print("Step 4: Updating eval metadata...")
r = requests.post(f"{BASE}/evals/{eval_id}", headers=HEADERS, json={"metadata": {"env": "test"}})
if r.status_code == 200:
    print("  PASS - Updated eval")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 5: Create run
print("Step 5: Creating eval run...")
r = requests.post(f"{BASE}/evals/{eval_id}/runs", headers=HEADERS, json={
    "name": "Test Run 1.82.0",
    "data_source": {
        "type": "jsonl",
        "source": {
            "type": "file_content",
            "content": [
                {"item": {"question": "What is 2+2?", "expected_answer": "4"},
                 "sample": {"output_text": "4", "model": "gpt-4o-mini"}},
                {"item": {"question": "Capital of France?", "expected_answer": "Paris"},
                 "sample": {"output_text": "Paris", "model": "gpt-4o-mini"}}
            ]
        }
    }
})
run_id = None
if r.status_code == 200:
    run_id = r.json()["id"]
    run_status = r.json().get("status", "unknown")
    print(f"  PASS - Created: {run_id} status={run_status}")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 6: List runs
print("Step 6: Listing eval runs (waiting 5s for run to process)...")
time.sleep(5)
r = requests.get(f"{BASE}/evals/{eval_id}/runs", headers=HEADERS)
if r.status_code == 200:
    runs = r.json().get("data", [])
    print(f"  PASS - {len(runs)} run(s)")
else:
    print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 7: Get run details
if run_id:
    print("Step 7: Getting run details...")
    r = requests.get(f"{BASE}/evals/{eval_id}/runs/{run_id}", headers=HEADERS)
    if r.status_code == 200:
        print(f"  PASS - status={r.json().get('status', '?')}")
    else:
        print(f"  FAIL - {r.status_code}: {r.text[:300]}")

# Step 8: Delete run
if run_id:
    print("Step 8: Deleting eval run...")
    r = requests.delete(f"{BASE}/evals/{eval_id}/runs/{run_id}", headers=HEADERS)
    status = "PASS" if r.status_code == 200 else "FAIL"
    print(f"  {status} - {r.status_code}")

# Step 9: Delete eval
print("Step 9: Deleting eval...")
r = requests.delete(f"{BASE}/evals/{eval_id}", headers=HEADERS)
status = "PASS" if r.status_code == 200 else "FAIL"
print(f"  {status} - {r.status_code}")

print("\n== Done ==")


### Relevant log output

```shell
### Bug 1: `POST /v1/evals/{eval_id}` — Update Eval → 500

The proxy injects `queue_time_seconds` (a float) into metadata before forwarding to OpenAI. OpenAI requires metadata values to be strings.


12:57:44 - LiteLLM Proxy:ERROR: common_request_processing.py:1131 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.APIConnectionError: openai - {
  "error": {
    "message": "Invalid type for 'metadata.queue_time_seconds': expected a string, but got a decimal number instead.",
    "type": "invalid_request_error",
    "param": "metadata.queue_time_seconds",
    "code": "invalid_type"
  }
}
Traceback (most recent call last):
  File "litellm\llms\custom_httpx\llm_http_handler.py", line 9771, in async_update_eval_handler
    response = await async_httpx_client.post(
        url=url, headers=headers, json=request_body, timeout=timeout
    )
  File "litellm\litellm_core_utils\logging_utils.py", line 297, in async_wrapper
    result = await func(*args, **kwargs)
  File "litellm\llms\custom_httpx\http_handler.py", line 510, in post
    raise e
  File "litellm\llms\custom_httpx\http_handler.py", line 466, in post
    response.raise_for_status()
httpx.HTTPStatusError: Client error '400 Bad Request' for url 'https://api.openai.com/v1/evals/eval_69b11968882c8191b871707d84f9c8a2'

  File "litellm\evals\main.py", line 627, in aupdate_eval
    response = await init_response
  File "litellm\llms\custom_httpx\llm_http_handler.py", line 9775, in async_update_eval_handler
    raise self._handle_error(e=e, provider_config=evals_api_provider_config)
litellm.llms.base_llm.chat.transformation.BaseLLMException: {
  "error": {
    "message": "Invalid type for 'metadata.queue_time_seconds': expected a string, but got a decimal number instead.",
    "type": "invalid_request_error",
    "param": "metadata.queue_time_seconds",
    "code": "invalid_type"
  }
}

  File "litellm\proxy\openai_evals_endpoints\endpoints.py", line 386, in update_eval
    return await processor.base_process_llm_request(...)
  File "litellm\proxy\common_request_processing.py", line 866, in base_process_llm_request
    responses = await llm_responses
  File "litellm\evals\main.py", line 632, in aupdate_eval
    raise litellm.exception_type(model=None, ...)
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: openai - {
  "error": {
    "message": "Invalid type for 'metadata.queue_time_seconds': expected a string, but got a decimal number instead.",
    "type": "invalid_request_error",
    "param": "metadata.queue_time_seconds",
    "code": "invalid_type"
  }
}


---

### Bug 2: `GET /v1/evals/{eval_id}/runs` — List Runs → 500

Pydantic `ListRunsResponse` validation fails because `PerTestingCriteriaResult` requires `testing_criteria_index` and `result_counts` fields that OpenAI doesn't return.


12:57:53 - LiteLLM Proxy:ERROR: common_request_processing.py:1131 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.APIConnectionError: 2 validation errors for ListRunsResponse
data.0.per_testing_criteria_results.0.testing_criteria_index
  Field required [type=missing, input_value={'failed': 1, 'passed': 1...4db3-b3ee-23ad9a70671e'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
data.0.per_testing_criteria_results.0.result_counts
  Field required [type=missing, input_value={'failed': 1, 'passed': 1...4db3-b3ee-23ad9a70671e'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

Traceback (most recent call last):
  File "litellm\evals\main.py", line 1332, in alist_runs
    response = await init_response
  File "litellm\llms\custom_httpx\llm_http_handler.py", line 10222, in async_list_runs_handler
    return evals_api_provider_config.transform_list_runs_response(
        raw_response=response, logging_obj=logging_obj,
    )
  File "litellm\llms\openai\evals\transformation.py", line 342, in transform_list_runs_response
    return ListRunsResponse(**response_json)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for ListRunsResponse
data.0.per_testing_criteria_results.0.testing_criteria_index
  Field required [type=missing, ...]
data.0.per_testing_criteria_results.0.result_counts
  Field required [type=missing, ...]

  File "litellm\proxy\openai_evals_endpoints\endpoints.py", line 768, in list_runs
    return await processor.base_process_llm_request(...)
  File "litellm\proxy\common_request_processing.py", line 866, in base_process_llm_request
    responses = await llm_responses
  File "litellm\evals\main.py", line 1337, in alist_runs
    raise litellm.exception_type(model=None, ...)
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: 2 validation errors for ListRunsResponse
data.0.per_testing_criteria_results.0.testing_criteria_index
  Field required [type=missing, ...]
data.0.per_testing_criteria_results.0.result_counts
  Field required [type=missing, ...]


---

### Bug 3: `GET /v1/evals/{eval_id}/runs/{run_id}` — Get Run → 500

Same Pydantic validation failure in the `Run` model (same root cause as Bug 2, different endpoint).


12:57:55 - LiteLLM Proxy:ERROR: common_request_processing.py:1131 - litellm.proxy.proxy_server._handle_llm_api_exception(): Exception occured - litellm.APIConnectionError: 2 validation errors for Run
per_testing_criteria_results.0.testing_criteria_index
  Field required [type=missing, input_value={'failed': 1, 'passed': 1...4db3-b3ee-23ad9a70671e'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
per_testing_criteria_results.0.result_counts
  Field required [type=missing, input_value={'failed': 1, 'passed': 1...4db3-b3ee-23ad9a70671e'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

Traceback (most recent call last):
  File "litellm\evals\main.py", line 1511, in aget_run
    response = await init_response
  File "litellm\llms\custom_httpx\llm_http_handler.py", line 10327, in async_get_run_handler
    return evals_api_provider_config.transform_get_run_response(
        raw_response=response, logging_obj=logging_obj,
    )
  File "litellm\llms\openai\evals\transformation.py", line 368, in transform_get_run_response
    return Run(**response_json)
pydantic_core._pydantic_core.ValidationError: 2 validation errors for Run
per_testing_criteria_results.0.testing_criteria_index
  Field required [type=missing, ...]
per_testing_criteria_results.0.result_counts
  Field required [type=missing, ...]

  File "litellm\proxy\openai_evals_endpoints\endpoints.py", line 857, in get_run
    return await processor.base_process_llm_request(...)
  File "litellm\proxy\common_request_processing.py", line 866, in base_process_llm_request
    responses = await llm_responses
  File "litellm\evals\main.py", line 1516, in aget_run
    raise litellm.exception_type(model=None, ...)
litellm.exceptions.APIConnectionError: litellm.APIConnectionError: 2 validation errors for Run
per_testing_criteria_results.0.testing_criteria_index
  Field required [type=missing, ...]
per_testing_criteria_results.0.result_counts
  Field required [type=missing, ...]


---

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v.1.82.0

Twitter / LinkedIn details

https://www.linkedin.com/in/jishnuraj07/

extent analysis

Fix Plan

To resolve the issues, we need to address the following:

Convert queue_time_seconds to a string before sending it to OpenAI.
Update the Pydantic models to make testing_criteria_index and result_counts optional.

Step 1: Convert `queue_time_seconds` to a string

In the update_eval function, convert queue_time_seconds to a string before sending the request to OpenAI.

# In update_eval function
metadata = {"queue_time_seconds": str(queue_time_seconds)}  # Convert to string

Step 2: Update Pydantic models

Update the PerTestingCriteriaResult and Run models to make testing_criteria_index and result_counts optional.

# In models.py
from pydantic import BaseModel

class PerTestingCriteriaResult(BaseModel):
    testing_criteria_index: Optional[int] = None  # Make optional
    result_counts: Optional[Dict[str, int]] = None  # Make optional

class Run(BaseModel):
    per_testing_criteria_results: List[PerTestingCriteriaResult]

Step 3: Handle missing fields in Pydantic models

Update the transform_list_runs_response and transform_get_run_response functions to handle missing fields.

# In transformation.py
def transform_list_runs_response(raw_response, logging_obj):
    # ...
    per_testing_criteria_results = raw_response.get("per_testing_criteria_results", [])
    for result in per_testing_criteria_results:
        result["testing_criteria_index"] = result.get("testing_criteria_index")
        result["result_counts"] = result.get("result_counts")
    # ...

def transform_get_run_response(raw_response, logging_obj):
    # ...
    per_testing_criteria_results = raw_response.get("per_testing_criteria_results", [])
    for result in per_testing_criteria_results:
        result["testing_criteria_index"] = result.get("testing_criteria_index")
        result["result_counts"] = result.get("result_counts")
    # ...

Verification

To verify the fixes, run the test code again and check that the endpoints return the expected responses without errors.

Extra Tips

Make sure to update the Pydantic models and transformation functions to handle missing fields correctly.
Consider adding additional error handling and logging to ensure that any future issues are properly handled and reported.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #autograd error #memory leak #API versioning #request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.