litellm - ✅(Solved) Fix [Bug]: policies/test-pipeline returns 500 after the pipeline already passes [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#26877Fetched 2026-05-01 05:34:36
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Participants
Timeline (top)
labeled ×2cross-referenced ×1

Error Message

litellm-1 | 07:43:12 - LiteLLM Proxy:DEBUG: pipeline_executor.py:89 - Pipeline 'test-pipeline' step 5: guardrail=SSN_PII Native Block, outcome=pass, action=next
litellm-1 | INFO: 10.221.98.236:57514 - "POST /policies/test-pipeline HTTP/1.1" 500 Internal Server Error
litellm-1 | ERROR: Exception in ASGI application
litellm-1 | Traceback (most recent call last):
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 337, in jsonable_encoder
litellm-1 | data = dict(obj)
litellm-1 | TypeError: 'InstrumentationInfo' object is not iterable
litellm-1 |
litellm-1 | During handling of the above exception, another exception occurred:
litellm-1 |
litellm-1 | Traceback (most recent call last):
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 342, in jsonable_encoder
litellm-1 | data = vars(obj)
litellm-1 | TypeError: vars() argument must have dict attribute
litellm-1 |
litellm-1 | The above exception was the direct cause of the following exception:
litellm-1 |
litellm-1 | Traceback (most recent call last):
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
litellm-1 | result = await app( # type: ignore[func-returns-value]
litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1 | self.scope, self.receive, self.send
litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1 | )
litellm-1 | ^
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
litellm-1 | return await self.app(scope, receive, send)
litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/applications.py", line 1139, in call
litellm-1 | await super().call(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/applications.py", line 107, in call
litellm-1 | await self.middleware_stack(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in call
litellm-1 | raise exc
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in call
litellm-1 | await self.app(scope, receive, _send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/litellm/proxy/middleware/in_flight_requests_middleware.py", line 46, in call
litellm-1 | await self.app(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/litellm/proxy/middleware/prometheus_auth_middleware.py", line 38, in call
litellm-1 | await self.app(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/cors.py", line 93, in call
litellm-1 | await self.simple_response(scope, receive, send, request_headers=headers)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/cors.py", line 144, in simple_response
litellm-1 | await self.app(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in call
litellm-1 | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
litellm-1 | raise exc
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
litellm-1 | await app(scope, receive, sender)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
litellm-1 | await self.app(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 716, in call
litellm-1 | await self.middleware_stack(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 736, in app
litellm-1 | await route.handle(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle
litellm-1 | await self.app(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app
litellm-1 | await wrap_app_handling_exceptions(app, request)(scope, receive, send)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
litellm-1 | raise exc
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
litellm-1 | await app(scope, receive, sender)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 106, in app
litellm-1 | response = await f(request)
litellm-1 | ^^^^^^^^^^^^^^^^
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 452, in app
litellm-1 | content = await serialize_response(
litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1 | ...<10 lines>...
litellm-1 | )
litellm-1 | ^
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 305, in serialize_response
litellm-1 | return jsonable_encoder(response_content)
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder
litellm-1 | encoded_value = jsonable_encoder(
litellm-1 | value,
litellm-1 | ...<4 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | )
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 316, in jsonable_encoder
litellm-1 | jsonable_encoder(
litellm-1 | ~~~~~~~~~~~~~~~~^
litellm-1 | item,
litellm-1 | ^^^^^
litellm-1 | ...<7 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1 | )
litellm-1 | ^
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder
litellm-1 | encoded_value = jsonable_encoder(
litellm-1 | value,
litellm-1 | ...<4 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | )
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder
litellm-1 | encoded_value = jsonable_encoder(
litellm-1 | value,
litellm-1 | ...<4 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | )
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder
litellm-1 | encoded_value = jsonable_encoder(
litellm-1 | value,
litellm-1 | ...<4 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | )
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 346, in jsonable_encoder
litellm-1 | return jsonable_encoder(
litellm-1 | data,
litellm-1 | ...<7 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | )
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder
litellm-1 | encoded_value = jsonable_encoder(
litellm-1 | value,
litellm-1 | ...<4 lines>...
litellm-1 | sqlalchemy_safe=sqlalchemy_safe,
litellm-1 | )
litellm-1 | File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 345, in jsonable_encoder
litellm-1 | raise ValueError(errors) from e
litellm-1 | ValueError: [TypeError("'InstrumentationInfo' object is not iterable"), TypeError('vars() argument must have dict attribute')]
litellm-1 | 07:43:13 - LiteLLM:DEBUG: custom_batch_logger.py:39 - CustomLogger periodic flush after 5 seconds

Fix Action

Fix / Workaround

  1. Configure some guardrails. In my case, I configured some content filter guardrails with custom regex
  2. Make a policy (no matter attach)
  3. Use the test pipeline and will see this error from the program logs.
  4. Besides, the content filter guardrails in the policy seems not working. Such as the block/mask will not take effect to the prompt which should be block.
  5. I downgrade to 1.83.7 then issue gone.

PR fix notes

PR #26896: fix: make policy pipeline test response JSON-safe

Description (problem / solution / changelog)

What

Fixes #26877 by making the /policies/test-pipeline response JSON-safe after the pipeline executor completes. The endpoint now recursively removes internal tracing/auth objects such as parent_otel_span, litellm_parent_otel_span, and user_api_key_auth from returned pipeline data, while preserving normal JSON values and stringifying any remaining opaque objects instead of letting FastAPI fail during response serialization.

Why

The pipeline can pass successfully but still return HTTP 500 if a guardrail-modified payload contains OpenTelemetry internals. The previous result.model_dump() used Python-mode values, which lets those objects leak into FastAPI jsonable_encoder.

Tests

  • uv run --extra proxy pytest tests/test_litellm/proxy/policy_engine/test_policy_endpoints.py tests/test_litellm/proxy/policy_engine/test_pipeline_executor.py -q
  • uv run --extra proxy black --check litellm/proxy/policy_engine/policy_endpoints.py tests/test_litellm/proxy/policy_engine/test_policy_endpoints.py
  • uv run --extra proxy ruff check litellm/proxy/policy_engine/policy_endpoints.py tests/test_litellm/proxy/policy_engine/test_policy_endpoints.py
  • uv run --extra proxy mypy litellm/proxy/policy_engine/policy_endpoints.py
  • git diff --check -- litellm/proxy/policy_engine/policy_endpoints.py tests/test_litellm/proxy/policy_engine/test_policy_endpoints.py

Changed files

  • litellm/proxy/policy_engine/policy_endpoints.py (modified, +86/-2)
  • tests/test_litellm/proxy/policy_engine/test_policy_endpoints.py (added, +78/-0)

Code Example

litellm-1  | 07:43:12 - LiteLLM 
Proxy:DEBUG: pipeline_executor.py:89 - Pipeline 'test-pipeline' step 5: guardrail=SSN_PII Native Block, outcome=pass, action=next                   
litellm-1  | INFO:     10.221.98.236:57514 - "POST /policies/test-pipeline HTTP/1.1" 500 Internal Server Error                                      
litellm-1  | ERROR:    Exception in ASGI application                                                                                                
litellm-1  | Traceback (most recent call last):                                                                                                     
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 337, in jsonable_encoder                                    
litellm-1  |     data = dict(obj)                                                                                                                   
litellm-1  | TypeError: 'InstrumentationInfo' object is not iterable                                                                                
litellm-1  |                                                                                                                                        
litellm-1  | During handling of the above exception, another exception occurred:                                                                    
litellm-1  |                                                                                                                                        
litellm-1  | Traceback (most recent call last):                                                                                                     
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 342, in jsonable_encoder                                    
litellm-1  |     data = vars(obj)                                                                                                                   
litellm-1  | TypeError: vars() argument must have __dict__ attribute                                                                                
litellm-1  |                                                                                                                                        
litellm-1  | The above exception was the direct cause of the following exception:                                                                   
litellm-1  |                                                                                                                                        
litellm-1  | Traceback (most recent call last):                                                                                                     
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi                             
litellm-1  |     result = await app(  # type: ignore[func-returns-value]                                                                            
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                            
litellm-1  |         self.scope, self.receive, self.send                                                                                            
litellm-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                            
litellm-1  |     )                                                                                                                                  
litellm-1  |     ^                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__                             
litellm-1  |     return await self.app(scope, receive, send)                                                                                        
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                        
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/applications.py", line 1139, in __call__                                       
litellm-1  |     await super().__call__(scope, receive, send)                                                                                       
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/applications.py", line 107, in __call__                                      
litellm-1  |     await self.middleware_stack(scope, receive, send)                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__                                 
litellm-1  |     raise exc                                                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__                                 
litellm-1  |     await self.app(scope, receive, _send)                                                                                              
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/litellm/proxy/middleware/in_flight_requests_middleware.py", line 46, in __call__       
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/litellm/proxy/middleware/prometheus_auth_middleware.py", line 38, in __call__          
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/cors.py", line 93, in __call__                                    
litellm-1  |     await self.simple_response(scope, receive, send, request_headers=headers)                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/cors.py", line 144, in simple_response                            
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__                              
litellm-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)                                                           
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app                              
litellm-1  |     raise exc                                                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app                              
litellm-1  |     await app(scope, receive, sender)                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__                            
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__                                           
litellm-1  |     await self.middleware_stack(scope, receive, send)                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 736, in app                                                
litellm-1  |     await route.handle(scope, receive, send)                                                                                           
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle                                             
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app                                                  
litellm-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)                                                             
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app                              
litellm-1  |     raise exc                                                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app                              
litellm-1  |     await app(scope, receive, sender)                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 106, in app                                                  
litellm-1  |     response = await f(request)                                                                                                        
litellm-1  |                ^^^^^^^^^^^^^^^^                                                                                                        
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 452, in app                                                  
litellm-1  |     content = await serialize_response(                                                                                                
litellm-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                
litellm-1  |     ...<10 lines>...                                                                                                                   
litellm-1  |     )                                                                                                                                  
litellm-1  |     ^                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 305, in serialize_response                                   
litellm-1  |     return jsonable_encoder(response_content)                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 316, in jsonable_encoder                                    
litellm-1  |     jsonable_encoder(                                                                                                                  
litellm-1  |     ~~~~~~~~~~~~~~~~^                                                                                                                  
litellm-1  |         item,                                                                                                                          
litellm-1  |         ^^^^^                                                                                                                          
litellm-1  |     ...<7 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |     ^                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 346, in jsonable_encoder                                    
litellm-1  |     return jsonable_encoder(                                                                                                           
litellm-1  |         data,                                                                                                                          
litellm-1  |     ...<7 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 345, in jsonable_encoder                                    
litellm-1  |     raise ValueError(errors) from e                                                                                                    
litellm-1  | ValueError: [TypeError("'InstrumentationInfo' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]     
litellm-1  | 07:43:13 - LiteLLM:DEBUG: custom_batch_logger.py:39 - CustomLogger periodic flush after 5 seconds
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

Hi team,

I think there is a bug in the policy pipeline test endpoint.

I am using LiteLLM 1.83.10. When I call /policies/test-pipeline, the pipeline itself appears to run successfully, but the endpoint still returns 500.

From the logs, I can see that the pipeline step completed normally first:

Pipeline 'test-pipeline' step 5: guardrail=SSN_PII Native Block, outcome=pass, action=next

Right after that, FastAPI fails while serializing the response:

TypeError: 'InstrumentationInfo' object is not iterable

During handling of the above exception, another exception occurred:

TypeError: vars() argument must have dict attribute

...

File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 305, in serialize_response return jsonable_encoder(response_content)

...

ValueError: [TypeError("'InstrumentationInfo' object is not iterable"), TypeError('vars() argument must have dict attribute')]

So this does not look like a guardrail decision failure. It looks like the endpoint runs the pipeline, but then crashes while building the JSON response.

Expected behavior: The endpoint should return the pipeline test result as normal JSON.

Actual behavior: The pipeline finishes, but /policies/test-pipeline returns 500 during FastAPI response serialization.

Why I think this is a LiteLLM bug:

The current code path seems to be:

policy_endpoints.py returns result.model_dump()

pipeline_executor.py can put raw guardrail return values into modified_data

metadata building copies fields from UserAPIKeyAuth

UserAPIKeyAuth includes parent_otel_span

My guess is that a non-JSON-serializable OpenTelemetry object is leaking into modified_data or litellm_metadata, and then FastAPI jsonable_encoder crashes on it.

I could not find an exact duplicate for this endpoint, but these issues look related:

#13554 #3820

Full log snippet:

07:43:12 - LiteLLM Proxy:DEBUG: pipeline_executor.py:89 - Pipeline 'test-pipeline' step 5: guardrail=SSN_PII Native Block, outcome=pass, action=next INFO: 10.221.98.236:57514 - "POST /policies/test-pipeline HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 337, in jsonable_encoder data = dict(obj) TypeError: 'InstrumentationInfo' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 342, in jsonable_encoder data = vars(obj) TypeError: vars() argument must have dict attribute

The above exception was the direct cause of the following exception:

...

File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 305, in serialize_response return jsonable_encoder(response_content)

...

ValueError: [TypeError("'InstrumentationInfo' object is not iterable"), TypeError('vars() argument must have dict attribute')]

Steps to Reproduce

  1. Configure some guardrails. In my case, I configured some content filter guardrails with custom regex
  2. Make a policy (no matter attach)
  3. Use the test pipeline and will see this error from the program logs.
  4. Besides, the content filter guardrails in the policy seems not working. Such as the block/mask will not take effect to the prompt which should be block.
  5. I downgrade to 1.83.7 then issue gone.

Relevant log output

litellm-1  | 07:43:12 - LiteLLM 
Proxy:DEBUG: pipeline_executor.py:89 - Pipeline 'test-pipeline' step 5: guardrail=SSN_PII Native Block, outcome=pass, action=next                   
litellm-1  | INFO:     10.221.98.236:57514 - "POST /policies/test-pipeline HTTP/1.1" 500 Internal Server Error                                      
litellm-1  | ERROR:    Exception in ASGI application                                                                                                
litellm-1  | Traceback (most recent call last):                                                                                                     
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 337, in jsonable_encoder                                    
litellm-1  |     data = dict(obj)                                                                                                                   
litellm-1  | TypeError: 'InstrumentationInfo' object is not iterable                                                                                
litellm-1  |                                                                                                                                        
litellm-1  | During handling of the above exception, another exception occurred:                                                                    
litellm-1  |                                                                                                                                        
litellm-1  | Traceback (most recent call last):                                                                                                     
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 342, in jsonable_encoder                                    
litellm-1  |     data = vars(obj)                                                                                                                   
litellm-1  | TypeError: vars() argument must have __dict__ attribute                                                                                
litellm-1  |                                                                                                                                        
litellm-1  | The above exception was the direct cause of the following exception:                                                                   
litellm-1  |                                                                                                                                        
litellm-1  | Traceback (most recent call last):                                                                                                     
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi                             
litellm-1  |     result = await app(  # type: ignore[func-returns-value]                                                                            
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                            
litellm-1  |         self.scope, self.receive, self.send                                                                                            
litellm-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                            
litellm-1  |     )                                                                                                                                  
litellm-1  |     ^                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__                             
litellm-1  |     return await self.app(scope, receive, send)                                                                                        
litellm-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                        
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/applications.py", line 1139, in __call__                                       
litellm-1  |     await super().__call__(scope, receive, send)                                                                                       
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/applications.py", line 107, in __call__                                      
litellm-1  |     await self.middleware_stack(scope, receive, send)                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__                                 
litellm-1  |     raise exc                                                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__                                 
litellm-1  |     await self.app(scope, receive, _send)                                                                                              
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/litellm/proxy/middleware/in_flight_requests_middleware.py", line 46, in __call__       
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/litellm/proxy/middleware/prometheus_auth_middleware.py", line 38, in __call__          
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/cors.py", line 93, in __call__                                    
litellm-1  |     await self.simple_response(scope, receive, send, request_headers=headers)                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/cors.py", line 144, in simple_response                            
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__                              
litellm-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)                                                           
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app                              
litellm-1  |     raise exc                                                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app                              
litellm-1  |     await app(scope, receive, sender)                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__                            
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__                                           
litellm-1  |     await self.middleware_stack(scope, receive, send)                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 736, in app                                                
litellm-1  |     await route.handle(scope, receive, send)                                                                                           
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle                                             
litellm-1  |     await self.app(scope, receive, send)                                                                                               
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 120, in app                                                  
litellm-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)                                                             
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app                              
litellm-1  |     raise exc                                                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app                              
litellm-1  |     await app(scope, receive, sender)                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 106, in app                                                  
litellm-1  |     response = await f(request)                                                                                                        
litellm-1  |                ^^^^^^^^^^^^^^^^                                                                                                        
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 452, in app                                                  
litellm-1  |     content = await serialize_response(                                                                                                
litellm-1  |               ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                
litellm-1  |     ...<10 lines>...                                                                                                                   
litellm-1  |     )                                                                                                                                  
litellm-1  |     ^                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/routing.py", line 305, in serialize_response                                   
litellm-1  |     return jsonable_encoder(response_content)                                                                                          
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 316, in jsonable_encoder                                    
litellm-1  |     jsonable_encoder(                                                                                                                  
litellm-1  |     ~~~~~~~~~~~~~~~~^                                                                                                                  
litellm-1  |         item,                                                                                                                          
litellm-1  |         ^^^^^                                                                                                                          
litellm-1  |     ...<7 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |     ^                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 346, in jsonable_encoder                                    
litellm-1  |     return jsonable_encoder(                                                                                                           
litellm-1  |         data,                                                                                                                          
litellm-1  |     ...<7 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 302, in jsonable_encoder                                    
litellm-1  |     encoded_value = jsonable_encoder(                                                                                                  
litellm-1  |         value,                                                                                                                         
litellm-1  |     ...<4 lines>...                                                                                                                    
litellm-1  |         sqlalchemy_safe=sqlalchemy_safe,                                                                                               
litellm-1  |     )                                                                                                                                  
litellm-1  |   File "/app/.venv/lib/python3.13/site-packages/fastapi/encoders.py", line 345, in jsonable_encoder                                    
litellm-1  |     raise ValueError(errors) from e                                                                                                    
litellm-1  | ValueError: [TypeError("'InstrumentationInfo' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]     
litellm-1  | 07:43:13 - LiteLLM:DEBUG: custom_batch_logger.py:39 - CustomLogger periodic flush after 5 seconds

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.83.10

Twitter / LinkedIn details

No response

extent analysis

TL;DR

Downgrade to LiteLLM version 1.83.7 to resolve the issue with the policy pipeline test endpoint returning a 500 error due to a non-JSON-serializable OpenTelemetry object.

Guidance

  • The error occurs because an InstrumentationInfo object is not iterable and does not have a __dict__ attribute, causing FastAPI's jsonable_encoder to fail.
  • The issue is likely caused by a non-JSON-serializable OpenTelemetry object leaking into modified_data or litellm_metadata.
  • To verify the issue, check the logs for the TypeError: 'InstrumentationInfo' object is not iterable error.
  • To mitigate the issue, try downgrading to LiteLLM version 1.83.7, which has been reported to resolve the issue.

Notes

  • The issue may be related to the integration of OpenTelemetry with LiteLLM, and further investigation may be needed to determine the root cause.
  • Downgrading to version 1.83.7 may not be a permanent solution, and it is recommended to monitor the issue and wait for a fix in a future version of LiteLLM.

Recommendation

Apply workaround: Downgrade to LiteLLM version 1.83.7. This version has been reported to resolve the issue, and it can be used as a temporary workaround until a fix is available in a future version.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: policies/test-pipeline returns 500 after the pipeline already passes [1 pull requests, 1 participants]