litellm - 💡(How to fix) Fix [Bug]: False-positive DD APM error spans on every request when allow_requests_on_db_unavailable: true [1 participants]

Traceback (most recent call last): File "/usr/lib/python3.13/site-packages/litellm/proxy/auth/user_api_key_auth.py", line 1029, in _user_api_key_auth_builder valid_token = await get_key_object( ^^^^^^^^^^^^^^^^^^^^^ ...<6 lines>... ) ^ File "/usr/lib/python3.13/site-packages/litellm/proxy/db/log_db_metrics.py", line 99, in wrapper raise e File "/usr/lib/python3.13/site-packages/litellm/proxy/db/log_db_metrics.py", line 42, in wrapper result = await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.13/site-packages/litellm/proxy/auth/auth_checks.py", line 2224, in get_key_object raise Exception( "No DB Connected. See - https://docs.litellm.ai/docs/proxy/virtual_keys" ) Exception: No DB Connected. See - https://docs.litellm.ai/docs/proxy/virtual_keys

TL;DR

• The issue can be mitigated by handling the "No DB Connected" exception in a way that does not mark the Datadog APM span as errored when allow_requests_on_db_unavailable is true.

Guidance

• Review the get_key_object function in auth_checks.py to understand how the exception is being raised and how it's affecting the APM span.
• Consider modifying the exception handling in _user_api_key_auth_builder to distinguish between critical errors and the expected "No DB Connected" exception when allow_requests_on_db_unavailable is true.
• Investigate Datadog APM configuration options to see if there's a way to ignore specific exceptions or error types from marking spans as errored.
• Verify that the allow_requests_on_db_unavailable setting is correctly applied and that the proxy requests are succeeding as expected despite the errored spans.

Example

try:
    valid_token = await get_key_object(
        # ...
    )
except Exception as e:
    if "No DB Connected" in str(e) and general_settings.allow_requests_on_db_unavailable:
        # Handle the exception without marking the span as errored
        pass
    else:
        # Re-raise or handle other exceptions as before
        raise

Notes

• This guidance assumes that the "No DB Connected" exception is expected and handled correctly by the application logic when allow_requests_on_db_unavailable is true.
• The exact implementation details may vary depending on the specific requirements and constraints of the LiteLLM proxy application.

Recommendation

• Apply workaround: Modify the exception handling to ignore the "No DB Connected" exception when allow_requests_on_db_unavailable is true, as this allows the application to continue functioning as intended while avoiding false-positive error reports in Datadog APM.