TL;DR

The issue with traffic dropping to zero for certain endpoints may be related to the least busy strategy in the _get_available_deployments method, which could be causing an uneven distribution of requests.

Guidance

• Review the _get_available_deployments method to ensure it's correctly implementing the least busy strategy, considering the potential for deployments to be tied at the minimum traffic level.
• Verify that the all_deployments dictionary is being updated correctly, as the issue may be related to the way deployments are being mapped to their respective traffic levels.
• Check if the healthy_deployments list is being populated correctly, as the method relies on this list to determine the available deployments.
• Consider adding logging or monitoring to track the traffic levels for each deployment, to better understand how the traffic is being distributed.

Example

# Example of how to add logging to track traffic levels
import logging

def _get_available_deployments(
        self,
        healthy_deployments: list,
        all_deployments: dict,
    ):
    logging.info(f"Healthy deployments: {healthy_deployments}")
    logging.info(f"All deployments: {all_deployments}")
    # ... rest of the method ...

Notes

The provided code snippet seems to be a part of the LiteLLM SDK, and the issue may be related to the specific implementation of the least busy strategy. Without more information about the traffic patterns and the deployment configurations, it's difficult to provide a more specific solution.

Recommendation

Apply workaround: Modify the _get_available_deployments method to include additional logging and monitoring to track the traffic levels for each deployment, to better understand how the traffic is being distributed. This will help identify the root cause of the issue and provide more insight into the behavior of the least busy strategy.