litellm - 💡(How to fix) Fix [Feature]: Custom Transport support for acompletion calls [1 comments, 1 participants]

TL;DR

To address the need for customizing the transport layer, consider adding an optional "client" argument to the acompletion() function that accepts a custom httpx client.

Guidance

• Review the httpx library documentation to understand how to create a custom client that can be used for tasks such as refreshing bearer tokens or injecting observability actions.
• Evaluate the feasibility of implementing a standardized middleware using httpx to simplify atomic manipulations.
• Assess the potential benefits and limitations of extending this customization to other execution paths beyond acompletion() calls.
• Consider the potential impact on existing users and the complexity of configuring and deploying a proxy versus using an in-process middleware solution.

Notes

The proposal suggests a targeted change to the acompletion() function, but the potential for broader application exists. Careful consideration of the trade-offs between complexity, flexibility, and maintainability is necessary.

Recommendation

Apply workaround: Implement a custom httpx client and pass it as an argument to the acompletion() function to test the proposed solution, allowing for evaluation of its effectiveness before potentially extending it to other areas.