litellm - 💡(How to fix) Fix Bedrock batch inference broken via proxy: credential fields and model silently dropped [1 participants]

Error 1 — S3/batch credential fields silently dropped

Error 2 — model not included in credentials dict

1. POST /v1/files with a batch JSONL file → fails with S3 bucket error
2. Even if you work around that, POST /v1/batches → fails with unsupported provider error

TL;DR

Update CredentialLiteLLMParams to include missing fields like s3_bucket_name and ensure model is included in the credentials dict to fix Bedrock batch inference issues.

Guidance

• Verify that CredentialLiteLLMParams includes all necessary fields such as s3_bucket_name, s3_region_name, s3_encryption_key_id, and aws_batch_role_arn to prevent silent dropping of credentials.
• Ensure get_deployment_credentials_with_provider() returns the deployment's model to correctly configure the batch endpoint.
• Review the proposed fix in #24548, which includes unit tests, to address both error cases.
• Consider updating to a version where the fix is merged, if available, to resolve the issue.

Example

No explicit code example is provided due to the lack of specific code snippets in the issue, but updating CredentialLiteLLMParams might look something like adding missing fields to its definition.

Notes

The issue seems to be specific to versions 1.83.1+ of LiteLLM, and the fix is pending review. Applying the fix from #24548 should resolve the issues with S3/batch credential fields being silently dropped and the model not being included in the credentials dict.

Recommendation

Apply the workaround by updating CredentialLiteLLMParams and ensuring model is included in the credentials dict, as a fix is not yet merged into the main version. This approach allows for a temporary resolution until the official fix is available.