litellm - ✅(Solved) Fix [Bug]: S3 v2 callback: SigV4 signature mismatch when object keys contain `=` (base64 padding) with S3-compatible endpoints [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#24583Fetched 2026-04-08 01:32:31
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Author
Participants
Timeline (top)
labeled ×2cross-referenced ×1

Error Message

LiteLLM:ERROR: s3_v2.py:380 - Error uploading to s3: Client error '403 Forbidden' for url 'https://garage.example.com/litellm-usage/2026-03-25/time-18-07-17-208437_resp_bGl0ZWxsbT...OQ==.json'

Garage's error response:

<Error> <Code>AccessDenied</Code> <Message>Forbidden: Invalid signature</Message> <Resource>/litellm-usage/2026-03-25/time-18-07-17-208437_resp_bGl0ZWxsbT...OQ==.json</Resource> </Error>

Fix Action

Fixed

PR fix notes

PR #24585: fix(s3_v2): URL-encode object keys to prevent SigV4 signature mismatch

Description (problem / solution / changelog)

S3 object keys containing special characters like '=' (from base64 padding in Responses API composite IDs) caused 403 Invalid signature errors on S3-compatible endpoints (Garage, MinIO).

botocore's SigV4Auth URL-encodes the path in the canonical request (turning '=' into '%3D'), but httpx sends the literal '=' on the wire. The server computes its signature using the unencoded path, causing a mismatch.

Fix: apply urllib.parse.quote() to the object key before embedding it in the URL, so both the SigV4 signature and the HTTP request agree on the path encoding.

Relevant issues

Fixes https://github.com/BerriAI/litellm/issues/24583

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.
  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Type

Bug Fix

Changed files

  • litellm/integrations/s3_v2.py (modified, +24/-9)
  • tests/test_litellm/integrations/test_s3_v2.py (modified, +114/-0)

Code Example

from urllib.parse import quote

# In async_upload_data_to_s3(), when constructing the URL:
encoded_key = quote(batch_logging_element.s3_object_key, safe="/")

if self.s3_endpoint_url and self.s3_bucket_name:
    if self.s3_use_virtual_hosted_style:
        url = f"{protocol}{self.s3_bucket_name}.{endpoint_host}/{encoded_key}"
    else:
        url = self.s3_endpoint_url + "/" + self.s3_bucket_name + "/" + encoded_key
else:
    url = f"https://{self.s3_bucket_name}.s3.{self.s3_region_name}.amazonaws.com/{encoded_key}"

---

import hashlib
import requests
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
import httpx

creds = Credentials('ACCESS_KEY', 'SECRET_KEY')
# Object key with base64 padding (==)
url = 'https://garage.example.com/bucket/key_with_base64==.json'
data = '{"test": 1}'
content_hash = hashlib.sha256(data.encode('utf-8')).hexdigest()

headers = {
    'Content-Type': 'application/json',
    'x-amz-content-sha256': content_hash,
}

req = requests.Request('PUT', url, data=data, headers=headers)
prepped = req.prepare()

aws_request = AWSRequest(
    method=prepped.method,
    url=prepped.url,
    data=prepped.body,
    headers=prepped.headers,
)

SigV4Auth(creds, 's3', 'region').add_auth(aws_request)

# botocore signed against canonical path: /bucket/key_with_base64%3D%3D.json
# httpx sends the request to: /bucket/key_with_base64==.json
# The server sees ==, computes signature with ==, doesn't match %3D%3D → 403
response = httpx.put(url, data=data, headers=dict(aws_request.headers.items()))
print(response.status_code)  # 403

---

LiteLLM:ERROR: s3_v2.py:380 - Error uploading to s3: Client error '403 Forbidden' for url
'https://garage.example.com/litellm-usage/2026-03-25/time-18-07-17-208437_resp_bGl0ZWxsbT...OQ==.json'


Garage's error response:

<Error>
  <Code>AccessDenied</Code>
  <Message>Forbidden: Invalid signature</Message>
  <Resource>/litellm-usage/2026-03-25/time-18-07-17-208437_resp_bGl0ZWxsbT...OQ==.json</Resource>
</Error>
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

The s3_v2 logging callback fails with 403 Forbidden: Invalid signature when uploading to S3-compatible endpoints (Garage, MinIO, etc.) if the S3 object key contains = characters — which happens whenever the response ID contains base64 padding (e.g. Responses API composite IDs like resp_bGl0ZWxsbT...OQ==).

All uploads with base64-padded response IDs fail. Uploads with simple response IDs (e.g. resp_0e7dd770... from chat completions) succeed.

In s3_v2.py async_upload_data_to_s3(), the URL is constructed with the raw object key (which may contain = from base64 padding), then passed to both botocore.awsrequest.AWSRequest for SigV4 signing and httpx for the actual HTTP request.

The problem is that botocore's SigV4Auth URL-encodes the path when computing the canonical request (turning = into %3D), but httpx sends the request with the literal unencoded = in the URL path. The S3-compatible server receives the unencoded URL, computes its own canonical request using = (not %3D), and the signatures don't match.

Suggested fix

URL-encode the object key before embedding it in the URL, so that both botocore and httpx agree on the path:

from urllib.parse import quote

# In async_upload_data_to_s3(), when constructing the URL:
encoded_key = quote(batch_logging_element.s3_object_key, safe="/")

if self.s3_endpoint_url and self.s3_bucket_name:
    if self.s3_use_virtual_hosted_style:
        url = f"{protocol}{self.s3_bucket_name}.{endpoint_host}/{encoded_key}"
    else:
        url = self.s3_endpoint_url + "/" + self.s3_bucket_name + "/" + encoded_key
else:
    url = f"https://{self.s3_bucket_name}.s3.{self.s3_region_name}.amazonaws.com/{encoded_key}"

Alternatively, use urlsafe_b64encode with .rstrip("=") when generating the composite response ID in responses/utils.py:_build_responses_api_response_id() — consistent with how llms/base_llm/managed_resources/utils.py:292 and base_managed_resource.py:437 already do it.

Steps to Reproduce

import hashlib
import requests
from botocore.auth import SigV4Auth
from botocore.awsrequest import AWSRequest
from botocore.credentials import Credentials
import httpx

creds = Credentials('ACCESS_KEY', 'SECRET_KEY')
# Object key with base64 padding (==)
url = 'https://garage.example.com/bucket/key_with_base64==.json'
data = '{"test": 1}'
content_hash = hashlib.sha256(data.encode('utf-8')).hexdigest()

headers = {
    'Content-Type': 'application/json',
    'x-amz-content-sha256': content_hash,
}

req = requests.Request('PUT', url, data=data, headers=headers)
prepped = req.prepare()

aws_request = AWSRequest(
    method=prepped.method,
    url=prepped.url,
    data=prepped.body,
    headers=prepped.headers,
)

SigV4Auth(creds, 's3', 'region').add_auth(aws_request)

# botocore signed against canonical path: /bucket/key_with_base64%3D%3D.json
# httpx sends the request to: /bucket/key_with_base64==.json
# The server sees ==, computes signature with ==, doesn't match %3D%3D → 403
response = httpx.put(url, data=data, headers=dict(aws_request.headers.items()))
print(response.status_code)  # 403

Same test with a key that has no = succeeds with 200 OK.

Relevant log output

LiteLLM:ERROR: s3_v2.py:380 - Error uploading to s3: Client error '403 Forbidden' for url
'https://garage.example.com/litellm-usage/2026-03-25/time-18-07-17-208437_resp_bGl0ZWxsbT...OQ==.json'


Garage's error response:

<Error>
  <Code>AccessDenied</Code>
  <Message>Forbidden: Invalid signature</Message>
  <Resource>/litellm-usage/2026-03-25/time-18-07-17-208437_resp_bGl0ZWxsbT...OQ==.json</Resource>
</Error>

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

1.82.3

Twitter / LinkedIn details

N/A

extent analysis

Fix Plan

To resolve the issue with the s3_v2 logging callback failing due to an invalid signature when uploading to S3-compatible endpoints, follow these steps:

  1. URL-encode the object key: Before constructing the URL, encode the object key using urllib.parse.quote() to ensure that both botocore and httpx agree on the path.
  2. Modify the async_upload_data_to_s3() function: Update the function to use the encoded object key when constructing the URL.

Example code:

from urllib.parse import quote

# In async_upload_data_to_s3(), when constructing the URL:
encoded_key = quote(batch_logging_element.s3_object_key, safe="/")

if self.s3_endpoint_url and self.s3_bucket_name:
    if self.s3_use_virtual_hosted_style:
        url = f"{protocol}{self.s3_bucket_name}.{endpoint_host}/{encoded_key}"
    else:
        url = self.s3_endpoint_url + "/" + self.s3_bucket_name + "/" + encoded_key
else:
    url = f"https://{self.s3_bucket_name}.s3.{self.s3_region_name}.amazonaws.com/{encoded_key}"

Alternatively, you can use urlsafe_b64encode with .rstrip("=") when generating the composite response ID in responses/utils.py:_build_responses_api_response_id().

Verification

To verify that the fix worked, test the upload with a key that contains = characters. The upload should succeed with a 200 OK status code.

Extra Tips

  • Make sure to test the fix with different types of object keys, including those with and without = characters.
  • Consider adding additional logging or error handling to detect and handle any potential issues with the upload process.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: S3 v2 callback: SigV4 signature mismatch when object keys contain `=` (base64 padding) with S3-compatible endpoints [1 pull requests, 1 participants]