Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: Virtual key/generate returns 403 with stale team MCP access groups immediately after team update [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#25286•Fetched 2026-04-08 03:02:09
View on GitHub
Comments
0
Participants
1
Timeline
2
Reactions
0
Author
vtarbinskyivtarbinskyi
Participants
vtarbinskyivtarbinskyi
Timeline (top)
labeled ×2

Error Message

12:05:30 - LiteLLM Proxy:DEBUG: key_management_endpoints.py:1232 - entered /key/generate 12:05:30 - LiteLLM Proxy:ERROR: key_management_endpoints.py:1326 - litellm.proxy.proxy_server.generate_key_fn(): Exception occured - 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."} Traceback (most recent call last): � File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 1318, in generate_key_fn return await _common_key_generation_helper( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ � File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 688, in _common_key_generation_helper await validate_key_mcp_servers_against_team( ...<2 lines>... ) � File "/usr/lib/python3.13/site-packages/litellm/proxy/management_helpers/object_permission_utils.py", line 378, in validate_key_mcp_servers_against_team raise HTTPException( ...<2 lines>... ) �fastapi.exceptions.HTTPException: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."} �12:05:30 - LiteLLM Proxy:ERROR: utils.py:5285 - Exception: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."} Traceback (most recent call last): � File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 1318, in generate_key_fn return await _common_key_generation_helper( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...<4 lines>... ) ^ � File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 688, in _common_key_generation_helper await validate_key_mcp_servers_against_team( ...<2 lines>... ) � File "/usr/lib/python3.13/site-packages/litellm/proxy/management_helpers/object_permission_utils.py", line 378, in validate_key_mcp_servers_against_team raise HTTPException( ...<2 lines>... ) �fastapi.exceptions.HTTPException: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."} INFO: 10.89.5.1:35134 - "POST /key/generate HTTP/1.1" 403 Forbidden

Fix Action

Fix / Workaround

  1. Create a team
  2. Set team initial object_permission.mcp_access_groups to a single group, e.g. ["G1"] (POST /team/update or /team/new as applicable).
  3. POST /key/generate for that team_id with permissions scoped to G1 only — succeeds (baseline key).
  4. POST /team/update — merge in a second group, e.g. mcp_access_groups: ["G1","G2"].
  5. Immediately POST /key/generate with same team_id and key permissions requesting G2.
  6. Behavior: 403 — error text lists Team allows without G2 (stale).
  7. Retry POST /key/generate after ~60s with the same payload — succeeds (or set DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0 and repeat step 6).

PR fix notes

PR #25552: fix(proxy): cache invalidation double-hashes token in bulk update and key rotation

Description (problem / solution / changelog)

Summary

  • Two remaining hash_token() calls in cache invalidation paths double-hash pre-hashed token IDs, causing cache invalidation to silently fail
  • Fix: use _hash_token_if_needed() (which passes pre-hashed tokens through unchanged), matching the pattern established by PR #24969

Root cause

When /key/update or /key/bulk_update is called with a pre-hashed token ID (not an sk- prefixed key), hash_token() produces a hash-of-hash that does not match the actual cache key. The _delete_cache_key_object call deletes nothing, and the stale cached key object (with outdated fields) remains.

This is compounded by update_cache() in proxy_server.py, which writes the stale cached key object back to user_api_key_cache with a fresh 60s TTL after every successful LLM request. As long as requests keep flowing, the stale entry never expires — the updated fields (e.g. max_budget, models, metadata) are never picked up.

PR #24969 fixed this in update_key_fn but missed two other call sites:

LocationLineBeforeAfter
_process_single_key_update1852hash_token(key_update_item.key)_hash_token_if_needed(key_update_item.key)
_execute_virtual_key_regeneration3724hash_token(key)_hash_token_if_needed(key)

Reproduction steps

  1. Create a virtual key and note the token hash from the response
  2. Update the key using the token hash (not the sk- key): 
    curl -X POST http://localhost:4000/key/update \
  -H "Authorization: Bearer sk-master" \
  -d '{"key": "<token_hash>", "max_budget": 100}'
  3. Verify via /key/info that the DB was updated (shows max_budget: 100)
  4. Send requests using the key — the budget is not enforced because the cached key object still has max_budget: null
  5. The stale cache entry persists indefinitely as long as requests keep flowing (each successful request refreshes the TTL via update_cache)

Related

  • #24969 — fixed the same bug in update_key_fn (merged April 3, 2026)
  • #25286 — similar stale-cache issue for team objects

Test plan

  • New test: test_process_single_key_update_cache_invalidation_with_token_hash — verifies the token hash is passed as-is (not double-hashed) to _delete_cache_key_object
  • Existing test_process_single_key_update continues to pass

Generated with Claude Code

Changed files

  • litellm/proxy/management_endpoints/key_management_endpoints.py (modified, +2/-2)
  • tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py (modified, +164/-0)

Code Example

12:05:30 - LiteLLM Proxy:DEBUG: key_management_endpoints.py:1232 - entered /key/generate
12:05:30 - LiteLLM Proxy:ERROR: key_management_endpoints.py:1326 - litellm.proxy.proxy_server.generate_key_fn(): Exception occured - 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
Traceback (most recent call last):
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 1318, in generate_key_fn
    return await _common_key_generation_helper(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 688, in _common_key_generation_helper
    await validate_key_mcp_servers_against_team(
    ...<2 lines>...
    )
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_helpers/object_permission_utils.py", line 378, in validate_key_mcp_servers_against_team
    raise HTTPException(
    ...<2 lines>...
    )
�fastapi.exceptions.HTTPException: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
�12:05:30 - LiteLLM Proxy:ERROR: utils.py:5285 - Exception: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
Traceback (most recent call last):
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 1318, in generate_key_fn
    return await _common_key_generation_helper(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 688, in _common_key_generation_helper
    await validate_key_mcp_servers_against_team(
    ...<2 lines>...
    )
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_helpers/object_permission_utils.py", line 378, in validate_key_mcp_servers_against_team
    raise HTTPException(
    ...<2 lines>...
    )
�fastapi.exceptions.HTTPException: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
INFO:     10.89.5.1:35134 - "POST /key/generate HTTP/1.1" 403 Forbidden
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

Searched for similar existing issues, haven't found anything, verified issue exists also on stable on 1.82.3 as well.

What happened?

After updating a team’s MCP access groups via POST /team/update, POST /key/generate can fail with 403 if called immediately afterward. The error says the requested MCP access groups are not allowed for the team, and the “Team allows” list in the message matches the pre-update state, not the merged permissions from the latest team/update. Retrying after ~1 minute succeeds. Setting DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0 avoids the issue, which suggests stale in-memory management/team state during key validation.

Steps to Reproduce

  1. Create a team
  2. Set team initial object_permission.mcp_access_groups to a single group, e.g. ["G1"] (POST /team/update or /team/new as applicable).
  3. POST /key/generate for that team_id with permissions scoped to G1 only — succeeds (baseline key).
  4. POST /team/update — merge in a second group, e.g. mcp_access_groups: ["G1","G2"].
  5. Immediately POST /key/generate with same team_id and key permissions requesting G2.
  6. Behavior: 403 — error text lists Team allows without G2 (stale).
  7. Retry POST /key/generate after ~60s with the same payload — succeeds (or set DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0 and repeat step 6).

Expected behavior key/generate should enforce MCP access groups against the current team permissions after a successful team/update, so update and key creation can run back-to-back without delay.

Actual behavior key/generate returns 403 with a message like: {"error":{"message":"{'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}","type":"internal_server_error","param":"None","code":"403"}}

Same request succeeds after a 1-minute delay or with DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0.

Relevant log output

12:05:30 - LiteLLM Proxy:DEBUG: key_management_endpoints.py:1232 - entered /key/generate
12:05:30 - LiteLLM Proxy:ERROR: key_management_endpoints.py:1326 - litellm.proxy.proxy_server.generate_key_fn(): Exception occured - 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
Traceback (most recent call last):
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 1318, in generate_key_fn
    return await _common_key_generation_helper(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 688, in _common_key_generation_helper
    await validate_key_mcp_servers_against_team(
    ...<2 lines>...
    )
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_helpers/object_permission_utils.py", line 378, in validate_key_mcp_servers_against_team
    raise HTTPException(
    ...<2 lines>...
    )
�fastapi.exceptions.HTTPException: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
�12:05:30 - LiteLLM Proxy:ERROR: utils.py:5285 - Exception: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
Traceback (most recent call last):
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 1318, in generate_key_fn
    return await _common_key_generation_helper(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_endpoints/key_management_endpoints.py", line 688, in _common_key_generation_helper
    await validate_key_mcp_servers_against_team(
    ...<2 lines>...
    )
�  File "/usr/lib/python3.13/site-packages/litellm/proxy/management_helpers/object_permission_utils.py", line 378, in validate_key_mcp_servers_against_team
    raise HTTPException(
    ...<2 lines>...
    )
�fastapi.exceptions.HTTPException: 403: {'error': "Key requests MCP access groups not allowed by team '9c645ef3-5c43-49eb-a1da-46092dffe823': ['test2']. Team allows: ['test1']."}
INFO:     10.89.5.1:35134 - "POST /key/generate HTTP/1.1" 403 Forbidden

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.83.4-nightly

Twitter / LinkedIn details

No response

extent analysis

TL;DR

Setting DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0 can workaround the issue of POST /key/generate failing with a 403 error immediately after updating a team's MCP access groups.

Guidance

  • The issue seems to be caused by stale in-memory management/team state during key validation, which can be mitigated by setting DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0.
  • To verify the fix, retry the POST /key/generate request after setting DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0 and check if it succeeds.
  • Another approach is to introduce a delay of around 1 minute between the POST /team/update and POST /key/generate requests, as the issue seems to resolve itself after this delay.
  • Review the code in object_permission_utils.py and key_management_endpoints.py to understand how the team permissions are being validated and updated, which may help in identifying a more permanent fix.

Notes

  • The workaround of setting DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0 may have performance implications, so it's essential to weigh the trade-offs before implementing it as a permanent solution.
  • The issue is specific to the Proxy component of LiteLLM, and the version being used is v1.83.4-nightly, which may be relevant when investigating or reporting the issue further.

Recommendation

  • Apply the workaround by setting DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL=0, as it provides a straightforward way to mitigate the issue without requiring significant code changes or upgrades.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#memory management#file not found#serialization error#model compatibility

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: Virtual key/generate returns 403 with stale team MCP access groups immediately after team update [1 pull requests, 1 participants]