litellm - 💡(How to fix) Fix [Bug]: Ghost models with --num_workers > 1 – Deleted models are not cleared from other workers' local cache (Redis Pub/Sub sync issue) [1 participants]

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

What happened: When running the LiteLLM Proxy with multiple workers (--num_workers 4) and a Redis cache configured, deleting a model added via API (/model/new) via the /model/delete API does not invalidate the local in-memory cache of the other workers.

The worker handling the delete request successfully removes the model from the Postgres DB and its own RAM. However, it seems no Pub/Sub broadcast is sent (or received) via Redis to tell the other workers to drop the model. As a result API calls to /model/info start "flapping" randomly, showing the deleted model or hiding it, depending on which worker handles the request.

Note: This was tested with both Valkey 8.0.1 and official Redis 8.6.3. The issue persists regardless of the Redis distribution.

What did you expect to happen? When /model/delete is called, the proxy should broadcast a cache invalidation event via Redis to all other Uvicorn workers immediately, purging the model from all local memories so the Proxy state remains consistent.

Steps to Reproduce

1. Start LiteLLM proxy with Redis and 4 workers: Configure config.yaml to use Redis cache and start the container: exec litellm --config /app/config.yaml --port 4000 --num_workers 4

litellm_settings:
  cache: True
  cache_params:
    type: redis
    host: redis-host
    port: 6379
    password: LiteLLM_REDIS_PW

general_settings:
  master_key: LiteLLM_MASTER_KEY
  proxy_batch_write_at: 60
  database_connection_pool_limit: 10
  disable_error_logs: True
  background_health_checks: True
  health_check_interval: 60
  health_check_details: False
  store_model_in_db: True

router_settings:
  routing_strategy: simple-shuffle
  enable_pre_call_checks: true

2. Add a model via API:

curl -X POST 'http://<LITELLM_URL>/model/new' \
-H 'Authorization: Bearer <YOUR_KEY>' \
-H 'Content-Type: application/json' \
-d '{
  "model_name": "test-model",
  "litellm_params": {
    "model": "openai/test",
    "api_base": "http://test:8000/v1",
    "api_key": "none"
  },
  "model_info": {
    "id": "test-model",
    "mode": "embedding"
  }
}'

3. Delete the model via API:

curl -X POST 'http://<LITELLM_URL>/model/delete' \
-H 'Authorization: Bearer <YOUR_KEY>' \
-H 'Content-Type: application/json' \
-d '{
  "id": "test-model"
}'

4. Spam the /health or /model/info endpoint:

curl -X GET 'http://<LITELLM_URL>/health' -H 'Authorization: Bearer <YOUR_KEY>'

curl -X GET 'http://<LITELLM_URL>/model/info' -H 'Authorization: Bearer <YOUR_KEY>'

Observe the output flipping between unhealthy_count: 0 and listing the deleted ghost model.

Relevant log output

# The delete request is processed successfully:
INFO:     "POST /model/delete HTTP/1.1" 200 OK

# The other workers poll the DB, see 0 models, but do NOT clear their local cache:
LiteLLM Proxy:DEBUG: proxy_server.py:4256 - len new_models: 0
LiteLLM Proxy:DEBUG: proxy_server.py:4256 - len new_models: 0

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.83.14-stable

Twitter / LinkedIn details

No response