Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Security]: /metrics endpoint default-unauthenticated exposes multi-tenant PII in production deployments [3 pull requests, 6 comments, 4 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#24530•Fetched 2026-04-08 01:27:19
View on GitHub
Comments
6
Participants
4
Timeline
13
Reactions
0
Author
Aitema-gmbhAitema-gmbh
Participants
Aitema-gmbhAitema-gmbhAsteriskZuoAsteriskZuoDmitriyAlergantDmitriyAlergantzhaog100zhaog100
Timeline (top)
commented ×4cross-referenced ×3referenced ×3labeled ×1

The /metrics Prometheus endpoint is unauthenticated by default and exposes sensitive multi-tenant data in production LiteLLM Proxy deployments. While an opt-in authentication mechanism exists (require_auth_for_metrics_endpoint: true), the insecure default leads to widespread real-world exposure.

This is a follow-up to #13644, which was auto-closed by the stale bot as NOT_PLANNED without any maintainer response — despite multiple users confirming the security impact.

Root Cause

The /metrics endpoint is registered without any authentication middleware by default. The existing require_auth_for_metrics_endpoint setting is:

  1. Opt-in rather than opt-out (insecure by default)
  2. Not prominently documented in deployment guides
  3. Not enabled in the default/example configurations

This violates the principle of secure by default — especially critical for a multi-tenant proxy handling API credentials.

PR fix notes

PR #24600: security: Make /metrics endpoint authentication required by default (#24530)

Description (problem / solution / changelog)

🚨 Security Vulnerability Fix

Severity: High (CVSS 7.5)
Issue: #24530

Problem

The /metrics endpoint was unauthenticated by default, exposing sensitive multi-tenant PII:

Data ExposedImpact
Hashed API keys + aliasesCredential mapping
Team names + employee emailsGDPR-relevant PII
Client IP addressesInfrastructure topology
User agents with workflow IDsAutomation reverse-engineering
Cross-tenant patternsCompetitive intelligence

Root Cause: require_auth_for_metrics_endpoint defaulted to False.

Solution

Changed default from False to True:

# litellm/__init__.py
require_auth_for_metrics_endpoint: Optional[bool] = True  # Was: False

Impact

  • ✅ Prevents unauthenticated PII exposure
  • ✅ Secure-by-default principle
  • ✅ Backward compatible (opt-out available)

Migration

Deployments needing unauthenticated metrics can opt-out:

litellm_settings:
  allow_unauthenticated_metrics: true

CVSS 3.1

AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N (7.5 High)

Closes #24530

Changed files

  • litellm/__init__.py (modified, +2/-1)

PR #24895: 🔒 Security: Enable authentication for /metrics endpoint by default

Description (problem / solution / changelog)

Summary

Fixes #24530 - /metrics endpoint exposed multi-tenant PII by default

Security Vulnerability

The /metrics Prometheus endpoint was unauthenticated by default, exposing:

  • Team aliases (company names and employee emails - PII)
  • User email addresses
  • Client IP addresses
  • User agent strings (revealing Azure Logic Apps workflow IDs)
  • Cross-tenant request patterns

CVSS 3.1: 7.5 (High) - AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N

Changes

  • Changed require_auth_for_metrics_endpoint default from False to True
  • Updated middleware documentation to reflect secure-by-default behavior

Breaking Change

⚠️ This is a breaking change. Users who want unauthenticated /metrics must now explicitly set: ```yaml litellm_settings: require_auth_for_metrics_endpoint: false ```

Rationale

Following the principle of secure by default:

  • Multi-tenant proxy handling API credentials should not expose data by default
  • The existing opt-in approach led to widespread real-world exposure (see #24530 for details)
  • Security should be the default, with explicit opt-out for advanced users

Test Plan

  • Local testing: Verified default behavior requires auth
  • Opt-out testing: Verified require_auth_for_metrics_endpoint: false disables auth
  • Integration testing: Recommend maintainers test in staging environment

Related Issues

  • #13644 (original report, auto-closed without fix)
  • #24530 (this vulnerability report)

Mitigation for Users

If you need unauthenticated /metrics (e.g., behind a firewall): ```yaml litellm_settings: require_auth_for_metrics_endpoint: false ```

Alternative Considered

Also considered sanitizing PII labels, but authentication is a more robust solution.

Changed files

  • litellm/__init__.py (modified, +3/-1)
  • litellm/integrations/prometheus.py (modified, +36/-16)
  • litellm/proxy/management_endpoints/key_management_endpoints.py (modified, +8/-2)
  • litellm/proxy/middleware/prometheus_auth_middleware.py (modified, +3/-4)
  • litellm/proxy/proxy_server.py (modified, +5/-10)
  • litellm/responses/litellm_completion_transformation/transformation.py (modified, +3/-3)
  • litellm/types/integrations/prometheus.py (modified, +10/-8)

PR #24977: Security: Fix metrics endpoint authentication & PII exposure (CVSS 7.5)

Description (problem / solution / changelog)

🔒 Security Fix: Metrics Endpoint Authentication & PII Removal

🚨 Vulnerability (CVSS 7.5 - HIGH)

Issue: LiteLLM metrics endpoint exposed sensitive data without authentication

  • PII labels in Prometheus metrics leaked: team_alias, user_email, client_ip
  • CVSS Score: 7.5 (High)

✅ Three-Phase Defense-in-Depth Fix

Phase 1: Enable authentication by default

  • Set require_auth_for_metrics_endpoint = True by default
  • Ensures metrics endpoint is protected without manual configuration

Phase 2: Remove PII labels

  • Remove: team_alias, user_email, client_ip, user_agent
  • Keep only safe labels: team_id, user_id, model, status

Phase 3: Add startup warning

  • Warning displayed if auth is disabled
  • Alert operators to security risk

📝 Changes

  1. litellm/init.py

    • Set require_auth_for_metrics_endpoint = True by default

  2. litellm/integrations/prometheus.py

    • Remove PII labels from all metrics
    • Keep only necessary labels for monitoring

  3. litellm/proxy/proxy_server.py

    • Add startup warning message

  4. litellm/proxy/middleware/prometheus_auth_middleware.py

    • Authentication middleware

  5. litellm/types/integrations/prometheus.py

    • Type definitions

✅ Testing

  • All existing tests pass
  • Black formatted (v23.12.0)
  • No breaking changes

📚 References

  • Fixes #24530

Security Impact: This fix prevents unauthorized access to sensitive metrics and removes PII from monitoring systems.

Changed files

  • litellm/__init__.py (modified, +1/-1)
  • litellm/integrations/prometheus.py (modified, +21/-7)
  • litellm/proxy/middleware/prometheus_auth_middleware.py (modified, +3/-4)
  • litellm/responses/litellm_completion_transformation/transformation.py (modified, +3/-3)
  • litellm/types/integrations/prometheus.py (modified, +10/-8)
RAW_BUFFERClick to expand / collapse

Summary

The /metrics Prometheus endpoint is unauthenticated by default and exposes sensitive multi-tenant data in production LiteLLM Proxy deployments. While an opt-in authentication mechanism exists (require_auth_for_metrics_endpoint: true), the insecure default leads to widespread real-world exposure.

This is a follow-up to #13644, which was auto-closed by the stale bot as NOT_PLANNED without any maintainer response — despite multiple users confirming the security impact.

Vulnerability Details

Type: Information Disclosure / Tenant Isolation Bypass
Default behavior: /metrics endpoint serves ~2MB of Prometheus metrics to any unauthenticated HTTP request
CVSS 3.1: 7.5 (High) — AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:N/A:N

Data Exposed via litellm_proxy_total_requests_metric Labels

The Prometheus metrics include per-request labels that leak the following data for every tenant:

LabelSensitive Data
hashed_api_keySHA-256 hash of customer API keys
api_key_aliasHuman-readable key names (often containing company names)
teamTeam UUIDs
team_aliasCompany names and employee email addresses (PII)
end_userEnd-user identifiers
userUser identifiers with email addresses
requester_ip_addressClient IP addresses revealing infrastructure topology
user_agentFull user-agent strings (exposing Azure Logic Apps workflow IDs, automation tools)
modelAI models accessed per customer
model_idInternal model UUIDs

Real-World Impact (Verified)

During authorized security testing, we identified production LiteLLM Proxy deployments where the unauthenticated /metrics endpoint exposed:

  • 15+ customer tenants with complete operational telemetry
  • Corporate email addresses in team_alias fields (GDPR-relevant PII)
  • Client IP addresses enabling infrastructure geolocation and cloud provider identification
  • Azure Logic Apps workflow IDs in user-agent strings, enabling reverse engineering of customer automation infrastructure
  • API key hashes with aliases mapping to specific companies
  • Cross-tenant request patterns enabling competitive intelligence gathering

A single unauthenticated GET request returns the complete dataset for all tenants.

Root Cause

The /metrics endpoint is registered without any authentication middleware by default. The existing require_auth_for_metrics_endpoint setting is:

  1. Opt-in rather than opt-out (insecure by default)
  2. Not prominently documented in deployment guides
  3. Not enabled in the default/example configurations

This violates the principle of secure by default — especially critical for a multi-tenant proxy handling API credentials.

Recommendation

  1. Make authentication the default for /metrics — require explicit opt-out (allow_unauthenticated_metrics: true) rather than opt-in authentication
  2. Sanitize metric labels — remove or hash PII fields (team_alias, user, requester_ip_address, user_agent) from Prometheus labels
  3. Add deployment warnings — surface a startup warning when /metrics is exposed without authentication
  4. Reopen #13644 — the stale-bot closure without maintainer response was inappropriate for a security issue

Timeline

  • 2025-08-15: Issue #13644 reported by @nekomeowww — no maintainer response
  • 2025-10-03: Additional users confirm severity, request fix — no response
  • 2025-10-17: User explicitly asks for update — no response
  • 2026-01-16: Auto-closed as stale (NOT_PLANNED) by bot
  • 2026-03: Real-world multi-tenant PII exposure verified during authorized security engagement

Disclosure

This report is submitted in good faith as responsible disclosure. No customer-identifying data from affected deployments is included. All testing was conducted under authorized bug bounty engagement scope.

Reported by aitema GmbH — AI Security Research

extent analysis

Fix Plan

To address the security issue, we will implement the following changes:

  • Make authentication the default for the /metrics endpoint
  • Sanitize metric labels to remove or hash PII fields
  • Add deployment warnings when /metrics is exposed without authentication

Step-by-Step Solution

  1. Enable authentication by default: Set require_auth_for_metrics_endpoint to true by default in the configuration file.
  2. Sanitize metric labels: Remove or hash PII fields (team_alias, user, requester_ip_address, user_agent) from Prometheus labels.
  3. Add deployment warnings: Surface a startup warning when /metrics is exposed without authentication.

Example Code Changes

# Enable authentication by default
require_auth_for_metrics_endpoint = True

# Sanitize metric labels
def sanitize_labels(labels):
    sensitive_fields = ['team_alias', 'user', 'requester_ip_address', 'user_agent']
    for field in sensitive_fields:
        if field in labels:
            # Remove or hash the field value
            labels[field] = hash(labels[field])
    return labels

# Add deployment warnings
if not require_auth_for_metrics_endpoint:
    print("Warning: /metrics endpoint is exposed without authentication")

Verification

To verify that the fix worked, check the following:

  • The /metrics endpoint requires authentication by default
  • PII fields are removed or hashed from Prometheus labels
  • A startup warning is surfaced when /metrics is exposed without authentication

Extra Tips

  • Ensure that the require_auth_for_metrics_endpoint setting is prominently documented in deployment guides
  • Consider implementing additional security measures, such as rate limiting and IP blocking, to prevent abuse of the /metrics endpoint.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING