litellm - ✅(Solved) Fix [Bug]: Helm 1.1.0 standalone DB secret drifts from postgresql.existingSecret and causes CrashLoop [1 pull requests, 1 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#27173Fetched 2026-05-06 06:15:48
View on GitHub
Comments
1
Participants
1
Timeline
2
Reactions
0
Author
Participants
Timeline (top)
commented ×1cross-referenced ×1

Error Message

PrismaClientInitializationError: Authentication failed against database server ... (P1000)

Fix Action

Fix / Workaround

  • I have searched existing issues and this appears related to (but not resolved by):
    • #13544
    • #21716

Temporary workaround that unblocks us

Patch litellm-dbcredentials to match litellm-credentials.password/postgres-password, then restart LiteLLM pod:

PR fix notes

PR #27176: [Fix] Helm: honor external DB secret in standalone mode

Description (problem / solution / changelog)

Summary

  • unify standalone DB credential secret resolution with clear precedence: db.dbCredentialsSecretName -> postgresql.auth.existingSecret -> <fullname>-dbcredentials
  • update deployment and migration job to read DB username/password from the resolved secret instead of hardcoded/independent sources
  • only render fallback secret-dbcredentials when no external DB secret is configured, preventing credential drift and startup auth failures

Why

Standalone mode could drift to chart-managed *-dbcredentials even when users configured external secrets, which caused mismatched DB credentials and Prisma auth failures (P1000) in real deployments.

Validation

  • rendered templates to confirm secret resolution and fallback-secret gating behavior
  • redeployed via ArgoCD on a clean k3s environment (delete app/namespace, repackage chart, repush, recreate app)
  • verified runtime outcomes:
    • litellm, postgresql, redis all reached Running
    • litellm-dbcredentials was not created when external secret config was provided
    • no P1000 DB auth failure observed in LiteLLM logs

Related

  • Closes #27173
  • Related: #13544, #21716

Changed files

  • deploy/charts/litellm-helm/templates/_helpers.tpl (modified, +29/-0)
  • deploy/charts/litellm-helm/templates/deployment.yaml (modified, +5/-5)
  • deploy/charts/litellm-helm/templates/migrations-job.yaml (modified, +15/-1)
  • deploy/charts/litellm-helm/templates/secret-dbcredentials.yaml (modified, +2/-2)
  • deploy/charts/litellm-helm/values.yaml (modified, +16/-1)

Code Example

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: litellm
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: agents
  source:
    repoURL: registry.registry.svc.cluster.local:5000/helm-charts
    chart: litellm-helm
    targetRevision: 1.1.0
    helm:
      releaseName: litellm
      valuesObject:
        image:
          repository: m.daocloud.io/ghcr.io/berriai/litellm
          tag: main-v1.83.14-stable
        migrationJob:
          enabled: false
        db:
          useExisting: false
          deployStandalone: true
          disableSchemaUpdate: true
          dbCredentialsSecretName: litellm-credentials
        postgresql:
          auth:
            username: litellm
            database: litellm
            existingSecret: litellm-credentials
            secretKeys:
              userPasswordKey: postgres-password
              adminPasswordKey: postgres-password
        redis:
          enabled: true
          auth:
            enabled: true
            existingSecret: litellm-credentials
            existingSecretPasswordKey: redis-password

---

PrismaClientInitializationError: Authentication failed against database server ... (P1000)

---

PASS=$(kubectl get secret -n agents litellm-credentials -o jsonpath='{.data.password}')
kubectl patch secret -n agents litellm-dbcredentials --type merge -p "{\"data\":{\"password\":\"$PASS\",\"postgres-password\":\"$PASS\"}}"
kubectl delete pod -n agents -l app.kubernetes.io/name=litellm
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched existing issues and this appears related to (but not resolved by):
    • #13544
    • #21716

What happened?

When deploying litellm-helm 1.1.0 via ArgoCD with bundled Postgres (db.deployStandalone=true) and postgresql.auth.existingSecret set, LiteLLM still uses/creates litellm-dbcredentials in a way that drifts from the intended secret.

In our test, litellm-credentials had the correct password, but litellm-dbcredentials ended up with a different/default value (e.g. NoTaGrEaTpAsSwOrD). LiteLLM then failed Prisma DB auth (P1000) and stayed in CrashLoopBackOff.

What I expected

  • If db.dbCredentialsSecretName and postgresql.auth.existingSecret are both set to the same Secret, all DB auth values should remain consistent.
  • The chart should not render or preserve conflicting DB credentials that make LiteLLM fail against the bundled Postgres.

Environment

  • Chart: litellm-helm:1.1.0
  • Deployment method: ArgoCD Application
  • Cluster: k3s
  • LiteLLM image: ghcr.io/berriai/litellm:main-v1.83.14-stable
  • Postgres image: bitnamilegacy/postgresql

Steps to Reproduce

  1. Create litellm-credentials secret with consistent values (password, postgres-password, redis-password, etc.).
  2. Deploy with:
    • db.useExisting: false
    • db.deployStandalone: true
    • db.dbCredentialsSecretName: litellm-credentials
    • postgresql.auth.existingSecret: litellm-credentials
  3. Sync ArgoCD app and wait for pods.
  4. Observe LiteLLM pod logs and secret values.

Relevant ArgoCD Application YAML (sanitized minimal repro)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: litellm
spec:
  project: default
  destination:
    server: https://kubernetes.default.svc
    namespace: agents
  source:
    repoURL: registry.registry.svc.cluster.local:5000/helm-charts
    chart: litellm-helm
    targetRevision: 1.1.0
    helm:
      releaseName: litellm
      valuesObject:
        image:
          repository: m.daocloud.io/ghcr.io/berriai/litellm
          tag: main-v1.83.14-stable
        migrationJob:
          enabled: false
        db:
          useExisting: false
          deployStandalone: true
          disableSchemaUpdate: true
          dbCredentialsSecretName: litellm-credentials
        postgresql:
          auth:
            username: litellm
            database: litellm
            existingSecret: litellm-credentials
            secretKeys:
              userPasswordKey: postgres-password
              adminPasswordKey: postgres-password
        redis:
          enabled: true
          auth:
            enabled: true
            existingSecret: litellm-credentials
            existingSecretPasswordKey: redis-password

Relevant log output

PrismaClientInitializationError: Authentication failed against database server ... (P1000)

(After this, LiteLLM repeatedly restarts.)

Temporary workaround that unblocks us

Patch litellm-dbcredentials to match litellm-credentials.password/postgres-password, then restart LiteLLM pod:

PASS=$(kubectl get secret -n agents litellm-credentials -o jsonpath='{.data.password}')
kubectl patch secret -n agents litellm-dbcredentials --type merge -p "{\"data\":{\"password\":\"$PASS\",\"postgres-password\":\"$PASS\"}}"
kubectl delete pod -n agents -l app.kubernetes.io/name=litellm

After this patch, LiteLLM starts successfully and serves traffic.

Request

Could maintainers clarify intended precedence and fix chart behavior so standalone mode does not generate/retain conflicting DB credentials when existingSecret is explicitly provided?

extent analysis

TL;DR

The issue can be fixed by ensuring that the litellm-dbcredentials secret is updated to match the litellm-credentials secret when using standalone Postgres deployment with an existing secret.

Guidance

  • Verify that the db.dbCredentialsSecretName and postgresql.auth.existingSecret values are set to the same secret name (litellm-credentials) in the ArgoCD application YAML.
  • Check the chart's behavior for generating or preserving conflicting DB credentials when existingSecret is provided.
  • Consider updating the chart to prioritize the existingSecret values over any generated credentials.
  • Use the provided temporary workaround to patch the litellm-dbcredentials secret and restart the LiteLLM pod as a temporary solution.

Example

The temporary workaround provided in the issue can be used to patch the litellm-dbcredentials secret:

PASS=$(kubectl get secret -n agents litellm-credentials -o jsonpath='{.data.password}')
kubectl patch secret -n agents litellm-dbcredentials --type merge -p "{\"data\":{\"password\":\"$PASS\",\"postgres-password\":\"$PASS\"}}"
kubectl delete pod -n agents -l app.kubernetes.io/name=litellm

Notes

The issue seems to be related to the chart's behavior when using standalone Postgres deployment with an existing secret. The maintainers' clarification on the intended precedence and a fix for the chart behavior are required to resolve the issue permanently.

Recommendation

Apply the temporary workaround to patch the litellm-dbcredentials secret and restart the LiteLLM pod, as it provides a working solution until the chart behavior is fixed.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING