litellm - 💡(How to fix) Fix Helm chart's migration job defaults to v1 prisma resolver, defeating the v2 fix from #26194 [1 participants]

Problem

PR #26194 added --use_v2_migration_resolver to address the diff-and-force schema-thrashing bug tracked in #22998. The flag works as advertised. However, the bundled Helm chart (deploy/charts/litellm-helm) hasn't been updated to take advantage of it:

• deploy/charts/litellm-helm/templates/migrations-job.yaml runs command: ["python", "litellm/proxy/prisma_migration.py"].
• litellm/proxy/prisma_migration.py calls run_server(["--skip_server_startup"], standalone_mode=False) with no flag and no envvar fallback (the click option at proxy_cli.py ~line 581 has no envvar=).
• There is no command:/args:/extraArgs: value on the chart's migration job to override this.

Net effect: any operator using the official chart silently gets the v1 resolver, even on the LiteLLM versions that ship the v2 fix. The bug from #22998 keeps reproducing under the chart's default rolling-deploy strategy.

Concrete repro (just lived this)

Bumped the chart's image.tag from main-v1.82.3 → v1.83.14.rc.1. The migration Helm-hook job ran and reported success. _prisma_migrations showed all 11 new migrations in the batch as finished_at = <deploy time> with no rolled_back_at. The proxy then crashed in a request loop with errors like:

prisma.errors.DataError: The column `LiteLLM_ObjectPermissionTable.mcp_toolsets` does not exist in the current database.
prisma.errors.DataError: The column `LiteLLM_MCPServerTable.instructions` does not exist in the current database.
prisma.errors.DataError: The column `LiteLLM_TeamMembership.total_spend` does not exist in the current database.

Audit confirmed four entire tables (LiteLLM_MCPToolsetTable, LiteLLM_MemoryTable, LiteLLM_AdaptiveRouterState, LiteLLM_AdaptiveRouterSession) and three columns from migrations recorded as applied were missing from the live schema — exactly the v1 diff-and-force failure mode #22998 describes. Manually running the same image with --use_v2_migration_resolver reported "no pending migrations to apply" because v1 had already stamped the ledger; recovery required replaying the missing DDL by hand.

The startup banner literally points at the fix:

Using default (v1) migration resolver. If your deployment has seen schema thrashing during rolling deploys, try --use_v2_migration_resolver (safer: avoids the diff-and-force recovery that caused the thrash).

But there's no chart-level path to take that advice without forking the chart or piggybacking on extraInitContainers, which is what we ended up doing.

Suggested fixes (any one would close this)

1. Forward extra args in prisma_migration.py — append sys.argv[1:] to the args list passed to run_server. Then add a migrationJob.extraArgs value on the chart that gets spliced into the container's args:. Keeps v1 default; one-line opt-in for chart users.
2. Add an envvar binding to the click option — envvar="USE_V2_MIGRATION_RESOLVER" on --use_v2_migration_resolver. I see PR #26194's discussion went the other way deliberately, but the chart use case argues for it: chart users configure via env, not argv. If you'd rather keep the flag explicit, option 1 is fine.
3. Flip the chart's default to v2 — bake --use_v2_migration_resolver into prisma_migration.py or the chart command. The v1-stays-default rationale (operators with corrupted ledgers from prior thrashing) applies to direct CLI users, not fresh chart deploys; new chart installs are never in that state.

Happy to send a PR if you have a preference on which path. Most surgical is (1).

Related

• #22998 (root bug)
• #26194 (added v2 resolver)
• #26712 (recent v2 polish — pooler URL handling)

Environment

• Chart: litellm-helm 1.82.3 (current latest)
• Image: ghcr.io/berriai/litellm-database:v1.83.14.rc.1
• DB: PostgreSQL 16 (AWS RDS)