Issue Station
Sign In
Tools hereToolbox

litellm - 💡(How to fix) Fix [Bug]: litellm_proxy_extras migrations recorded as applied but columns not created — causes silent failures and 500 errors on upgrade [5 comments, 5 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#22998•Fetched 2026-04-08 00:38:56
View on GitHub
Comments
5
Participants
5
Timeline
6
Reactions
2
Author
madhu19991madhu19991
Participants
a-grellmanna-grellmannemerzonemerzonfishingpvaluesfishingpvaluesGamewalkerGamewalkermadhu19991madhu19991
Timeline (top)
commented ×5subscribed ×1

Error Message

prisma.errors.DataError: The column LiteLLM_VerificationToken.project_id does not exist in the current database.

Root Cause

The extras migrations were recorded as applied in _prisma_migrations (i.e. prisma migrate deploy reported "No pending migrations to apply") but the actual ALTER TABLE SQL was never executed against the database. The schema and DB were silently out of sync.

Fix Action

Workaround

Applied missing columns manually via prisma db execute:

-- LiteLLM_VerificationToken
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "project_id" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_active" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_count" INTEGER DEFAULT 0;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "auto_rotate" BOOLEAN DEFAULT false;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_interval" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_rotation_at" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "key_rotation_at" TIMESTAMP(3);

-- LiteLLM_MCPServerTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN IF NOT EXISTS "spec_path" TEXT;

Code Example

prisma.errors.DataError: The column `LiteLLM_VerificationToken.project_id` does not exist in the current database.

---

prisma.errors.DataError: The column `LiteLLM_VerificationToken.last_active` does not exist in the current database.

---

prisma.errors.DataError: The column `LiteLLM_MCPServerTable.spec_path` does not exist in the current database.

---

-- LiteLLM_VerificationToken
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "project_id" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_active" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_count" INTEGER DEFAULT 0;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "auto_rotate" BOOLEAN DEFAULT false;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_interval" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_rotation_at" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "key_rotation_at" TIMESTAMP(3);

-- LiteLLM_MCPServerTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN IF NOT EXISTS "spec_path" TEXT;
RAW_BUFFERClick to expand / collapse

What happened?

After upgrading to v1.81.14-stable (ghcr.io/berriai/litellm-non_root), the proxy started returning 500 Internal Server Error on /v2/login and silently returning empty arrays from /v1/mcp/server. Both failures were caused by missing DB columns that should have been added by litellm_proxy_extras migrations.

Root cause

The extras migrations were recorded as applied in _prisma_migrations (i.e. prisma migrate deploy reported "No pending migrations to apply") but the actual ALTER TABLE SQL was never executed against the database. The schema and DB were silently out of sync.

Symptoms

Login broken (500 on /v2/login):

prisma.errors.DataError: The column `LiteLLM_VerificationToken.project_id` does not exist in the current database.

Then after fixing project_id:

prisma.errors.DataError: The column `LiteLLM_VerificationToken.last_active` does not exist in the current database.

MCP servers silently returning empty list:

prisma.errors.DataError: The column `LiteLLM_MCPServerTable.spec_path` does not exist in the current database.

Note: get_all_mcp_servers() in db.py catches the exception at debug level and returns [] — so no error appears in logs, just an empty API response despite servers being in the DB.

Missing columns and their migrations

All were in litellm_proxy_extras and all were recorded as applied in _prisma_migrations:

TableMissing Column(s)Migration
LiteLLM_VerificationTokenproject_id20260221000000_ensure_project_id_verification_token
LiteLLM_VerificationTokenlast_active, rotation_count, auto_rotate, rotation_interval, last_rotation_at, key_rotation_at20260218231534, 20250926194702
LiteLLM_MCPServerTablespec_path20260220124742_add_spec_path_to_mcp_servers

Additional gaps still present (lower severity, features not yet in use):

  • LiteLLM_DeprecatedVerificationToken table entirely missing (20260203120000)
  • LiteLLM_EndUserTable.object_permission_id (20260214185341)
  • LiteLLM_ProjectTable entirely missing (20251113000000)

Workaround

Applied missing columns manually via prisma db execute:

-- LiteLLM_VerificationToken
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "project_id" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_active" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_count" INTEGER DEFAULT 0;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "auto_rotate" BOOLEAN DEFAULT false;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_interval" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_rotation_at" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "key_rotation_at" TIMESTAMP(3);

-- LiteLLM_MCPServerTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN IF NOT EXISTS "spec_path" TEXT;

Environment

  • Image: ghcr.io/berriai/litellm-non_root:main-v1.81.14-stable
  • Database: PostgreSQL 16 (Cloud SQL)
  • Deployment: GKE, 5 replicas
  • OS in container: wolfi (prints "Falling back to Prisma engines built debian" on every prisma command)

Suggested fixes

  1. Startup sanity check: After prisma migrate deploy, run a schema diff (the extras package already does this partially) and fail loudly (or at minimum log at ERROR level) if columns in schema.prisma are missing from the DB.

  2. get_all_mcp_servers() error handling: The current except Exception swallows the Prisma error silently and returns []. This should log at ERROR level so the missing column is visible without digging into debug logs.

  3. Idempotent migration re-run: Consider adding a startup flag or script that re-applies all extras migrations with IF NOT EXISTS guards, so upgrades are safe even if migrations were previously mis-recorded.

extent analysis

Fix Plan

To address the issue, we will implement the following fixes:

  1. Startup sanity check:

    • Run a schema diff after prisma migrate deploy to check for missing columns.
    • Fail loudly or log at ERROR level if columns are missing.

  2. Error handling in get_all_mcp_servers():

    • Modify the except block to log the error at ERROR level instead of swallowing it silently.

  3. Idempotent migration re-run:

    • Create a startup script that re-applies all extras migrations with IF NOT EXISTS guards.

Code Changes

Startup Sanity Check

import os
from prisma import Prisma

# After prisma migrate deploy
def check_schema_diff():
    schema = Prisma().schema
    db_columns = schema.get_columns()
    schema_columns = schema.get_schema_columns()

    missing_columns = [column for column in schema_columns if column not in db_columns]

    if missing_columns:
        print(f"ERROR: Missing columns in database: {missing_columns}")
        # Fail loudly or log at ERROR level
        os.exit(1)

check_schema_diff()

Error Handling in get_all_mcp_servers()

try:
    # Existing code to get MCP servers
except Exception as e:
    # Log the error at ERROR level
    logging.error(f"Error getting MCP servers: {e}")
    # Return an empty list or a custom error response
    return []

Idempotent Migration Re-run

# Create a startup script to re-apply extras migrations
prisma migrate deploy --create-only
prisma db execute --sql-file extras_migrations.sql

Create a file extras_migrations.sql with the following content:

-- LiteLLM_VerificationToken
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "project_id" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_active" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_count" INTEGER DEFAULT 0;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "auto_rotate" BOOLEAN DEFAULT false;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "rotation_interval" TEXT;
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "last_rotation_at" TIMESTAMP(3);
ALTER TABLE "LiteLLM_VerificationToken" ADD COLUMN IF NOT EXISTS "key_rotation_at" TIMESTAMP(3);

-- LiteLLM_MCPServerTable
ALTER TABLE "LiteLLM_MCPServerTable" ADD COLUMN IF NOT EXISTS "spec_path" TEXT;

Verification

To verify the fixes, run the following tests:

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#autograd error#authentication issue#prompt issue#agent setup#task chaining

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING