Issue Station
Sign In
Tools hereToolbox

litellm - ✅(Solved) Fix [Bug]: LiteLLM non-root image fails Prisma migrations on ECS Fargate with readOnlyRootFilesystem (v1.82.0) [1 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
BerriAI/litellm#23750•Fetched 2026-04-08 00:49:23
View on GitHub
Comments
0
Participants
1
Timeline
4
Reactions
1
Author
chell0veckchell0veck
Participants
chell0veckchell0veck
Timeline (top)
labeled ×2cross-referenced ×1referenced ×1

Error Message

shutil.Error: [ ( '/usr/lib/python3.13/site-packages/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql', '/app/migrations/migrations/20251006143948_add_mcp_tool_permissions/migration.sql', "[Errno 1] Operation not permitted" ), ( <DirEntry '20251006143948_add_mcp_tool_permissions'>, '/app/migrations/migrations/20251006143948_add_mcp_tool_permissions', "[Errno 1] Operation not permitted" ), ... ]

Fix Action

Fixed

PR fix notes

PR #23756: fix: Prisma migrations fail with readOnlyRootFilesystem on ECS Fargate

Description (problem / solution / changelog)

Problem

When running LiteLLM as a non-root container on ECS Fargate with readOnlyRootFilesystem enabled, Prisma migrations fail because:

  1. Files are copied from the read-only root filesystem to a writable volume using shutil.copytree/copy2, which preserves source permissions
  2. The copied files retain read-only permissions from the source
  3. Prisma cannot write migration_lock.toml in the migrations directory, causing EPERM / "Operation not permitted" errors

This was reported in #23750.

Solution

This fix adds helper functions that ensure copied files are writable:

  • _copy_with_permissions() - Copies a file and ensures the destination is writable by adding owner write permission
  • _copytree_with_permissions() - Recursively copies a directory tree with writable permissions on all files and directories
  • ProxyExtrasDBManager._get_prisma_dir() - Updated to use these helpers when LITELLM_MIGRATION_DIR is set

Changes

  • litellm-proxy-extras/litellm_proxy_extras/utils.py: Added permission fix helpers and updated _get_prisma_dir()
  • tests/litellm-proxy-extras/test_litellm_proxy_extras_utils.py: Added comprehensive tests for the fix

Testing

All tests pass:

  • test_copy_with_permissions_makes_file_writable: Verifies copied files are writable
  • test_copytree_with_permissions_makes_files_and_dirs_writable: Verifies recursive copies are writable
  • test_get_prisma_dir_with_custom_dir_makes_writable: Verifies the fix works in the actual migration flow

Fixes #23750

Changed files

  • litellm-proxy-extras/litellm_proxy_extras/utils.py (modified, +33/-3)
  • tests/litellm-proxy-extras/test_litellm_proxy_extras_utils.py (modified, +111/-1)

Code Example

shutil.Error: [
  (
    '/usr/lib/python3.13/site-packages/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql',
    '/app/migrations/migrations/20251006143948_add_mcp_tool_permissions/migration.sql',
    "[Errno 1] Operation not permitted"
  ),
  (
    <DirEntry '20251006143948_add_mcp_tool_permissions'>,
    '/app/migrations/migrations/20251006143948_add_mcp_tool_permissions',
    "[Errno 1] Operation not permitted"
  ),
  ...
]
RAW_BUFFERClick to expand / collapse

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When running LiteLLM as an ECS Fargate task using the official non-root image with readOnlyRootFilesystem enabled, LiteLLM fails during startup while initializing Prisma migrations.

This occurs even when the ECS task is configured with:

  • Non-root execution (LITELLM_NON_ROOT=true, UID 101)
  • readOnlyRootFilesystem = true
  • Writable ECS volumes mounted for:
    • /tmp
    • /app/cache
    • /app/migrations
  • An init container (ECS dependsOn pattern) running as root that:
    • Pre-creates all required directories
    • Applies correct ownership (chown to UID 101)
    • Sets proper permissions (e.g. 1777 for /tmp)

During startup, Prisma attempts to copy bundled migrations from the LiteLLM image into the writable migration directory using shutil.copytree(). The copy operation preserves read-only permissions from the source files inside the image, causing the destination migration files and directories to remain read-only.

Prisma then attempts to write metadata (e.g. migration_lock.toml) and fails with EPERM / Operation not permitted.

Expected behavior: LiteLLM should successfully initialize Prisma migrations when running as a non-root container in an ECS Fargate task with a read-only root filesystem and writable volumes.

litellm-fargate-task-revision.json

Steps to Reproduce

  1. Create an ECS Fargate task definition using: ghcr.io/berriai/litellm-non_root:main-v1.82.0-stable (digest: sha256:381a6923a5b2126ce1032b6b974f58d43cc137aab83e0fa775f590e44ac071bc)

  2. Configure the LiteLLM container with:

    • readOnlyRootFilesystem = true
    • LITELLM_NON_ROOT=true
    • Writable ECS volumes mounted at:
      • /tmp
      • /app/cache
      • /app/migrations

  3. Add an init-style ECS container (essential=false, dependsOn=SUCCESS) that:

    • Runs as root
    • mkdir -p /tmp /app/cache /app/migrations
    • chmod 1777 /tmp
    • chown -R 101:101 /app /tmp

  4. Start the ECS task.

  5. Observe LiteLLM failing during Prisma initialization with permission errors.

This reproduces consistently on ECS Fargate and can also be reproduced locally with Docker using --read-only and a non-root user.

Relevant log output

shutil.Error: [
  (
    '/usr/lib/python3.13/site-packages/litellm_proxy_extras/migrations/20251006143948_add_mcp_tool_permissions/migration.sql',
    '/app/migrations/migrations/20251006143948_add_mcp_tool_permissions/migration.sql',
    "[Errno 1] Operation not permitted"
  ),
  (
    <DirEntry '20251006143948_add_mcp_tool_permissions'>,
    '/app/migrations/migrations/20251006143948_add_mcp_tool_permissions',
    "[Errno 1] Operation not permitted"
  ),
  ...
]

What part of LiteLLM is this about?

SDK (litellm Python package)

What LiteLLM version are you on ?

ghcr.io/berriai/litellm-non_root:main-v1.82.0-stable@sha256:381a6923a5b2126ce1032b6b974f58d43cc137aab83e0fa775f590e44ac071bc

Twitter / LinkedIn details

No response

extent analysis

Fix Plan

To resolve the issue, we need to ensure that the copied migration files and directories have the correct permissions. We can achieve this by modifying the shutil.copytree() call to not preserve the permissions from the source files.

Here are the steps to fix the issue:

  • Modify the shutil.copytree() call to use the dirs_exist_ok parameter and set the permissions manually after copying.
  • Use the os and shutil modules to set the correct permissions for the destination migration files and directories.

Example code:

import os
import shutil

# ... (rest of the code remains the same)

# Copy the migration files and directories
shutil.copytree(src, dst, dirs_exist_ok=True)

# Set the correct permissions for the destination migration files and directories
for root, dirs, files in os.walk(dst):
    for dir in dirs:
        dir_path = os.path.join(root, dir)
        os.chmod(dir_path, 0o755)  # Set directory permissions to 755
    for file in files:
        file_path = os.path.join(root, file)
        os.chmod(file_path, 0o644)  # Set file permissions to 644

Alternatively, you can use the copytree function from the distutils module, which allows you to specify the permissions for the destination files and directories:

from distutils.dir_util import copy_tree

# ... (rest of the code remains the same)

# Copy the migration files and directories
copy_tree(src, dst, preserve_mode=0, preserve_times=0, preserve_symlinks=1)

Verification

To verify that the fix worked, you can check the permissions of the destination migration files and directories after the copy operation. You can use the os module to check the permissions:

import os

# ... (rest of the code remains the same)

# Check the permissions of the destination migration files and directories
for root, dirs, files in os.walk(dst):
    for dir in dirs:
        dir_path = os.path.join(root, dir)
        print(f"Directory {dir_path} has permissions {oct(os.stat(dir_path).st_mode)}")
    for file in files:
        file_path = os.path.join(root, file)
        print(f"File {file_path} has permissions {oct(os.stat(file_path).st_mode)}")

If the permissions are correct, you should see output indicating that the directories have permissions 0o755 and the files have permissions 0o644.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#permission error#ISR setup#authentication setup#request error#file not found

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING

litellm - ✅(Solved) Fix [Bug]: LiteLLM non-root image fails Prisma migrations on ECS Fargate with readOnlyRootFilesystem (v1.82.0) [1 pull requests, 1 participants]