vllm - 💡(How to fix) Fix [Bug]: DSV4 TileLang JIT fails with 'Permission denied: /tmp/tvm-debug-mode-tempdirs/...' on shared multi-user hosts [1 comments, 2 participants]

mgoin · 2026-04-30T20:37:02Z

[vllm] Your current environment The output of python collect_env.py ```text ============================== System Info ============================== OS : CentOS Stream 9 (x86_64) GCC version : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11) Clang version : 20.1.8 (CentOS 20.1.8-3.el9) CMake version : version 3.26.5 Libc version : glibc-2.34 ============================== PyTorch Info ============================== PyTorch version : 2.11.0+cu130 Is debug build : False CUDA used to build PyTorch : 13.0 ============================== Python Environment ============================== Python version : 3.12.12 Python platform : Linux-5.14.0-620.el9.x86_64-x86_64-with-glibc2.34 ============================== CUDA / GPU Info ============================== Is CUDA available : True CUDA runtime version : 13.0.88 GPU models and configuration : GPU 0-7: NVIDIA H100 80GB HBM3 Nvidia driver version : 590.48.01 ============================== Versions of relevant libraries ============================== [pip3] flashinfer-python==0.6.8.post1 [pip3] numpy==2.3.5 [pip3] torch==2.11.0+cu130 [pip3] transformers==5.7.0 [pip3] triton==3.6.0 tilelang : 0.1.9 ============================== vLLM Info ============================== vLLM Version : 0.20.1rc1.dev106+g307b17ce3 (git sha: 307b17ce3) ``` ### 🐛 Describe the bug On a shared multi-user machine, starting vLLM with a model that JIT-compiles a TileLang kernel (e.g. DeepSeek-V4 with the `mhc_pre_big_fuse_tilelang` indexer kernel) crashes during `profile_run` with: ``` RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol' ``` #### Reproduction ```bash vllm serve deepseek-ai/DeepSeek-V4-Flash \ --trust-remote-code \ --kv-cache-dtype fp8 \ --block-size 256 \ --enable-expert-parallel \ --tensor-parallel-size 4 \ --tokenizer-mode deepseek_v4 \ --tool-call-parser deepseek_v4 \ --enable-auto-tool-choice \ --reasoning-parser deepseek_v4 ``` …on a host where another user has previously run TileLang and `/tmp/tvm-debug-mode-tempdirs/` already exists, owned by them with mode `0755`: ``` $ ls -ld /tmp/tvm-debug-mode-tempdirs/ drwxr-xr-x. 14 otheruser otheruser 280 Apr 30 17:31 /tmp/tvm-debug-mode-tempdirs/ ``` #### Root cause `tilelang/contrib/nvcc.py:70` calls: ```python temp = utils.tempdir(keep_for_debug=not env.should_cleanup_temp_files()) ``` `should_cleanup_temp_files()` is False by default (env var `TILELANG_CLEANUP_TEMP_FILES` unset), so `keep_for_debug=True`. That branch in `tvm/contrib/utils.py` routes through a hardcoded shared parent dir: ```python all_parents = f"{tempfile.gettempdir()}/tvm-debug-mode-tempdirs" os.makedirs(all_parents, exist_ok=True) cls._DEBUG_PARENT_DIR = tempfile.mkdtemp(..., dir=all_parents) ``` The first user to compile creates `/tmp/tvm-debug-mode-tempdirs/` with mode `0755` (owned by them). Every subsequent user on the host is then blocked from `mkdir`-ing inside it. #### Workaround ```bash export TILELANG_CLEANUP_TEMP_FILES=1 ``` This flips `keep_for_debug` to False and TileLang uses a private `tempfile.mkdtemp()` instead of the shared parent dir. #### Suggested fix vLLM should either set `TILELANG_CLEANUP_TEMP_FILES=1` by default for TileLang invocations, or otherwise ensure the TileLang JIT path doesn't rely on a world-shared `/tmp` subdir whose ownership is determined by whichever user happened to compile first. (Upstream fix in TileLang/TVM would be even better — using a per-user prefix.) #### Truncated traceback ```text File ".../vllm/v1/worker/gpu_model_runner.py", line 5852, in profile_run hidden_states, last_hidden_states = self._dummy_run(...) ... File ".../vllm/model_executor/layers/mhc.py", line 285, in mhc_pre mhc_pre_big_fuse_tilelang(...) File ".../tilelang/jit/kernel.py", line 247, in _compile_and_create_adapter artifact = tilelang.lower(...) File ".../tilelang/engine/lower.py", line 150, in tilelang_callback_cuda_compile ptx = nvcc.compile_cuda(...) File ".../tilelang/contrib/nvcc.py", line 70, in compile_cuda temp = utils.tempdir(keep_for_debug=not env.should_cleanup_temp_files()) File ".../tvm/contrib/utils.py", line 115, in __init__ parent_dir = self._get_debug_parent_dir() File ".../tvm/contrib/utils.py", line 69, in _get_debug_parent_dir cls._DEBUG_PARENT_DIR = tempfile.mkdtemp(...) File ".../python3.12/tempfile.py", line 384, in mkdtemp _os.mkdir(file, 0o700) RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol' ``` --- _Reported with AI assistance (Claude Code)._ ### Before submitting a new issue... - [X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the [documentation page](https://docs.vllm.ai/en/latest/), which can answer lo

vllm2026-04-30 20:37:02

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

vllm-project/vllm#41410•Fetched 2026-05-01 05:33:47

View on GitHub

Comments

Participants

Timeline

Reactions

Author

mgoin

Participants

gau-nernst

mgoin

Timeline (top)

commented ×1labeled ×1renamed ×1

Error Message

RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol'

Root Cause

Root cause

Fix Action

Fix / Workaround

Workaround

Code Example

==============================
        System Info
==============================
OS                           : CentOS Stream 9 (x86_64)
GCC version                  : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11)
Clang version                : 20.1.8 (CentOS 20.1.8-3.el9)
CMake version                : version 3.26.5
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0

==============================
      Python Environment
==============================
Python version               : 3.12.12
Python platform              : Linux-5.14.0-620.el9.x86_64-x86_64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
GPU models and configuration :
GPU 0-7: NVIDIA H100 80GB HBM3
Nvidia driver version        : 590.48.01

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.3.5
[pip3] torch==2.11.0+cu130
[pip3] transformers==5.7.0
[pip3] triton==3.6.0
tilelang                     : 0.1.9

==============================
         vLLM Info
==============================
vLLM Version                 : 0.20.1rc1.dev106+g307b17ce3 (git sha: 307b17ce3)

---

RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol'

---

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 4 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

---

$ ls -ld /tmp/tvm-debug-mode-tempdirs/
drwxr-xr-x. 14 otheruser otheruser 280 Apr 30 17:31 /tmp/tvm-debug-mode-tempdirs/

---

temp = utils.tempdir(keep_for_debug=not env.should_cleanup_temp_files())

---

all_parents = f"{tempfile.gettempdir()}/tvm-debug-mode-tempdirs"
os.makedirs(all_parents, exist_ok=True)
cls._DEBUG_PARENT_DIR = tempfile.mkdtemp(..., dir=all_parents)

---

export TILELANG_CLEANUP_TEMP_FILES=1

---

File ".../vllm/v1/worker/gpu_model_runner.py", line 5852, in profile_run
    hidden_states, last_hidden_states = self._dummy_run(...)
...
File ".../vllm/model_executor/layers/mhc.py", line 285, in mhc_pre
    mhc_pre_big_fuse_tilelang(...)
File ".../tilelang/jit/kernel.py", line 247, in _compile_and_create_adapter
    artifact = tilelang.lower(...)
File ".../tilelang/engine/lower.py", line 150, in tilelang_callback_cuda_compile
    ptx = nvcc.compile_cuda(...)
File ".../tilelang/contrib/nvcc.py", line 70, in compile_cuda
    temp = utils.tempdir(keep_for_debug=not env.should_cleanup_temp_files())
File ".../tvm/contrib/utils.py", line 115, in __init__
    parent_dir = self._get_debug_parent_dir()
File ".../tvm/contrib/utils.py", line 69, in _get_debug_parent_dir
    cls._DEBUG_PARENT_DIR = tempfile.mkdtemp(...)
File ".../python3.12/tempfile.py", line 384, in mkdtemp
    _os.mkdir(file, 0o700)
RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol'

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

==============================
        System Info
==============================
OS                           : CentOS Stream 9 (x86_64)
GCC version                  : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-11)
Clang version                : 20.1.8 (CentOS 20.1.8-3.el9)
CMake version                : version 3.26.5
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0

==============================
      Python Environment
==============================
Python version               : 3.12.12
Python platform              : Linux-5.14.0-620.el9.x86_64-x86_64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 13.0.88
GPU models and configuration :
GPU 0-7: NVIDIA H100 80GB HBM3
Nvidia driver version        : 590.48.01

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.3.5
[pip3] torch==2.11.0+cu130
[pip3] transformers==5.7.0
[pip3] triton==3.6.0
tilelang                     : 0.1.9

==============================
         vLLM Info
==============================
vLLM Version                 : 0.20.1rc1.dev106+g307b17ce3 (git sha: 307b17ce3)

</details>

🐛 Describe the bug

On a shared multi-user machine, starting vLLM with a model that JIT-compiles a TileLang kernel (e.g. DeepSeek-V4 with the mhc_pre_big_fuse_tilelang indexer kernel) crashes during profile_run with:

RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol'

Reproduction

vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 4 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

…on a host where another user has previously run TileLang and /tmp/tvm-debug-mode-tempdirs/ already exists, owned by them with mode 0755:

$ ls -ld /tmp/tvm-debug-mode-tempdirs/
drwxr-xr-x. 14 otheruser otheruser 280 Apr 30 17:31 /tmp/tvm-debug-mode-tempdirs/

Root cause

tilelang/contrib/nvcc.py:70 calls:

temp = utils.tempdir(keep_for_debug=not env.should_cleanup_temp_files())

should_cleanup_temp_files() is False by default (env var TILELANG_CLEANUP_TEMP_FILES unset), so keep_for_debug=True. That branch in tvm/contrib/utils.py routes through a hardcoded shared parent dir:

all_parents = f"{tempfile.gettempdir()}/tvm-debug-mode-tempdirs"
os.makedirs(all_parents, exist_ok=True)
cls._DEBUG_PARENT_DIR = tempfile.mkdtemp(..., dir=all_parents)

The first user to compile creates /tmp/tvm-debug-mode-tempdirs/ with mode 0755 (owned by them). Every subsequent user on the host is then blocked from mkdir-ing inside it.

Workaround

export TILELANG_CLEANUP_TEMP_FILES=1

This flips keep_for_debug to False and TileLang uses a private tempfile.mkdtemp() instead of the shared parent dir.

Suggested fix

vLLM should either set TILELANG_CLEANUP_TEMP_FILES=1 by default for TileLang invocations, or otherwise ensure the TileLang JIT path doesn't rely on a world-shared /tmp subdir whose ownership is determined by whichever user happened to compile first. (Upstream fix in TileLang/TVM would be even better — using a per-user prefix.)

Truncated traceback

File ".../vllm/v1/worker/gpu_model_runner.py", line 5852, in profile_run
    hidden_states, last_hidden_states = self._dummy_run(...)
...
File ".../vllm/model_executor/layers/mhc.py", line 285, in mhc_pre
    mhc_pre_big_fuse_tilelang(...)
File ".../tilelang/jit/kernel.py", line 247, in _compile_and_create_adapter
    artifact = tilelang.lower(...)
File ".../tilelang/engine/lower.py", line 150, in tilelang_callback_cuda_compile
    ptx = nvcc.compile_cuda(...)
File ".../tilelang/contrib/nvcc.py", line 70, in compile_cuda
    temp = utils.tempdir(keep_for_debug=not env.should_cleanup_temp_files())
File ".../tvm/contrib/utils.py", line 115, in __init__
    parent_dir = self._get_debug_parent_dir()
File ".../tvm/contrib/utils.py", line 69, in _get_debug_parent_dir
    cls._DEBUG_PARENT_DIR = tempfile.mkdtemp(...)
File ".../python3.12/tempfile.py", line 384, in mkdtemp
    _os.mkdir(file, 0o700)
RuntimeError: [Errno 13] Permission denied: '/tmp/tvm-debug-mode-tempdirs/2026-04-30T19-45-18___mxg1ylol'

Reported with AI assistance (Claude Code).

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

Set the environment variable TILELANG_CLEANUP_TEMP_FILES=1 to avoid using a shared temporary directory.

Guidance

Identify the shared temporary directory /tmp/tvm-debug-mode-tempdirs/ and its ownership.
Set TILELANG_CLEANUP_TEMP_FILES=1 to use a private temporary directory instead of the shared one.
Consider implementing a per-user prefix for the temporary directory to avoid permission issues.
Verify that the issue is resolved by running the vllm serve command again.

Example

export TILELANG_CLEANUP_TEMP_FILES=1
vllm serve deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --block-size 256 \
  --enable-expert-parallel \
  --tensor-parallel-size 4 \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4

Notes

This solution assumes that the issue is caused by the shared temporary directory and its ownership. If the issue persists, further investigation may be necessary.

Recommendation

Apply the workaround by setting TILELANG_CLEANUP_TEMP_FILES=1 to avoid using a shared temporary directory. This should resolve the permission issue and allow the vllm serve command to run successfully.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: DSV4 TileLang JIT fails with 'Permission denied: /tmp/tvm-debug-mode-tempdirs/...' on shared multi-user hosts [1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Root cause

Fix Action

Fix / Workaround

Workaround

Code Example

Your current environment

🐛 Describe the bug

Reproduction

Root cause

Workaround

Suggested fix

Truncated traceback

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: DSV4 TileLang JIT fails with 'Permission denied: /tmp/tvm-debug-mode-tempdirs/...' on shared multi-user hosts [1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Root Cause

Root cause

Fix Action

Fix / Workaround

Workaround

Code Example

Your current environment

🐛 Describe the bug

Reproduction

Root cause

Workaround

Suggested fix

Truncated traceback

Before submitting a new issue...

extent analysis

TL;DR

Guidance

Example

Notes

Recommendation

Still need to ship something?

RELATED_DISCOVERY

TRENDING