pytorch - 💡(How to fix) Fix RuntimeError: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH in F.conv2d on PyTorch nightly (cu130)

🐛 Describe the bug

When calling F.conv2d on CUDA tensors with PyTorch nightly (2.13.0.dev20260521+cu130), the operation fails with a CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH RuntimeError.

The issue seems strictly related to the cuDNN backend. If I set torch.backends.cudnn.enabled = False, the code runs successfully and outputs the correct result. This occurs in a completely fresh and clean virtual environment.

To Reproduce

import os
import torch
import torch.nn.functional as F

def main():
    assert torch.cuda.is_available()

    print("torch:", torch.__version__)
    print("cuda:", torch.version.cuda)
    print("cudnn available:", torch.backends.cudnn.is_available())
    print("cudnn enabled:", torch.backends.cudnn.enabled)
    print("cudnn version:", torch.backends.cudnn.version())
    print("LD_LIBRARY_PATH:", os.environ.get("LD_LIBRARY_PATH", ""))

    x = torch.randn(2, 3, 6, 7, device="cuda")
    weight = torch.randn(4, 3, 3, 3, device="cuda")
    bias = torch.zeros(4, device="cuda")

    print("x shape:", x.shape)
    print("x stride:", x.stride())
    print("x contiguous:", x.is_contiguous())

    out = F.conv2d(x, weight, bias, padding=1)
    torch.cuda.synchronize()

    print("PASS")
    print(out.shape)
    print(out.sum().item())

if __name__ == "__main__":
    main()

Error Logs

(debug) xyt19@Oasis:/tmp$ python bug.py
torch: 2.13.0.dev20260521+cu130
cuda: 13.0
cudnn available: True
cudnn enabled: True
cudnn version: 92000
LD_LIBRARY_PATH:
x shape: torch.Size([2, 3, 6, 7])
x stride: (126, 42, 7, 1)
x contiguous: True
Traceback (most recent call last):
  File "/tmp/bug.py", line 31, in <module>
    main()
  File "/tmp/bug.py", line 23, in main
    out = F.conv2d(x, weight, bias, padding=1)
RuntimeError: CUDNN_BACKEND_TENSOR_DESCRIPTOR cudnnFinalize failedptrDesc->finalize() cudnn_status: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH

Additional context

Tested in a completely clean virtual environment.

Versions

PyTorch version: 2.13.0.dev20260521+cu130 Is debug build: False CUDA used to build PyTorch: 13.0 ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.4 LTS (x86_64) GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version: 18.1.3 (1ubuntu1) CMake version: version 3.28.3 Libc version: glibc-2.39

Python version: 3.10.20 (main, Mar 11 2026, 17:46:40) [GCC 14.3.0] (64-bit runtime) Python platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39 Is CUDA available: True CUDA runtime version: 12.0.140

cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_tensor_ir.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.21.1 Is XPU available: False HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True Caching allocator config: N/A

Versions of relevant libraries: [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-nccl-cu13==2.29.7 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvtx==13.0.85 [pip3] torch==2.13.0.dev20260521+cu130 [pip3] torchaudio==2.11.0.dev20260527+cu130 [pip3] torchvision==0.28.0.dev20260527+cu130 [pip3] triton==3.7.0+git88b227e2 [conda] numpy 2.2.6 pypi_0 pypi [conda] nvidia-cublas 13.1.1.3 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.20.0.48 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.1 pypi_0 pypi [conda] nvidia-nccl-cu13 2.29.7 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] torch 2.13.0.dev20260521+cu130 pypi_0 pypi [conda] torchaudio 2.11.0.dev20260527+cu130 pypi_0 pypi [conda] torchvision 0.28.0.dev20260527+cu130 pypi_0 pypi [conda] triton 3.7.0+git88b227e2 pypi_0 pypi

cc @malfet @atalman @tinglvv @nWEIdia @csarofeen @ptrblck @eqy