pytorch - ✅(Solved) Fix "pending uninvoked backwards" warning remediation instructions incorrect [1 pull requests, 1 comments, 2 participants]

pytorch2026-03-08 21:49:32

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

pytorch/pytorch#176846•Fetched 2026-04-08 00:24:11

View on GitHub

Comments

Participants

Timeline

Reactions

Author

rwkeane

Participants

rwkeane

Sahitya0805

Timeline (top)

mentioned ×10subscribed ×10referenced ×5labeled ×3

Error Message

Error logs

warnings.warn( warnings.warn( warnings.warn(

Root Cause

In the following scenario, the printed warning string is wrong: UserWarning: Unable to hit fast path of CUDAGraphs because of pending, uninvoked backwards. Consider running with torch.no_grad() or using torch.compiler.cudagraph_mark_step_begin() before each model invocation and the suggested remediation does not work.

Fix Action

Fix / Workaround

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 12 On-line CPU(s) list: 0-11 Vendor ID: AuthenticAMD Model name: AMD Ryzen 5 7600X 6-Core Processor CPU family: 25 Model: 97 Thread(s) per core: 2 Core(s) per socket: 6 Socket(s): 1 Stepping: 2 Frequency boost: enabled CPU(s) scaling MHz: 87% CPU max MHz: 5457.1050 CPU min MHz: 427.3640 BogoMIPS: 9381.76 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze Virtualization: AMD-V L1d cache: 192 KiB (6 instances) L1i cache: 192 KiB (6 instances) L2 cache: 6 MiB (6 instances) L3 cache: 32 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-11 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Old microcode: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; Safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsa: Mitigation; Clear CPU buffers Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

PR fix notes

PR #176865: Fix incorrect remediation instructions in cudagraph pending backward warning

Repository: pytorch/pytorch
Author: Sahitya0805
State: closed | merged: False
Link: https://github.com/pytorch/pytorch/pull/176865

Description (problem / solution / changelog)

Summary

The current warning message for CUDAGraph fast path failure due to pending backwards suggests using torch.no_grad(). However, in some cases (e.g., when using torch.func.grad_and_value()), torch.no_grad() is ineffective because the function internally constructs a backward graph. This PR updates the warning message to correctly identify the cause and suggest valid remediation steps like calling backward(), detaching outputs, or using mark_step_begin().

Test Plan

Verified the warning text update via grep in torch/_inductor/cudagraph_trees.py.
Tested that the updated warning now accurately reflects the issues reported in #176846.

Fixes #176846

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo @Lucaskabela

Changed files

benchmarks/dynamo/common.py (modified, +2/-2)
test/inductor/test_cudagraph_trees.py (modified, +6/-3)
torch/_inductor/cudagraph_trees.py (modified, +3/-3)

Code Example

from __future__ import annotations

import torch
import torch.nn as nn

DEVICE = torch.device("cuda")

class TinyModel(nn.Module):
    """A minimal model — one linear layer returning a scalar sum."""

    def __init__(self) -> None:
        """Initialise weights."""
        super().__init__()
        self.fc = nn.Linear(8, 1, device=DEVICE)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Run a forward pass."""
        return self.fc(x).sum()


# ---------------------------------------------------------------------------
# Case 1: WITHOUT torch.no_grad() — warning fires on step 2.
# ---------------------------------------------------------------------------

print("\n=== Case 1: compiled grad_and_value WITHOUT torch.no_grad() ===")

model1 = TinyModel()
params1 = dict(model1.named_parameters())
buffers1 = dict(model1.named_buffers())


def loss_fn1(
    p: dict[str, torch.Tensor],
    b: dict[str, torch.Tensor],
    x: torch.Tensor,
) -> torch.Tensor:
    """Purely functional forward pass."""
    return torch.func.functional_call(model1, (p, b), (x,))


compiled_gav1 = torch.compile(
    torch.func.grad_and_value(loss_fn1),
    mode="reduce-overhead",
)

kept1: list[object] = []
for i in range(5):
    x = torch.randn(4, 8, device=DEVICE)
    g, v = compiled_gav1(params1, buffers1, x)
    kept1.append((g, v))
    print(f"  step {i}: loss={v.item():.4f}")

# ---------------------------------------------------------------------------
# Case 2: WITH torch.no_grad() — warning still fires, remedy is ineffective.
# ---------------------------------------------------------------------------

print("\n=== Case 2: compiled grad_and_value WITH torch.no_grad() ===")

model2 = TinyModel()
params2 = dict(model2.named_parameters())
buffers2 = dict(model2.named_buffers())


def loss_fn2(
    p: dict[str, torch.Tensor],
    b: dict[str, torch.Tensor],
    x: torch.Tensor,
) -> torch.Tensor:
    """Purely functional forward pass."""
    return torch.func.functional_call(model2, (p, b), (x,))


compiled_gav2 = torch.compile(
    torch.func.grad_and_value(loss_fn2),
    mode="reduce-overhead",
)

kept2: list[object] = []
with torch.no_grad():
    for i in range(5):
        x = torch.randn(4, 8, device=DEVICE)
        g, v = compiled_gav2(params2, buffers2, x)
        kept2.append((g, v))
        print(f"  step {i}: loss={v.item():.4f}")

---

=== Case 1: compiled grad_and_value WITHOUT torch.no_grad() ===
/home/ryan/anaconda3/envs/equicloud_210/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:321: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
W0308 16:43:03.733000 1281458 site-packages/torch/_inductor/utils.py:1679] [0/0] Not enough SMs to use max_autotune_gemm mode
  step 0: loss=-0.5066
  step 1: loss=0.4607
/home/ryan/anaconda3/envs/equicloud_210/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py:2471: UserWarning: Unable to hit fast path of CUDAGraphs because of pending, uninvoked backwards. Consider running with torch.no_grad() or using torch.compiler.cudagraph_mark_step_begin() before each model invocation
  warnings.warn(
  step 2: loss=0.8372
  step 3: loss=-0.2559
  step 4: loss=-2.8114

=== Case 2: compiled grad_and_value WITH torch.no_grad() ===
  step 0: loss=-0.2864
  step 1: loss=-0.8360
  step 2: loss=1.5232
/home/ryan/anaconda3/envs/equicloud_210/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py:2471: UserWarning: Unable to hit fast path of CUDAGraphs because of pending, uninvoked backwards. Consider running with torch.no_grad() or using torch.compiler.cudagraph_mark_step_begin() before each model invocation
  warnings.warn(
  step 3: loss=-2.9598
  step 4: loss=0.8202

---

(env_210) ryan@ryan-dev-box:~/src/env$ curl -OL https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 31107  100 31107    0     0  80083      0 --:--:-- --:--:-- --:--:-- 79966
(env_210) ryan@ryan-dev-box:~/src/env$ python3 collect_env.py
Collecting environment information...
PyTorch version: 2.10.0+cu129
Is debug build: False
CUDA used to build PyTorch: 12.9
ROCM used to build PyTorch: N/A

OS: Ubuntu 25.10 (x86_64)
GCC version: (Ubuntu 15.2.0-4ubuntu4) 15.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.42

Python version: 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.17.0-12-generic-x86_64-with-glibc2.42
Is CUDA available: True
CUDA runtime version: 12.9.86
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 5060 Ti
Nvidia driver version: 580.126.09
cuDNN version: Could not collect
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: N/A

CPU:
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           48 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen 5 7600X 6-Core Processor
CPU family:                              25
Model:                                   97
Thread(s) per core:                      2
Core(s) per socket:                      6
Socket(s):                               1
Stepping:                                2
Frequency boost:                         enabled
CPU(s) scaling MHz:                      87%
CPU max MHz:                             5457.1050
CPU min MHz:                             427.3640
BogoMIPS:                                9381.76
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
Virtualization:                          AMD-V
L1d cache:                               192 KiB (6 instances)
L1i cache:                               192 KiB (6 instances)
L2 cache:                                6 MiB (6 instances)
L3 cache:                                32 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

Versions of relevant libraries:
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] torch==2.10.0+cu129
[pip3] torch-geometric==2.7.0
[pip3] torchvision==0.25.0+cu129
[pip3] triton==3.6.0
[conda] numpy                       2.2.6            pypi_0           pypi
[conda] nvidia-cublas-cu12          12.9.1.4         pypi_0           pypi
[conda] nvidia-cuda-cupti-cu12      12.9.79          pypi_0           pypi
[conda] nvidia-cuda-nvrtc-cu12      12.9.86          pypi_0           pypi
[conda] nvidia-cuda-runtime-cu12    12.9.79          pypi_0           pypi
[conda] nvidia-cudnn-cu12           9.10.2.21        pypi_0           pypi
[conda] nvidia-cufft-cu12           11.4.1.4         pypi_0           pypi
[conda] nvidia-curand-cu12          10.3.10.19       pypi_0           pypi
[conda] nvidia-cusolver-cu12        11.7.5.82        pypi_0           pypi
[conda] nvidia-cusparse-cu12        12.5.10.65       pypi_0           pypi
[conda] nvidia-cusparselt-cu12      0.7.1            pypi_0           pypi
[conda] nvidia-nccl-cu12            2.27.5           pypi_0           pypi
[conda] nvidia-nvjitlink-cu12       12.9.86          pypi_0           pypi
[conda] nvidia-nvtx-cu12            12.9.79          pypi_0           pypi
[conda] torch                       2.10.0+cu129     pypi_0           pypi
[conda] torch-geometric             2.7.0            pypi_0           pypi
[conda] torchvision                 0.25.0+cu129     pypi_0           pypi
[conda] triton                      3.6.0            pypi_0           pypi

RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

from __future__ import annotations

import torch
import torch.nn as nn

DEVICE = torch.device("cuda")

class TinyModel(nn.Module):
    """A minimal model — one linear layer returning a scalar sum."""

    def __init__(self) -> None:
        """Initialise weights."""
        super().__init__()
        self.fc = nn.Linear(8, 1, device=DEVICE)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Run a forward pass."""
        return self.fc(x).sum()


# ---------------------------------------------------------------------------
# Case 1: WITHOUT torch.no_grad() — warning fires on step 2.
# ---------------------------------------------------------------------------

print("\n=== Case 1: compiled grad_and_value WITHOUT torch.no_grad() ===")

model1 = TinyModel()
params1 = dict(model1.named_parameters())
buffers1 = dict(model1.named_buffers())


def loss_fn1(
    p: dict[str, torch.Tensor],
    b: dict[str, torch.Tensor],
    x: torch.Tensor,
) -> torch.Tensor:
    """Purely functional forward pass."""
    return torch.func.functional_call(model1, (p, b), (x,))


compiled_gav1 = torch.compile(
    torch.func.grad_and_value(loss_fn1),
    mode="reduce-overhead",
)

kept1: list[object] = []
for i in range(5):
    x = torch.randn(4, 8, device=DEVICE)
    g, v = compiled_gav1(params1, buffers1, x)
    kept1.append((g, v))
    print(f"  step {i}: loss={v.item():.4f}")

# ---------------------------------------------------------------------------
# Case 2: WITH torch.no_grad() — warning still fires, remedy is ineffective.
# ---------------------------------------------------------------------------

print("\n=== Case 2: compiled grad_and_value WITH torch.no_grad() ===")

model2 = TinyModel()
params2 = dict(model2.named_parameters())
buffers2 = dict(model2.named_buffers())


def loss_fn2(
    p: dict[str, torch.Tensor],
    b: dict[str, torch.Tensor],
    x: torch.Tensor,
) -> torch.Tensor:
    """Purely functional forward pass."""
    return torch.func.functional_call(model2, (p, b), (x,))


compiled_gav2 = torch.compile(
    torch.func.grad_and_value(loss_fn2),
    mode="reduce-overhead",
)

kept2: list[object] = []
with torch.no_grad():
    for i in range(5):
        x = torch.randn(4, 8, device=DEVICE)
        g, v = compiled_gav2(params2, buffers2, x)
        kept2.append((g, v))
        print(f"  step {i}: loss={v.item():.4f}")

Error logs

=== Case 1: compiled grad_and_value WITHOUT torch.no_grad() ===
/home/ryan/anaconda3/envs/equicloud_210/lib/python3.12/site-packages/torch/_inductor/compile_fx.py:321: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
W0308 16:43:03.733000 1281458 site-packages/torch/_inductor/utils.py:1679] [0/0] Not enough SMs to use max_autotune_gemm mode
  step 0: loss=-0.5066
  step 1: loss=0.4607
/home/ryan/anaconda3/envs/equicloud_210/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py:2471: UserWarning: Unable to hit fast path of CUDAGraphs because of pending, uninvoked backwards. Consider running with torch.no_grad() or using torch.compiler.cudagraph_mark_step_begin() before each model invocation
  warnings.warn(
  step 2: loss=0.8372
  step 3: loss=-0.2559
  step 4: loss=-2.8114

=== Case 2: compiled grad_and_value WITH torch.no_grad() ===
  step 0: loss=-0.2864
  step 1: loss=-0.8360
  step 2: loss=1.5232
/home/ryan/anaconda3/envs/equicloud_210/lib/python3.12/site-packages/torch/_inductor/cudagraph_trees.py:2471: UserWarning: Unable to hit fast path of CUDAGraphs because of pending, uninvoked backwards. Consider running with torch.no_grad() or using torch.compiler.cudagraph_mark_step_begin() before each model invocation
  warnings.warn(
  step 3: loss=-2.9598
  step 4: loss=0.8202

Versions

(env_210) ryan@ryan-dev-box:~/src/env$ curl -OL https://raw.githubusercontent.com/pytorch/pytorch/main/torch/utils/collect_env.py
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 31107  100 31107    0     0  80083      0 --:--:-- --:--:-- --:--:-- 79966
(env_210) ryan@ryan-dev-box:~/src/env$ python3 collect_env.py
Collecting environment information...
PyTorch version: 2.10.0+cu129
Is debug build: False
CUDA used to build PyTorch: 12.9
ROCM used to build PyTorch: N/A

OS: Ubuntu 25.10 (x86_64)
GCC version: (Ubuntu 15.2.0-4ubuntu4) 15.2.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.42

Python version: 3.12.12 | packaged by Anaconda, Inc. | (main, Oct 21 2025, 20:16:04) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.17.0-12-generic-x86_64-with-glibc2.42
Is CUDA available: True
CUDA runtime version: 12.9.86
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 5060 Ti
Nvidia driver version: 580.126.09
cuDNN version: Could not collect
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: N/A

CPU:
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           48 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  12
On-line CPU(s) list:                     0-11
Vendor ID:                               AuthenticAMD
Model name:                              AMD Ryzen 5 7600X 6-Core Processor
CPU family:                              25
Model:                                   97
Thread(s) per core:                      2
Core(s) per socket:                      6
Socket(s):                               1
Stepping:                                2
Frequency boost:                         enabled
CPU(s) scaling MHz:                      87%
CPU max MHz:                             5457.1050
CPU min MHz:                             427.3640
BogoMIPS:                                9381.76
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_v2 nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpuid_fault cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic vgif x2avic v_spec_ctrl vnmi avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d amd_lbr_pmc_freeze
Virtualization:                          AMD-V
L1d cache:                               192 KiB (6 instances)
L1i cache:                               192 KiB (6 instances)
L2 cache:                                6 MiB (6 instances)
L3 cache:                                32 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-11
Vulnerability Gather data sampling:      Not affected
Vulnerability Ghostwrite:                Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Old microcode:             Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; Safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; STIBP always-on; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

Versions of relevant libraries:
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] torch==2.10.0+cu129
[pip3] torch-geometric==2.7.0
[pip3] torchvision==0.25.0+cu129
[pip3] triton==3.6.0
[conda] numpy                       2.2.6            pypi_0           pypi
[conda] nvidia-cublas-cu12          12.9.1.4         pypi_0           pypi
[conda] nvidia-cuda-cupti-cu12      12.9.79          pypi_0           pypi
[conda] nvidia-cuda-nvrtc-cu12      12.9.86          pypi_0           pypi
[conda] nvidia-cuda-runtime-cu12    12.9.79          pypi_0           pypi
[conda] nvidia-cudnn-cu12           9.10.2.21        pypi_0           pypi
[conda] nvidia-cufft-cu12           11.4.1.4         pypi_0           pypi
[conda] nvidia-curand-cu12          10.3.10.19       pypi_0           pypi
[conda] nvidia-cusolver-cu12        11.7.5.82        pypi_0           pypi
[conda] nvidia-cusparse-cu12        12.5.10.65       pypi_0           pypi
[conda] nvidia-cusparselt-cu12      0.7.1            pypi_0           pypi
[conda] nvidia-nccl-cu12            2.27.5           pypi_0           pypi
[conda] nvidia-nvjitlink-cu12       12.9.86          pypi_0           pypi
[conda] nvidia-nvtx-cu12            12.9.79          pypi_0           pypi
[conda] torch                       2.10.0+cu129     pypi_0           pypi
[conda] torch-geometric             2.7.0            pypi_0           pypi
[conda] torchvision                 0.25.0+cu129     pypi_0           pypi
[conda] triton                      3.6.0            pypi_0           pypi

cc @mcarilli @ezyang @eellison @penguinwu @BoyuanFeng @chauhang

extent analysis

Fix Plan

Problem Summary

The issue is related to a warning message when using PyTorch's torch.compile function, specifically when trying to use the cudagraphs feature.

Root Cause Analysis

The root cause of the issue is likely due to the fact that the cudagraphs feature is not properly configured or initialized.

Fix Plan

Step 1: Check PyTorch Version

Make sure you are using the latest version of PyTorch. You can check the version by running torch.__version__.

Step 2: Update PyTorch

If you are not using the latest version, update PyTorch by running pip install --upgrade torch torchvision.

Step 3: Check CUDA Version

Make sure you are using a compatible CUDA version. You can check the CUDA version by running nvcc --version.

Step 4: Update CUDA

If you are not using the latest CUDA version, update CUDA by following the instructions on the NVIDIA website.

Step 5: Re-compile PyTorch

After updating PyTorch and CUDA, re-compile PyTorch by running python -m torch.utils.compile.

Step 6: Check CUDAGraphs Configuration

Make sure that the cudagraphs feature is properly configured. You can check the configuration by running torch.backends.cudagraphs.enabled.

Step 7: Update CUDAGraphs Configuration

If the cudagraphs feature is not enabled, enable it by running torch.backends.cudagraphs.enabled = True.

Step 8: Re-run the Code

After updating the configuration, re-run the code that was causing the warning message.

Example Code

import torch
import torch.nn as nn

# Create a model
model = nn.Linear(8, 1)

# Compile the model
compiled_model = torch.compile(model

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #autograd error #API rate limit #retriever error #indexing error #inference speed

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

pytorch - ✅(Solved) Fix "pending uninvoked backwards" warning remediation instructions incorrect [1 pull requests, 1 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Error logs

Root Cause

Fix Action

Fix / Workaround

PR fix notes

PR #176865: Fix incorrect remediation instructions in cudagraph pending backward warning

Description (problem / solution / changelog)

Summary

Test Plan

Changed files

Code Example

🐛 Describe the bug

Error logs

Versions

extent analysis

Fix Plan

Problem Summary

Root Cause Analysis

Fix Plan

Step 1: Check PyTorch Version

Step 2: Update PyTorch

Step 3: Check CUDA Version

Step 4: Update CUDA

Step 5: Re-compile PyTorch

Step 6: Check CUDAGraphs Configuration

Step 7: Update CUDAGraphs Configuration

Step 8: Re-run the Code

Example Code

Still need to ship something?

RELATED_DISCOVERY

TRENDING