Issue Station
Sign In
Tools hereToolbox

pytorch - ✅(Solved) Fix `torch.compile` crashes with FakeTensor view stride error on valid model with `index_put_` scatter [2 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
pytorch/pytorch#178039•Fetched 2026-04-08 01:07:30
View on GitHub
Comments
0
Participants
1
Timeline
80
Reactions
0
Author
himi1008himi1008
Participants
himi1008himi1008
Timeline (top)
mentioned ×32subscribed ×32labeled ×11cross-referenced ×2

Error Message

import torch import torch.nn as nn import torch.nn.functional as F

class ScatterModel(nn.Module): def init(self, vocab_size=5000, hidden_dim=256, num_heads=8): super().init() self.hidden_dim = hidden_dim self.embedding = nn.Embedding(vocab_size, hidden_dim) self.attention = nn.MultiheadAttention(hidden_dim, num_heads, batch_first=True) self.norm = nn.LayerNorm(hidden_dim) self.ffn = nn.Sequential( nn.Linear(hidden_dim, hidden_dim * 4), nn.ReLU(), nn.Linear(hidden_dim * 4, hidden_dim) ) self.output_proj = nn.Linear(hidden_dim, vocab_size) self.register_buffer('buffer', torch.zeros(vocab_size, hidden_dim))

def forward(self, input_ids):
    x = self.embedding(input_ids)
    attn_output, _ = self.attention(x, x, x)
    x = self.norm(x + attn_output)
    x = self.norm(x + self.ffn(x))
    logits = self.output_proj(x)

    # Scatter-add with high contention
    indices = input_ids.view(-1)
    values = attn_output.view(-1, self.hidden_dim)  # <-- Crashes here in compile
    buf = self.buffer.clone()
    torch.index_put_(buf, (indices,), values, accumulate=True)

    return F.softmax(logits, dim=-1), buf

device = "cuda" model = ScatterModel().to(device).eval() x = torch.randint(0, 5000, (4, 32), dtype=torch.long, device=device)

Eager: works

with torch.no_grad(): probs, buf = model(x) print(f"Eager: OK, probs={probs.shape}, buf={buf.shape}")

Compiled: crashes

torch._dynamo.reset() compiled = torch.compile(model, backend="inductor") try: with torch.no_grad(): probs2, buf2 = compiled(x) print(f"Compiled: OK") except Exception as e: print(f"Compiled: CRASH — {type(e).name}: {e}")

Root Cause

The error message reveals a stride mismatch in Dynamo's FakeTensor:

Cannot view a tensor with shape torch.Size([4, 32, 256]) and strides (256, 1024, 1)
as a tensor with shape (128, 256)!
  • Expected strides (contiguous): (8192, 256, 1) — 4*32*256, 32*256, 1
  • FakeTensor strides: (256, 1024, 1) — incorrect, not contiguous

The nn.MultiheadAttention output tensor has correct contiguous strides in eager mode, but Dynamo's symbolic shape propagation / FakeTensor computes incorrect strides for the attention output, causing the subsequent view(-1, hidden_dim) to fail.

This is a Dynamo FakeTensor stride inference bug for nn.MultiheadAttention output tensors.

Fix Action

Fix / Workaround

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: AuthenticAMD Model name: AMD Ryzen 7 5800H with Radeon Graphics CPU family: 25 Model: 80 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 Stepping: 0 BogoMIPS: 6387.83 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm Virtualization: AMD-V Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64 KiB (2 instances) L1i cache: 64 KiB (2 instances) L2 cache: 1 MiB (2 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-3 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

PR fix notes

PR #178706: Align slow-path MHA batch_first layout

Description (problem / solution / changelog)

Fix #178039

Summary

  1. Root cause problem The slow nn.MultiheadAttention(batch_first=True) path returns attn_output.transpose(1, 0) directly, which is a non-contiguous view. Eager runs that hit _native_multi_head_attention return a contiguous batch-first tensor instead, so torch.compile can observe a different layout than eager and reject downstream .view() calls.

  2. Proposed fix Return a contiguous tensor from the slow batch_first path by materializing the transposed output before returning it, and add a regression test that disables the MHA fast path and verifies the returned tensor is contiguous and supports .view().

  3. Why the proposed fix is the right long term fix This aligns the observable output layout of the slow and fast batch_first paths, which removes layout-dependent behavior differences between eager and compiled execution while preserving the existing tensor values and API shape contract.

Drafted via Codex, published after manual review by @bobrenjc93

Changed files

  • test/test_transformers.py (modified, +25/-0)
  • torch/nn/modules/activation.py (modified, +1/-1)

PR #178997: [dynamo] Preserve MHA fastpath layout while tracing

Description (problem / solution / changelog)

Fix #178039

Summary

  1. What is the root cause problem nn.MultiheadAttention disables _native_multi_head_attention while Dynamo/proxy-tensor tracing is active, so compile falls back to the transpose-based Python path even when eager would use the native fastpath. That slowpath returns a non-contiguous batch-first attention output, which makes a downstream view() fail during FakeTensor tracing on otherwise valid eager models.

  2. What is the proposed fix Keep tracing on the slowpath, but when tracing is the only reason the native fastpath was skipped, make the final batch-first attention output contiguous before returning it. This preserves the eager fastpath layout without reintroducing _native_multi_head_attention into traced graphs. The PR also adds a Dynamo regression test covering a compiled batch-first self-attention module followed by view().

  3. Why the proposed fix is the right long term fix This fixes the correctness gap at the layout boundary that users observe today while preserving the existing export/compile decision to avoid _native_multi_head_attention in traced graphs. It keeps the change narrowly scoped to the tracing-only divergence instead of broadening eager behavior or reintroducing an op that export still cannot decompose.

Drafted via Codex, published after manual review by @bobrenjc93

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @chauhang @amjames @Lucaskabela @jataylo @azahed98

Changed files

  • test/dynamo/test_repros.py (modified, +28/-0)
  • torch/nn/modules/activation.py (modified, +9/-3)

Code Example

import torch
import torch.nn as nn
import torch.nn.functional as F

class ScatterModel(nn.Module):
    def __init__(self, vocab_size=5000, hidden_dim=256, num_heads=8):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.embedding = nn.Embedding(vocab_size, hidden_dim)
        self.attention = nn.MultiheadAttention(hidden_dim, num_heads, batch_first=True)
        self.norm = nn.LayerNorm(hidden_dim)
        self.ffn = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim * 4),
            nn.ReLU(),
            nn.Linear(hidden_dim * 4, hidden_dim)
        )
        self.output_proj = nn.Linear(hidden_dim, vocab_size)
        self.register_buffer('buffer', torch.zeros(vocab_size, hidden_dim))

    def forward(self, input_ids):
        x = self.embedding(input_ids)
        attn_output, _ = self.attention(x, x, x)
        x = self.norm(x + attn_output)
        x = self.norm(x + self.ffn(x))
        logits = self.output_proj(x)

        # Scatter-add with high contention
        indices = input_ids.view(-1)
        values = attn_output.view(-1, self.hidden_dim)  # <-- Crashes here in compile
        buf = self.buffer.clone()
        torch.index_put_(buf, (indices,), values, accumulate=True)

        return F.softmax(logits, dim=-1), buf

device = "cuda"
model = ScatterModel().to(device).eval()
x = torch.randint(0, 5000, (4, 32), dtype=torch.long, device=device)

# Eager: works
with torch.no_grad():
    probs, buf = model(x)
print(f"Eager: OK, probs={probs.shape}, buf={buf.shape}")

# Compiled: crashes
torch._dynamo.reset()
compiled = torch.compile(model, backend="inductor")
try:
    with torch.no_grad():
        probs2, buf2 = compiled(x)
    print(f"Compiled: OK")
except Exception as e:
    print(f"Compiled: CRASH — {type(e).__name__}: {e}")

---

Cannot view a tensor with shape torch.Size([4, 32, 256]) and strides (256, 1024, 1)
as a tensor with shape (128, 256)!

---

RuntimeError when making fake tensor call
  Explanation: Dynamo failed to run FX node with fake tensors:
    call_method view(*(FakeTensor(..., device='cuda:0', size=(4, 32, 256)), -1, 256), **{}):
    got ValueError('Cannot view a tensor with shape torch.Size([4, 32, 256]) and
    strides (256, 1024, 1) as a tensor with shape (128, 256)!')
  Hint: Your code may result in an error when running in eager.
    Please double check that your code doesn't contain a similar error when actually running eager/uncompiled
RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

torch.compile with inductor backend crashes during Dynamo tracing with a ValueError: Cannot view a tensor with shape and strides error when compiling a valid transformer model that uses torch.index_put_ with accumulate=True (scatter-add pattern). The same model runs correctly in eager mode.

The error occurs because Dynamo's FakeTensor mode infers incorrect strides for the attention output tensor. When the model calls attn_output.view(-1, self.hidden_dim), FakeTensor computes strides (256, 1024, 1) for shape (4, 32, 256), making the view operation invalid. In eager mode, the actual tensor is contiguous with strides (8192, 256, 1), so view succeeds.

This was discovered via a fuzzer-generated sparse gradient accumulation model targeting the partitioned_scatter Inductor optimization (reduced atomic contention for high-contention index_put_).

Minimal reproducer

import torch
import torch.nn as nn
import torch.nn.functional as F

class ScatterModel(nn.Module):
    def __init__(self, vocab_size=5000, hidden_dim=256, num_heads=8):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.embedding = nn.Embedding(vocab_size, hidden_dim)
        self.attention = nn.MultiheadAttention(hidden_dim, num_heads, batch_first=True)
        self.norm = nn.LayerNorm(hidden_dim)
        self.ffn = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim * 4),
            nn.ReLU(),
            nn.Linear(hidden_dim * 4, hidden_dim)
        )
        self.output_proj = nn.Linear(hidden_dim, vocab_size)
        self.register_buffer('buffer', torch.zeros(vocab_size, hidden_dim))

    def forward(self, input_ids):
        x = self.embedding(input_ids)
        attn_output, _ = self.attention(x, x, x)
        x = self.norm(x + attn_output)
        x = self.norm(x + self.ffn(x))
        logits = self.output_proj(x)

        # Scatter-add with high contention
        indices = input_ids.view(-1)
        values = attn_output.view(-1, self.hidden_dim)  # <-- Crashes here in compile
        buf = self.buffer.clone()
        torch.index_put_(buf, (indices,), values, accumulate=True)

        return F.softmax(logits, dim=-1), buf

device = "cuda"
model = ScatterModel().to(device).eval()
x = torch.randint(0, 5000, (4, 32), dtype=torch.long, device=device)

# Eager: works
with torch.no_grad():
    probs, buf = model(x)
print(f"Eager: OK, probs={probs.shape}, buf={buf.shape}")

# Compiled: crashes
torch._dynamo.reset()
compiled = torch.compile(model, backend="inductor")
try:
    with torch.no_grad():
        probs2, buf2 = compiled(x)
    print(f"Compiled: OK")
except Exception as e:
    print(f"Compiled: CRASH — {type(e).__name__}: {e}")

Behavior summary

ModeResult
EagerSucceeds
torch.compile(backend="inductor")Crashes: ValueError: Cannot view a tensor with shape torch.Size([4, 32, 256]) and strides (256, 1024, 1) as a tensor with shape (128, 256)!

Root cause analysis

The error message reveals a stride mismatch in Dynamo's FakeTensor:

Cannot view a tensor with shape torch.Size([4, 32, 256]) and strides (256, 1024, 1)
as a tensor with shape (128, 256)!
  • Expected strides (contiguous): (8192, 256, 1) — 4*32*256, 32*256, 1
  • FakeTensor strides: (256, 1024, 1) — incorrect, not contiguous

The nn.MultiheadAttention output tensor has correct contiguous strides in eager mode, but Dynamo's symbolic shape propagation / FakeTensor computes incorrect strides for the attention output, causing the subsequent view(-1, hidden_dim) to fail.

This is a Dynamo FakeTensor stride inference bug for nn.MultiheadAttention output tensors.

Error logs

RuntimeError when making fake tensor call
  Explanation: Dynamo failed to run FX node with fake tensors:
    call_method view(*(FakeTensor(..., device='cuda:0', size=(4, 32, 256)), -1, 256), **{}):
    got ValueError('Cannot view a tensor with shape torch.Size([4, 32, 256]) and
    strides (256, 1024, 1) as a tensor with shape (128, 256)!')
  Hint: Your code may result in an error when running in eager.
    Please double check that your code doesn't contain a similar error when actually running eager/uncompiled

Note: The hint is misleading — eager mode works correctly because the actual tensor IS contiguous.

Versions

PyTorch version: 2.12.0.dev20260315+cu126 Is debug build: False CUDA used to build PyTorch: 12.6 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 4 2025, 08:48:33) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3060 Laptop GPU Nvidia driver version: 546.30 cuDNN version: Could not collect Is XPU available: False HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True Caching allocator config: N/A

CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Vendor ID: AuthenticAMD Model name: AMD Ryzen 7 5800H with Radeon Graphics CPU family: 25 Model: 80 Thread(s) per core: 2 Core(s) per socket: 2 Socket(s): 1 Stepping: 0 BogoMIPS: 6387.83 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr arat npt nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload umip vaes vpclmulqdq rdpid fsrm Virtualization: AMD-V Hypervisor vendor: Microsoft Virtualization type: full L1d cache: 64 KiB (2 instances) L1i cache: 64 KiB (2 instances) L2 cache: 1 MiB (2 instances) L3 cache: 16 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-3 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected

Versions of relevant libraries: [pip3] numpy==2.2.6 [pip3] nvidia-cublas-cu12==12.6.4.1 [pip3] nvidia-cuda-cupti-cu12==12.6.80 [pip3] nvidia-cuda-nvrtc-cu12==12.6.85 [pip3] nvidia-cuda-runtime-cu12==12.6.77 [pip3] nvidia-cudnn-cu12==9.10.2.21 [pip3] nvidia-cufft-cu12==11.3.0.4 [pip3] nvidia-curand-cu12==10.3.7.77 [pip3] nvidia-cusolver-cu12==11.7.1.2 [pip3] nvidia-cusparse-cu12==12.5.4.2 [pip3] nvidia-cusparselt-cu12==0.7.1 [pip3] nvidia-nccl-cu12==2.29.3 [pip3] nvidia-nvjitlink-cu12==12.6.85 [pip3] nvidia-nvtx-cu12==12.6.77 [pip3] torch==2.12.0.dev20260315+cu126 [pip3] torchaudio==2.11.0.dev20260315+cu126 [pip3] torchvision==0.26.0.dev20260315+cu126 [pip3] triton==3.6.0+git9844da95 [conda] Could not collect

cc @chauhang @penguinwu @eellison @aorenste @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @kadeng @amjames @Lucaskabela @jataylo @bdhirsh @bobrenjc93

extent analysis

Fix Plan

To fix the issue with torch.compile and inductor backend crashing during Dynamo tracing, we need to ensure that the tensor strides are correctly computed.

Here are the steps to fix the issue:

  • Modify the forward method of the ScatterModel class to avoid using torch.index_put_ with accumulate=True, which causes the stride mismatch.
  • Instead, use torch.scatter_add to achieve the same result.

Code Changes

class ScatterModel(nn.Module):
    # ...

    def forward(self, input_ids):
        # ...
        logits = self.output_proj(x)

        # Replace torch.index_put_ with torch.scatter_add
        indices = input_ids.view(-1)
        values = attn_output.view(-1, self.hidden_dim)
        buf = self.buffer.clone()
        buf.scatter_add_(0, indices.unsqueeze(-1).expand(-1, self.hidden_dim), values)

        return F.softmax(logits, dim=-1), buf

Verification

To verify that the fix worked, run the model in both eager and compiled modes:

# Eager: works
with torch.no_grad():
    probs, buf = model(x)
print(f"Eager: OK, probs={probs.shape}, buf={buf.shape}")

# Compiled: should work now
torch._dynamo.reset()
compiled = torch.compile(model, backend="inductor")
try:
    with torch.no_grad():
        probs2, buf2 = compiled(x)
    print(f"Compiled: OK")
except Exception as e:
    print(f"Compiled: CRASH — {type(e).__name__}: {e}")

If the fix is successful, the model should run without crashing in both eager and compiled modes.

Extra Tips

  • When using torch.compile with the inductor backend, ensure that the model is correctly handling tensor strides to avoid crashes during Dynamo tracing.
  • Consider using torch.scatter_add instead of torch.index_put_ with accumulate=True to avoid stride mismatches.
  • Always verify the model's behavior in both eager and compiled modes to ensure that the fix is working as expected.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#ssr#installation#tensor shape#optimization#request error#file not found#serialization error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING