pytorch - 💡(How to fix) Fix `torch.compile` produces numerically different results for addmm fusion pattern (`mm + bias`) compared to eager mode [6 comments, 5 participants]

pytorch2026-03-21 08:06:28

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

pytorch/pytorch#178047•Fetched 2026-04-08 01:07:23

View on GitHub

Comments

Participants

Timeline

Reactions

Author

himi1008

Participants

eqy

Timeline (top)

mentioned ×28subscribed ×28labeled ×11commented ×6

Error Message

import torch import torch.nn as nn

class AddmmModel(nn.Module): def init(self): super().init() self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1) self.conv2 = nn.Conv2d(32, 64, kernel_size=1) # Pointwise self.conv3 = nn.Conv2d(64, 128, kernel_size=1) # Pointwise self.pool = nn.MaxPool2d(2, 2) self.flatten = nn.Flatten() # Manual addmm pattern: mm + bias self.fc1_weight = nn.Parameter(torch.randn(128 * 7 * 7, 512)) self.fc1_bias = nn.Parameter(torch.randn(512)) self.fc2_weight = nn.Parameter(torch.randn(512, 256)) self.fc2_bias = nn.Parameter(torch.randn(256)) self.out_weight = nn.Parameter(torch.randn(256, 10)) self.out_bias = nn.Parameter(torch.randn(10))

def forward(self, x):
    x = torch.relu(self.conv1(x))
    x = self.pool(x)
    # GELU via erf approximation
    x = self.conv2(x)
    x = 0.5 * x * (1 + torch.erf(x * 0.7071067811865476))
    x = self.pool(x)
    x = self.conv3(x)
    x = 0.5 * x * (1 + torch.erf(x * 0.7071067811865476))
    x = self.pool(x)
    x = self.flatten(x)
    # addmm pattern 1: mm + bias
    x = torch.mm(x, self.fc1_weight) + self.fc1_bias
    x = torch.relu(x)
    # addmm pattern 2: bias + mm
    x = self.fc2_bias + torch.mm(x, self.fc2_weight)
    x = torch.relu(x)
    x = torch.mm(x, self.out_weight) + self.out_bias
    return x

device = "cuda" model = AddmmModel().to(device).eval() x = torch.randn(4, 3, 56, 56, device=device)

Eager: deterministic

with torch.no_grad(): ref1 = model(x) ref2 = model(x) print(f"Eager deterministic: {(ref1 - ref2).abs().max().item():.6e}")

Compiled: different result

torch._dynamo.reset() compiled = torch.compile(model, backend="inductor") with torch.no_grad(): out = compiled(x)

diff = (ref1.float() - out.float()).abs() print(f"max_diff={diff.max().item():.6e}") print(f"Output range: [{ref1.min().item():.2f}, {ref1.max().item():.2f}]") print(f"Relative error: {diff.max().item() / ref1.abs().max().item():.6e}")

Output:

Eager deterministic: 0.000000e+00

max_diff=1.573486e+00

Relative error: 3.902675e-04

Root Cause

The root cause is likely that the Inductor backend fuses mm + add into a single addmm CUBLAS call, which uses different floating-point accumulation order. While individual layer differences may be small, they compound through multiple sequential layers (conv → GELU → pool → mm+bias → relu → mm+bias → relu → mm+bias), amplifying the final discrepancy.

Code Example

import torch
import torch.nn as nn

class AddmmModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=1)  # Pointwise
        self.conv3 = nn.Conv2d(64, 128, kernel_size=1)  # Pointwise
        self.pool = nn.MaxPool2d(2, 2)
        self.flatten = nn.Flatten()
        # Manual addmm pattern: mm + bias
        self.fc1_weight = nn.Parameter(torch.randn(128 * 7 * 7, 512))
        self.fc1_bias = nn.Parameter(torch.randn(512))
        self.fc2_weight = nn.Parameter(torch.randn(512, 256))
        self.fc2_bias = nn.Parameter(torch.randn(256))
        self.out_weight = nn.Parameter(torch.randn(256, 10))
        self.out_bias = nn.Parameter(torch.randn(10))

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = self.pool(x)
        # GELU via erf approximation
        x = self.conv2(x)
        x = 0.5 * x * (1 + torch.erf(x * 0.7071067811865476))
        x = self.pool(x)
        x = self.conv3(x)
        x = 0.5 * x * (1 + torch.erf(x * 0.7071067811865476))
        x = self.pool(x)
        x = self.flatten(x)
        # addmm pattern 1: mm + bias
        x = torch.mm(x, self.fc1_weight) + self.fc1_bias
        x = torch.relu(x)
        # addmm pattern 2: bias + mm
        x = self.fc2_bias + torch.mm(x, self.fc2_weight)
        x = torch.relu(x)
        x = torch.mm(x, self.out_weight) + self.out_bias
        return x

device = "cuda"
model = AddmmModel().to(device).eval()
x = torch.randn(4, 3, 56, 56, device=device)

# Eager: deterministic
with torch.no_grad():
    ref1 = model(x)
    ref2 = model(x)
print(f"Eager deterministic: {(ref1 - ref2).abs().max().item():.6e}")

# Compiled: different result
torch._dynamo.reset()
compiled = torch.compile(model, backend="inductor")
with torch.no_grad():
    out = compiled(x)

diff = (ref1.float() - out.float()).abs()
print(f"max_diff={diff.max().item():.6e}")
print(f"Output range: [{ref1.min().item():.2f}, {ref1.max().item():.2f}]")
print(f"Relative error: {diff.max().item() / ref1.abs().max().item():.6e}")
# Output:
# Eager deterministic: 0.000000e+00
# max_diff=1.573486e+00
# Relative error: 3.902675e-04

---

Eager deterministic: 0.000000e+00
Eager vs Compiled: max_diff=1.573486e+00, mean_diff=4.845871e-01
Output range: [-4031.81, 1971.11]
Relative max: 3.902675e-04

---

PyTorch version: 2.12.0.dev20260315+cu126
Python: 3.10.12
OS: Ubuntu 22.04.5 LTS (WSL2)
GPU: NVIDIA GeForce RTX 3060 Laptop GPU
CUDA: 12.6

RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

torch.compile with inductor backend produces numerically different results compared to eager mode when the model uses the torch.mm(x, w) + bias pattern (addmm fusion). The maximum absolute difference is ~1.57 with output values in the range [-4000, 2000], giving a relative error of ~4e-4. This exceeds PyTorch's standard float32 tolerance (rtol=1.3e-6).

This was discovered via a fuzzer-generated CNN+MLP model that uses explicit addmm patterns (torch.mm(x, w) + b and b + torch.mm(x, w)), targeting the addmm Inductor optimization.

Minimal reproducer

import torch
import torch.nn as nn

class AddmmModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=1)  # Pointwise
        self.conv3 = nn.Conv2d(64, 128, kernel_size=1)  # Pointwise
        self.pool = nn.MaxPool2d(2, 2)
        self.flatten = nn.Flatten()
        # Manual addmm pattern: mm + bias
        self.fc1_weight = nn.Parameter(torch.randn(128 * 7 * 7, 512))
        self.fc1_bias = nn.Parameter(torch.randn(512))
        self.fc2_weight = nn.Parameter(torch.randn(512, 256))
        self.fc2_bias = nn.Parameter(torch.randn(256))
        self.out_weight = nn.Parameter(torch.randn(256, 10))
        self.out_bias = nn.Parameter(torch.randn(10))

    def forward(self, x):
        x = torch.relu(self.conv1(x))
        x = self.pool(x)
        # GELU via erf approximation
        x = self.conv2(x)
        x = 0.5 * x * (1 + torch.erf(x * 0.7071067811865476))
        x = self.pool(x)
        x = self.conv3(x)
        x = 0.5 * x * (1 + torch.erf(x * 0.7071067811865476))
        x = self.pool(x)
        x = self.flatten(x)
        # addmm pattern 1: mm + bias
        x = torch.mm(x, self.fc1_weight) + self.fc1_bias
        x = torch.relu(x)
        # addmm pattern 2: bias + mm
        x = self.fc2_bias + torch.mm(x, self.fc2_weight)
        x = torch.relu(x)
        x = torch.mm(x, self.out_weight) + self.out_bias
        return x

device = "cuda"
model = AddmmModel().to(device).eval()
x = torch.randn(4, 3, 56, 56, device=device)

# Eager: deterministic
with torch.no_grad():
    ref1 = model(x)
    ref2 = model(x)
print(f"Eager deterministic: {(ref1 - ref2).abs().max().item():.6e}")

# Compiled: different result
torch._dynamo.reset()
compiled = torch.compile(model, backend="inductor")
with torch.no_grad():
    out = compiled(x)

diff = (ref1.float() - out.float()).abs()
print(f"max_diff={diff.max().item():.6e}")
print(f"Output range: [{ref1.min().item():.2f}, {ref1.max().item():.2f}]")
print(f"Relative error: {diff.max().item() / ref1.abs().max().item():.6e}")
# Output:
# Eager deterministic: 0.000000e+00
# max_diff=1.573486e+00
# Relative error: 3.902675e-04

Behavior summary

Mode	Result	Notes
Eager	Reference output	Perfectly deterministic across 10 runs
`torch.compile(backend="inductor")`	Different output	max_diff=1.57, relative=4e-4

Notes

Eager mode is perfectly deterministic (max_var=0 across 10 runs), ruling out GPU non-determinism.
The difference compounds through 3 sequential mm+bias layers with ReLU activations.
The mm + bias pattern is fused to addmm by the Inductor, which may use a different CUBLAS algorithm.

Error logs

No error — the compiled model produces a result, but it differs from eager:

Eager deterministic: 0.000000e+00
Eager vs Compiled: max_diff=1.573486e+00, mean_diff=4.845871e-01
Output range: [-4031.81, 1971.11]
Relative max: 3.902675e-04

Versions

PyTorch version: 2.12.0.dev20260315+cu126
Python: 3.10.12
OS: Ubuntu 22.04.5 LTS (WSL2)
GPU: NVIDIA GeForce RTX 3060 Laptop GPU
CUDA: 12.6

cc @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia @csarofeen @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @aakhundov @coconutruben @jataylo

extent analysis

Fix Plan

To address the issue of numerical differences between torch.compile with inductor backend and eager mode, we can try the following steps:

Disable addmm fusion: One possible solution is to disable the addmm fusion in the Inductor backend. However, this might not be directly possible with the current API.
Use a different backend: Try using a different backend, such as aot_autograd or aot_eager, to see if the issue persists.
Modify the model: Modify the model to avoid using the mm + bias pattern, which is fused to addmm by the Inductor.

Here's an example of how you can modify the model to avoid using the mm + bias pattern:

class AddmmModel(nn.Module):
    def __init__(self):
        super().__init__()
        # ... (rest of the model remains the same)

    def forward(self, x):
        # ... (rest of the forward pass remains the same)
        # Replace mm + bias with a custom implementation
        x = torch.matmul(x, self.fc1_weight)
        x = x + self.fc1_bias
        x = torch.relu(x)
        # ... (rest of the forward pass remains the same)

Alternatively, you can try using a custom addmm implementation that avoids fusion:

class CustomAddmm(nn.Module):
    def __init__(self, weight, bias):
        super().__init__()
        self.weight = weight
        self.bias = bias

    def forward(self, x):
        return torch.matmul(x, self.weight) + self.bias

class AddmmModel(nn.Module):
    def __init__(self):
        super().__init__()
        # ... (rest of the model remains the same)
        self.fc1_addmm = CustomAddmm(self.fc1_weight, self.fc1_bias)
        # ... (rest of the model remains the same)

    def forward(self, x):
        # ... (rest of the forward pass remains the same)
        x = self.fc1_addmm(x)
        x = torch.relu(x)
        # ... (rest of the forward pass remains the same)

Verification

To verify that the fix worked, you can compare the output of the modified model with the output of the original model in eager mode:

# Eager mode
with torch.no_grad():
    ref1 = model(x)
    ref2 = model(x)
print(f"Eager deterministic: {(ref1 - ref2).abs().max().item():.6e}")

# Compiled mode
torch._dynamo.reset()
compiled = torch.compile(model, backend="inductor")
with torch.no_grad():
    out = compiled(x)

diff = (ref1.float() - out.float()).abs()
print(f"max_diff={diff.max().item():.6e

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #optimization #API middleware #SSR setup #ISR setup #authentication setup

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

pytorch - 💡(How to fix) Fix `torch.compile` produces numerically different results for addmm fusion pattern (`mm + bias`) compared to eager mode [6 comments, 5 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Eager: deterministic

Compiled: different result

Output:

Eager deterministic: 0.000000e+00

max_diff=1.573486e+00

Relative error: 3.902675e-04

Root Cause

Code Example

🐛 Describe the bug

Minimal reproducer

Behavior summary

Notes

Error logs

Versions

extent analysis

Fix Plan

Verification

Still need to ship something?

TRENDING

pytorch - 💡(How to fix) Fix `torch.compile` produces numerically different results for addmm fusion pattern (`mm + bias`) compared to eager mode [6 comments, 5 participants]

Recommended Tools

GitHub issue graph ai analysis

Error Message

Eager: deterministic

Compiled: different result

Output:

Eager deterministic: 0.000000e+00

max_diff=1.573486e+00

Relative error: 3.902675e-04

Root Cause

Code Example

🐛 Describe the bug

Minimal reproducer

Behavior summary

Notes

Error logs

Versions

extent analysis

Fix Plan

Verification

Still need to ship something?

RELATED_DISCOVERY

TRENDING