Issue Station
Sign In
Tools hereToolbox

pytorch - 💡(How to fix) Fix Numerical discrepancy in torch.special.erfcx between CPU and CUDA implementations

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…

Code Example

import torch
import torch.nn as nn

torch.manual_seed(0)

fc1 = nn.Linear(8, 8)
bn  = nn.BatchNorm1d(8)
fc2 = nn.Linear(8, 8)
logsigmoid = nn.LogSigmoid()

def forward(x):
    x = fc1(x)
    x = torch.nn.functional.hardshrink(x)
    x = bn(x)
    x = fc2(x)
    x = torch.sqrt(x.abs())          # ensure non-negative for sqrt
    x = torch.special.erfcx(x)       # scaled complementary erf — CPU/GPU differ
    return logsigmoid(x)

fc1_g = nn.Linear(8, 8).cuda(); fc1_g.load_state_dict(fc1.state_dict())
bn_g  = nn.BatchNorm1d(8).cuda(); bn_g.load_state_dict(bn.state_dict())
fc2_g = nn.Linear(8, 8).cuda(); fc2_g.load_state_dict(fc2.state_dict())
ls_g  = nn.LogSigmoid().cuda()

def forward_gpu(x):
    x = fc1_g(x)
    x = torch.nn.functional.hardshrink(x)
    x = bn_g(x)
    x = fc2_g(x)
    x = torch.sqrt(x.abs())
    x = torch.special.erfcx(x)
    return ls_g(x)

x = torch.randn(4, 8)
cpu_out = forward(x)
gpu_out = forward_gpu(x.cuda()).cpu()

diff = (cpu_out - gpu_out).abs()
print(f"max |cpu - gpu|: {diff.max():.4e}")

# Minimal 2-line version:
print("\nMinimal:")
v = torch.rand(4, 8) * 3          # erfcx is most sensitive in [0, 3]
print(f"erfcx cpu/gpu diff: {(torch.special.erfcx(v) - torch.special.erfcx(v.cuda()).cpu()).abs().max():.4e}")
RAW_BUFFERClick to expand / collapse

🐛 Describe the bug

import torch
import torch.nn as nn

torch.manual_seed(0)

fc1 = nn.Linear(8, 8)
bn  = nn.BatchNorm1d(8)
fc2 = nn.Linear(8, 8)
logsigmoid = nn.LogSigmoid()

def forward(x):
    x = fc1(x)
    x = torch.nn.functional.hardshrink(x)
    x = bn(x)
    x = fc2(x)
    x = torch.sqrt(x.abs())          # ensure non-negative for sqrt
    x = torch.special.erfcx(x)       # scaled complementary erf — CPU/GPU differ
    return logsigmoid(x)

fc1_g = nn.Linear(8, 8).cuda(); fc1_g.load_state_dict(fc1.state_dict())
bn_g  = nn.BatchNorm1d(8).cuda(); bn_g.load_state_dict(bn.state_dict())
fc2_g = nn.Linear(8, 8).cuda(); fc2_g.load_state_dict(fc2.state_dict())
ls_g  = nn.LogSigmoid().cuda()

def forward_gpu(x):
    x = fc1_g(x)
    x = torch.nn.functional.hardshrink(x)
    x = bn_g(x)
    x = fc2_g(x)
    x = torch.sqrt(x.abs())
    x = torch.special.erfcx(x)
    return ls_g(x)

x = torch.randn(4, 8)
cpu_out = forward(x)
gpu_out = forward_gpu(x.cuda()).cpu()

diff = (cpu_out - gpu_out).abs()
print(f"max |cpu - gpu|: {diff.max():.4e}")

# Minimal 2-line version:
print("\nMinimal:")
v = torch.rand(4, 8) * 3          # erfcx is most sensitive in [0, 3]
print(f"erfcx cpu/gpu diff: {(torch.special.erfcx(v) - torch.special.erfcx(v.cuda()).cpu()).abs().max():.4e}")

Versions

2.9.1+cu128 (PyTorch 2.9.1, CUDA 12.8)

torch.special.erfcx produces noticeably different results between CPU and CUDA for the same inputs. The discrepancy can be amplified in typical neural network pipelines and may affect numerical stability and reproducibility.

cc @ptrblck @msaroufim @eqy @jerryzh168 @tinglvv @nWEIdia @mruberry @kshitij12345

extent analysis

TL;DR

The discrepancy in torch.special.erfcx results between CPU and CUDA can be mitigated by using a consistent computation device for the entire pipeline.

Guidance

  • Identify the specific operations that are sensitive to device differences, such as torch.special.erfcx, and ensure they are executed on the same device.
  • Consider using the torch.cuda device for the entire pipeline to maintain consistency, especially when using CUDA-enabled PyTorch versions like 2.9.1+cu128.
  • To verify the fix, compare the results of the forward and forward_gpu functions after ensuring consistent device usage.
  • Be aware that device differences may affect numerical stability and reproducibility in neural network pipelines.

Example

No explicit code example is provided, but the issue code itself demonstrates the discrepancy and can be used as a starting point for testing the mitigation strategy.

Notes

The provided code snippet and issue description suggest that the discrepancy is most noticeable for inputs in the range [0, 3], which may be a common range for certain neural network activations.

Recommendation

Apply workaround: Ensure consistent device usage throughout the pipeline to mitigate the discrepancy in torch.special.erfcx results. This approach allows for maintaining the current PyTorch version while addressing the specific issue at hand.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#prompt template#agent execution#callback error#memory management#API rate limit

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING