pytorch - 💡(How to fix) Fix Addcmul CPU bitwise numerics not matching eager w/ torch.compile [1 participants]

🐛 Describe the bug

CPU addcmul_ bitwise mismatch

Summary: The dynamo decomposition of addcmul_ (tensor value path) is not bitwise-identical to ATen on CPU. ~18% of elements differ by up to 1-2 ULP.

Root cause: ATen's CPU addcmul kernel (aten/src/ATen/native/cpu/PointwiseOpsKernel.cpp) computes the expression self + scalar * t1 * t2 as a single C++ expression in a vectorized loop. With -ffp-contract=fast (gcc/clang default at -O2), the compiler may contract the multiply-add chain into hardware FMA instructions, choosing which sub-expression to fuse. Our decomposition forces a specific operation sequence, which always gets different FP contraction.

All four decomposition strategies produce identical mismatch counts — confirming the issue is the compiler's FMA contraction of ATen's single-expression kernel vs any sequence of separate ops:

• fma(t1*t2, value, self) → 751/4096 mismatches
• self + (value*t1)*t2 → 751/4096 mismatches
• self + (t1*t2)*value → 751/4096 mismatches
• self + value*(t1*t2) → 751/4096 mismatches

Why CUDA is fine: On CUDA, ATen uses explicit std::fma intrinsics (DeviceAddCmulCdiv.cuh), and inductor_prims.fma lowers to tl.fma — both are explicit and match.

Possible fixes:

1. Create an inductor prim that emits the exact C++ expression self + value * t1 * t2 in a single codegen kernel, letting the same compiler optimize it identically to ATen
2. Control FP contraction flags (-ffp-contract) in inductor's C++ codegen to match ATen's build configuration
3. Accept ~1 ULP difference on CPU (current approach)

Repro: import torch import torch._dynamo.config

def addcmul_fn(x, tensor1, tensor2, value): return x.addcmul_(tensor1, tensor2, value=value)

torch.manual_seed(42) x = torch.randn(64, 64) t1 = torch.randn(64, 64) t2 = torch.randn(64, 64) value = torch.tensor(0.5)

expected = addcmul_fn(x.clone(), t1, t2, value)

with torch._dynamo.config.patch(enable_dynamo_decompositions=True): actual = torch.compile(addcmul_fn, fullgraph=True)(x.clone(), t1, t2, value)

diff = (expected - actual).abs() mismatches = (diff > 0).sum().item() print(f"mismatches: {mismatches} / {diff.numel()} ({100*mismatches/diff.numel():.1f}%)") print(f"max abs diff: {diff.max().item()}") print(f"bitwise equal: {torch.equal(expected, actual)}")

Error logs

No response

Versions

1d13be67fea5d1f44998483271fc258f550fa524

cc @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @aakhundov @coconutruben @jataylo