pytorch - 💡(How to fix) Fix torch.fmax with uint8 out produces inconsistent overflow casting on CPU and CUDA [1 participants]

🐛 Describe the bug

Summary

torch.fmax produces inconsistent results between CPU and CUDA when the inputs are int64 tensors and the out tensor has dtype uint8.

For the same operation, CPU writes 160 to the uint8 output, while CUDA writes 255.

This suggests that CPU and CUDA use different overflow conversion behavior when writing the int64 result into a narrower unsigned integer out tensor.

Reproduction

import torch

lhs = torch.tensor([100000], dtype=torch.int64)
rhs = torch.tensor([-1], dtype=torch.int64)

out_cpu = torch.empty(1, dtype=torch.uint8)
out_cuda = torch.empty(1, dtype=torch.uint8, device="cuda")

torch.fmax(lhs, rhs, out=out_cpu)
torch.fmax(lhs.cuda(), rhs.cuda(), out=out_cuda)

print("cpu:", out_cpu)
print("cuda:", out_cuda.cpu())

Actual output

cpu: tensor([160], dtype=torch.uint8)
cuda: tensor([255], dtype=torch.uint8)

Expected behavior

CPU and CUDA should use consistent casting behavior when writing the result of torch.fmax into an out tensor with dtype uint8.

The mathematically selected value is 100000, but this value cannot be represented by uint8. The operation should either:

1. produce the same converted value on both backends, or
2. document that overflow behavior for integer out tensors is backend-dependent.

Why this seems like a bug

This is not a floating-point tolerance issue. The inputs are integer tensors, and the selected maximum value is exactly 100000.

The discrepancy appears only when the result is written into a narrower uint8 output tensor:

CPU result : 160  # consistent with wraparound / modulo 256
CUDA result: 255  # consistent with saturation to uint8 max

This indicates inconsistent integer overflow casting semantics between CPU and CUDA for torch.fmax(..., out=...).

### Versions

PyTorch 2.11.0+cu128, CUDA 12.8