pytorch - 💡(How to fix) Fix [inductor] Gradient mismatch between eager and inductor for torch.quantile on tied values

🐛 Describe the bug

When using torch.quantile on a tensor containing tied values (e.g., identical values like 0.0), Eager mode and torch.compile (inductor backend) produce different gradients. Specifically, they make different tie-breaking choices for routing the gradient.

In the provided script (using q=0.0), Eager mode routes the gradient to the last element [0., 0., 1.], while Inductor routes it to the first element [1., 0., 0.].

Reproduction Script:

import torch

assert torch.cuda.is_available()

print("torch:", torch.__version__)
print("cuda:", torch.version.cuda)

def fn(x):
    y = torch.quantile(x, 0.0, dim=1, keepdim=True)
    return torch.nan_to_num(y, nan=0.0).sum()

def run(f, x0):
    x = x0.detach().clone().requires_grad_(True)
    loss = f(x)
    grad, = torch.autograd.grad(loss, x)
    torch.cuda.synchronize()
    return loss.detach(), grad.detach()

x = torch.tensor(
    [
        [float("nan"), float("nan"), float("nan")],
        [0.0, 0.0, 0.0],
    ],
    device="cuda",
    dtype=torch.float64,
)

compiled_fn = torch.compile(fn, backend="inductor", fullgraph=True)

loss_eager, grad_eager = run(fn, x)
loss_inductor, grad_inductor = run(compiled_fn, x)

print("\ninput:")
print(x)

print("\neager loss:")
print(loss_eager)
print("inductor loss:")
print(loss_inductor)
print("loss equal:", torch.equal(loss_eager, loss_inductor))

print("\neager grad:")
print(grad_eager)

print("\ninductor grad:")
print(grad_inductor)

print("\ngrad equal:", torch.equal(grad_eager, grad_inductor))
print("eager nonzero mask:")
print((grad_eager != 0).cpu().tolist())
print("inductor nonzero mask:")
print((grad_inductor != 0).cpu().tolist())

assert torch.equal(loss_eager, loss_inductor)
assert not torch.equal(grad_eager, grad_inductor)

Actual Output:

(torch-nightly) xyt19@Oasis:/tmp$ python bug.py
torch: 2.13.0.dev20260521+cu130
cuda: 13.0

input:
tensor([[nan, nan, nan],
        [0., 0., 0.]], device='cuda:0', dtype=torch.float64)

eager loss:
tensor(0., device='cuda:0', dtype=torch.float64)
inductor loss:
tensor(0., device='cuda:0', dtype=torch.float64)
loss equal: True

eager grad:
tensor([[0., 0., 0.],
        [0., 0., 1.]], device='cuda:0', dtype=torch.float64)

inductor grad:
tensor([[0., 0., 0.],
        [1., 0., 0.]], device='cuda:0', dtype=torch.float64)

grad equal: False
eager nonzero mask:
[[False, False, False], [False, False, True]]
inductor nonzero mask:
[[False, False, False], [True, False, False]]

Expected Behavior: The gradients produced by the inductor backend should match the eager mode gradients, or this expected divergence in tie-breaking behavior should be officially documented.

Versions

PyTorch version: 2.13.0.dev20260521+cu130 Is debug build: False CUDA used to build PyTorch: 13.0 ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04.4 LTS (x86_64) GCC version: (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0 Clang version: 18.1.3 (1ubuntu1) CMake version: version 3.28.3 Libc version: glibc-2.39

Python version: 3.10.20 (main, Mar 11 2026, 17:46:40) [GCC 14.3.0] (64-bit runtime) Python platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39 Is CUDA available: True CUDA runtime version: 12.0.140 Nvidia driver version: 596.49 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_tensor_ir.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.21.1 /usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.21.1 Is XPU available: False HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True Caching allocator config: N/A ersions of relevant libraries: [pip3] numpy==2.2.6 [pip3] nvidia-cublas==13.1.1.3 [pip3] nvidia-cuda-cupti==13.0.85 [pip3] nvidia-cuda-nvrtc==13.0.88 [pip3] nvidia-cuda-runtime==13.0.96 [pip3] nvidia-cudnn-cu13==9.20.0.48 [pip3] nvidia-cufft==12.0.0.61 [pip3] nvidia-curand==10.4.0.35 [pip3] nvidia-cusolver==12.0.4.66 [pip3] nvidia-cusparse==12.6.3.3 [pip3] nvidia-cusparselt-cu13==0.8.1 [pip3] nvidia-nccl-cu13==2.29.7 [pip3] nvidia-nvjitlink==13.0.88 [pip3] nvidia-nvtx==13.0.85 [pip3] torch==2.13.0.dev20260521+cu130 [pip3] torchaudio==2.11.0.dev20260525+cu130 [pip3] torchvision==0.28.0.dev20260525+cu130 [pip3] triton==3.7.0+git88b227e2 [conda] numpy 2.2.6 pypi_0 pypi [conda] nvidia-cublas 13.1.1.3 pypi_0 pypi [conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi [conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi [conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi [conda] nvidia-cudnn-cu13 9.20.0.48 pypi_0 pypi [conda] nvidia-cufft 12.0.0.61 pypi_0 pypi [conda] nvidia-curand 10.4.0.35 pypi_0 pypi [conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi [conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi [conda] nvidia-cusparselt-cu13 0.8.1 pypi_0 pypi [conda] nvidia-nccl-cu13 2.29.7 pypi_0 pypi [conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi [conda] nvidia-nvtx 13.0.85 pypi_0 pypi [conda] torch 2.13.0.dev20260521+cu130 pypi_0 pypi [conda] torchaudio 2.11.0.dev20260525+cu130 pypi_0 pypi [conda] torchvision 0.28.0.dev20260525+cu130 pypi_0 pypi [conda] triton 3.7.0+git88b227e2 pypi_0 pypi

cc @albanD @chauhang @penguinwu @voznesenskym @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @aakhundov @coconutruben @jataylo