pytorch - 💡(How to fix) Fix `make_graphed_callables` silently drops in-place mutations on user-provided input tensors [5 comments, 4 participants]

torch.cuda.make_graphed_callables silently produces incorrect results when the captured callable mutates its input tensors in-place. The mutation occurs on an internal static copy but is never propagated back to the caller's tensor. No error or warning is raised.

• Raise an error / emit a warning

Fix Plan

To address the issue with torch.cuda.make_graphed_callables silently producing incorrect results when the captured callable mutates its input tensors in-place, we can implement a fix that detects the mutation and either copies the mutated values back to the original tensor or raises an error.

Step-by-Step Solution

1. Detect Mutation: Compare the static buffer before and after replay to detect any mutations.
2. Copy Mutated Values: If a mutation is detected, copy the mutated values back to the original tensor.
3. Raise Error or Warning: Alternatively, raise an error or emit a warning to notify the user of the mutation.

Example Code

import torch

def fn(x):
    x.add_(1)  # in-place mutation
    return x * 2

def make_graphed_callables_with_mutation_detection(fn, args):
    # Create a static buffer to store the input tensors
    static_buffer = [arg.clone() for arg in args]
    
    # Replay the function with the static buffer
    output = fn(*static_buffer)
    
    # Detect mutation by comparing the static buffer before and after replay
    for i, arg in enumerate(args):
        if not torch.equal(arg, static_buffer[i]):
            # Copy mutated values back to the original tensor
            args[i].copy_(static_buffer[i])
            print("Warning: In-place mutation detected. Mutated values copied back to original tensor.")
    
    return output

sample = torch.randn(4, device="cuda")
x = torch.randn(4, device="cuda")
original = x.clone()
out = make_graphed_callables_with_mutation_detection(fn, (x,))

print(f"x before: {original}")
print(f"x after:  {x}")
print(f"x changed: {not torch.equal(x, original)}")

Verification

To verify that the fix worked, run the example code and check that the output tensor x has been mutated correctly. The x changed flag should be True, indicating that the in-place mutation was successfully detected and propagated back to the original tensor.

Extra Tips

• When using torch.cuda.make_graphed_callables, be aware of the potential issue with in-place mutations on user-provided input tensors.
• Consider using torch.compile(mode="reduce-overhead") as an alternative, which detects input mutations and safely falls back to eager execution with a logged warning.