pytorch - 💡(How to fix) Fix C++ Can't capture Transformer CUDA Graph [1 comments, 1 participants]

Fix Plan

The fix involves modifying the torch::equal call in the Multi Headed Attention layer to avoid using .item() in CUDA Graph Capture mode.

Step-by-Step Solution

• Identify the torch::equal call in the Multi Headed Attention layer.
• Replace .item() with a tensor comparison that works in CUDA Graph Capture mode.
• Example code snippet:

// Before
if (torch::equal(input, other).item<bool>()) {
    // ...
}

// After
auto equality = torch::equal(input, other);
if (equality.all().item<bool>()) {
    // ...
}

Alternatively, you can use torch::allclose for floating point comparisons:

if (torch::allclose(input, other).all().item<bool>()) {
    // ...
}

Verification

Verify the fix by running the Transformer layer with CUDA Graph Capture mode enabled and checking that it no longer fails due to the torch::equal call.

Extra Tips

• Make sure to test the modified code thoroughly to ensure it works correctly in all scenarios.
• Consider adding a check to ensure that the input tensors are on the same device before comparing them.