Fix Plan

To address the OOMing event of Gemma3, we will implement a solution to reduce the memory usage of the fake tensor used for profiling.

Steps to Fix

• Reduce the size of the fake tensor to fit within the 64 GB memory limit of the MI250 GPUs.
• Implement a more efficient profiling method that doesn't require large tensors.

Example Code

# Before
fake_tensor = torch.randn(4096, 4096)  # large tensor

# After
fake_tensor = torch.randn(1024, 1024)  # reduced size tensor

Alternatively, consider using a more memory-efficient profiling method, such as:

# Using a smaller tensor and iterating over it
for i in range(0, 4096, 1024):
    fake_tensor = torch.randn(1024, 1024)
    # profiling code here

Verification

Verify that the OOMing event is resolved by running the profiling test with the reduced tensor size or the new profiling method.

Extra Tips

• Monitor memory usage during profiling to ensure the fix is effective.
• Consider implementing a dynamic tensor sizing mechanism to adapt to different GPU memory limits.