Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/inductor/test_mix_order_reduction.py", line 363, in test_rms_norm_bwd ref = fwd_bwd(f) File "/var/lib/jenkins/workspace/test/inductor/test_mix_order_reduction.py", line 332, in fwd_bwd out.backward(dy) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 631, in backward torch.autograd.backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/init.py", line 379, in backward _engine_run_backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 877, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 124.00 MiB. GPU 0 has a total capacity of 22.03 GiB of which 13.77 GiB is free. Process 9284 has 186.00 MiB memory in use. Process 9343 has 596.00 MiB memory in use. Process 18017 has 186.00 MiB memory in use. Process 18051 has 528.00 MiB memory in use. Process 18638 has 186.00 MiB memory in use. Including non-PyTorch memory, this process has 6.59 GiB memory in use. 6.39 GiB allowed; Of the allocated memory 6.22 GiB is allocated by PyTorch, and 138.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

To execute this test, run the following from the base repo dir: python test/inductor/test_mix_order_reduction.py NoMixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_False

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

Fix Plan

Increase GPU Memory Allocation

To fix the CUDA out of memory error, we need to increase the GPU memory allocation.

Step 1: Update PYTORCH_CUDA_ALLOC_CONF Environment Variable

Update the PYTORCH_CUDA_ALLOC_CONF environment variable to expandable_segments:True to avoid memory fragmentation.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Step 2: Increase GPU Memory Allocation

Increase the GPU memory allocation by setting the CUDA_VISIBLE_DEVICES environment variable to a specific GPU device with more memory.

export CUDA_VISIBLE_DEVICES=0

Step 3: Update Test Code

Update the test code to use a smaller batch size or reduce the number of iterations to reduce memory usage.

import torch

# Reduce batch size
batch_size = 32
# Reduce number of iterations
num_iterations = 10

Step 4: Run Test with Updated Configuration

Run the test with the updated configuration.

python test/inductor/test_mix_order_reduction.py NoMixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_False

Verification

Verify that the fix worked by checking the test output and GPU memory usage.

• Check the test output for any errors or warnings.
• Use nvidia-smi to check the GPU memory usage.

nvidia-smi

Extra Tips

• Monitor GPU memory usage and adjust the configuration as needed.
• Consider using a larger GPU device or a distributed training setup to reduce memory usage.
• Refer to the PyTorch documentation for more information on memory management and optimization.