pytorch - 💡(How to fix) Fix DISABLED test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_True (__main__.MixOrderReductionTest) [2 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/inductor/test_mix_order_reduction.py", line 363, in test_rms_norm_bwd ref = fwd_bwd(f) File "/var/lib/jenkins/workspace/test/inductor/test_mix_order_reduction.py", line 332, in fwd_bwd out.backward(dy) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 631, in backward torch.autograd.backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/init.py", line 379, in backward _engine_run_backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 877, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 124.00 MiB. GPU 0 has a total capacity of 22.03 GiB of which 13.34 GiB is free. Process 13154 has 186.00 MiB memory in use. Process 13173 has 1.28 GiB memory in use. Process 13212 has 186.00 MiB memory in use. Process 13231 has 334.00 MiB memory in use. Process 16329 has 186.00 MiB memory in use. Including non-PyTorch memory, this process has 6.51 GiB memory in use. 6.39 GiB allowed; Of the allocated memory 6.22 GiB is allocated by PyTorch, and 62.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

To execute this test, run the following from the base repo dir: python test/inductor/test_mix_order_reduction.py MixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_True

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

Fix Plan

1. Increase GPU Memory Allocation

To fix the CUDA out of memory error, we need to increase the GPU memory allocation. We can do this by setting the PYTORCH_CUDA_ALLOC_CONF environment variable to expandable_segments=True.

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments=True

2. Reduce Memory Usage

We can also try to reduce memory usage by using the torch.cuda.empty_cache() function to free up unused CUDA memory.

import torch

# Before running the test
torch.cuda.empty_cache()

3. Increase Batch Size

If the test is running on a small batch size, we can try increasing the batch size to reduce the memory usage.

# In the test file
batch_size = 1024  # Increase the batch size

4. Use Mixed Precision Training

We can also try using mixed precision training to reduce memory usage.

# In the test file
from torch.cuda.amp import autocast

# Use mixed precision training
with autocast():
    # Run the test

5. Run the Test on a Larger GPU

If none of the above solutions work, we can try running the test on a larger GPU.

# Run the test on a larger GPU
nvidia-smi -L  # List available GPUs
nvidia-smi -i <GPU_ID> --mem  # Set the memory for the GPU

Verification

To verify that the fix worked, we can run the test again and check if the CUDA out of memory error is still occurring. We can also use the nvidia-smi command to check the GPU memory usage.

# Run the test again
python test/inductor/test_mix_order_reduction.py MixOrderReductionTest.test_rms_norm_bwd_b