Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/inductor/test_mix_order_reduction.py", line 363, in test_rms_norm_bwd ref = fwd_bwd(f) File "/var/lib/jenkins/workspace/test/inductor/test_mix_order_reduction.py", line 332, in fwd_bwd out.backward(dy) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 631, in backward torch.autograd.backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/init.py", line 379, in backward _engine_run_backward( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 877, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 124.00 MiB. GPU 0 has a total capacity of 22.03 GiB of which 13.27 GiB is free. Process 13154 has 186.00 MiB memory in use. Process 13173 has 1.28 GiB memory in use. Process 13212 has 186.00 MiB memory in use. Process 13231 has 334.00 MiB memory in use. Process 16329 has 186.00 MiB memory in use. Including non-PyTorch memory, this process has 6.59 GiB memory in use. 6.39 GiB allowed; Of the allocated memory 6.22 GiB is allocated by PyTorch, and 138.28 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://docs.pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf)

To execute this test, run the following from the base repo dir: python test/inductor/test_mix_order_reduction.py NoMixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_True

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

Fix Plan

1. Increase GPU Memory Allocation

• Set the PYTORCH_CUDA_ALLOC_CONF environment variable to expandable_segments:True to avoid memory fragmentation.
• Run the test with the following command:

  PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True python test/inductor/test_mix_order_reduction.py NoMixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_True

• Alternatively, you can also try to increase the GPU memory allocation by setting the CUDA_VISIBLE_DEVICES environment variable to a specific GPU with more memory.

2. Reduce Memory Usage

• Review the test code and identify any unnecessary memory allocations or large data structures.
• Optimize the test code to reduce memory usage, such as by using smaller data types or reducing the size of intermediate results.

3. Disable CUDA

• If the test is not dependent on CUDA, consider disabling it to reduce memory usage.
• Run the test with the following command:

  python test/inductor/test_mix_order_reduction.py NoMixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_True --no-cuda

4. Increase Test Timeout

• If the test is timing out due to memory allocation issues, consider increasing the test timeout.
• Run the test with the following command:

  python test/inductor/test_mix_order_reduction.py NoMixOrderReductionTest.test_rms_norm_bwd_bfloat16_split_reductions_False_shape0_max_autotune_False_initial_xblock_1_add_1dim_True --timeout 300