There is currently an error: :0:rocdevice.cpp :3675: 1143953758580 us: Callback: Queue 0x7eb85a100000 aborting with error : HSA_STATUS_ERROR_MEMORY_APERTURE_VIOLATION: The agent attempted to access memory beyond the largest legal address. code: 0x29 Even with enforce_eager set to True error happens.

Fix Plan

The fix involves addressing the memory aperture violation error by optimizing memory usage in the model execution.

• Step 1: Reduce Model Size
  • Review the model architecture and reduce the size of the model if possible.
  • This can be done by reducing the number of layers, the number of parameters in each layer, or using a more efficient architecture.
• Step 2: Optimize Memory Allocation
  • Ensure that memory allocation is optimized for the GPU.
  • This can be done by using torch.cuda.empty_cache() to release any unused memory before running the model.
  • Example:

import torch

Release unused memory

torch.cuda.empty_cache()

Run the model

model_output = model(input_data)

*   **Step 3: Use Gradient Checkpointing**
    *   Implement gradient checkpointing to reduce memory usage during backpropagation.
    *   This can be done using `torch.utils.checkpoint` module.
    *   Example:
        ```python
import torch
from torch.utils.checkpoint import checkpoint

# Define a checkpointed function
def checkpointed_function(x):
    return checkpoint(model, x)

# Run the checkpointed function
model_output = checkpointed_function(input_data)

• Step 4: Increase GPU Memory
  • If possible, increase the GPU memory available to the model.
  • This can be done by using a GPU with more memory or by reducing the batch size.

Verification

To verify that the fix worked, run the test again with the optimized model and memory allocation:

HIP_VISIBLE_DEVICES=7 pytest -s -v tests/models/language/generation_ppl_test/test_qwen.py::test_ppl[model_info2]

If the test passes without any memory errors, the fix is successful.

Extra Tips

• Monitor GPU memory usage during model execution to identify potential memory bottlenecks.
• Use tools like nvidia-smi to monitor GPU memory usage.
• Consider using mixed precision training to reduce memory usage.