Problem Summary

Switching from Muon optimizer to PyTorch's MuonWithAuxAdam optimizer.

Root Cause Analysis

The issue is due to the lack of a minimal example in the PyTorch documentation for using MuonWithAuxAdam optimizer.

Fix Plan

To fix this issue, we need to create a custom optimizer using MuonWithAuxAdam. Here are the steps:

Step 1: Import necessary modules

from muon import MuonWithAuxAdam
import torch

Step 2: Separate model parameters into different groups

hidden_weights = [p for p in model.body.parameters() if p.ndim >= 2]
hidden_gains_biases = [p for p in model.body.parameters() if p.ndim < 2]
nonhidden_params = [*model.head.parameters(), *model.embed.parameters()]

Step 3: Create parameter groups for the optimizer

param_groups = [
    dict(params=hidden_weights, use_muon=True,
         lr=0.02, weight_decay=0.01),
    dict(params=hidden_gains_biases+nonhidden_params, use_muon=False,
         lr=3e-4, betas=(0.9, 0.95), weight_decay=0.01),
]

Step 4: Create the optimizer

optimizer = MuonWithAuxAdam(param_groups)

Verification

To verify that the fix worked, you can check if the optimizer is created correctly and if the model is being updated correctly during training.

Extra Tips

Make sure to update the PyTorch documentation with a minimal example for using MuonWithAuxAdam optimizer.