vllm - 💡(How to fix) Fix [Bug]: EAGLE Autograd issue in Mix Hidden States mode [1 comments, 1 participants]

[rank1]: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [CUDABFloat16Type [2, 4096, 2880]], which is output 0 of IndexPutBackward0, is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Fix Plan

The fix involves disabling mix hidden states or modifying the model to avoid inplace operations. Here are the steps:

• Disable mix hidden states by setting mix_hidden_states=False in the model configuration.
• Alternatively, modify the model to avoid inplace operations by using the torch.autograd.set_detect_anomaly(True) hint.

Example Code

import torch

# Disable mix hidden states
model_config = {
    'mix_hidden_states': False
}

# Alternatively, detect anomaly
torch.autograd.set_detect_anomaly(True)

# Example model modification to avoid inplace operations
class EagleModel(torch.nn.Module):
    def __init__(self):
        super(EagleModel, self).__init__()
        self.linear = torch.nn.Linear(4096, 2880)

    def forward(self, x):
        # Avoid inplace operations by cloning the tensor
        x_clone = x.clone()
        return self.linear(x_clone)

Verification

To verify the fix, re-run the training script with the modified model configuration or code. Check that the error message is no longer present and that the training process completes successfully.

Extra Tips

• When using torch.autograd.set_detect_anomaly(True), be aware that it may impact performance.
• Consider using torch.nn.utils.parameters_to_vector and torch.nn.utils.vector_to_parameters to avoid inplace operations when updating model parameters.