".../transformers/loss/loss_lw_detr.py", line 139, in loss_labels pos_weights[pos_ind] = t ~~~~~~~~~~~^^^^^^^^^ RuntimeError: Index put requires the source and destination dtypes match, got Half for the destination and Float for the source.

Fix Plan

To resolve the issue, we need to ensure that the data types match when performing operations. We can achieve this by casting the pos_weights tensor to torch.float16 before assigning the value.

Here are the steps:

• Cast pos_weights to torch.float16 in the loss_labels function.
• Alternatively, you can also cast the t value to torch.float16 before assignment.

Example code snippet:

# In loss_lw_detr.py, line 139
pos_weights = pos_weights.to(torch.float16)  # Cast pos_weights to torch.float16
pos_weights[pos_ind] = t

# Alternatively, cast t to torch.float16
# pos_weights[pos_ind] = t.to(torch.float16)

However, since you are using a pre-trained model, modifying the model's code directly might not be feasible. In that case, you can try to cast the inputs to torch.float16 before passing them to the model:

inputs = {
    "pixel_values": torch.rand((1, 3, 640, 640), device="cuda", dtype=torch.float16),
    "pixel_mask": torch.ones((1, 640, 640), device="cuda", dtype=torch.float16),
    "labels": [
        {
            "class_labels": torch.tensor([1], device="cuda", dtype=torch.float16),
            "boxes": torch.tensor([[0.5, 0.5, 0.1, 0.1]], device="cuda", dtype=torch.float16),
        }
    ],
}

Verification

To verify that the fix worked, run your original code with the modified inputs or model. The code should execute without throwing a RuntimeError due to type mismatch.

Extra Tips

When working with mixed precision training, it's essential to ensure that the data types of the tensors match during operations. You can use the to() method to cast tensors to the desired data type. Additionally, be aware of the data types of the inputs and outputs of the model to avoid type mismatches.