transformers - 💡(How to fix) Fix AutoModelForSequenceClassification with attn_implementation="flash_attention_3" causes degenerate training (loss increases, model predicts all-one-class) [1 comments, 2 participants]

Fix Plan

The issue seems to be related to the attn_implementation="flash_attention_3" in the Qwen3ForSequenceClassification model. To fix this, we can try the following steps:

• Disable flash attention: Remove the attn_implementation="flash_attention_3" argument when loading the model.
• Update flash-attn library: Ensure that the flash-attn library is up-to-date, as newer versions may have fixed the issue.
• Modify the model configuration: Try modifying the model configuration to use a different attention implementation.

Here's an example code snippet that demonstrates how to disable flash attention:

model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen/Qwen3-Embedding-8B",
    num_labels=2,
    torch_dtype=torch.bfloat16,
    # Remove attn_implementation argument
)

Alternatively, you can try updating the flash-attn library and then load the model with the attn_implementation="flash_attention_3" argument.

Verification

To verify that the fix worked, you can train the model and check the loss and evaluation metrics. The loss should decrease, and the evaluation metrics (e.g., F1 score) should improve.

Extra Tips

• Make sure to test the model on a small dataset before training on the full dataset to ensure that the issue is resolved.
• If the issue persists, try debugging the model by printing the intermediate outputs and gradients to identify where the problem lies.
• Consider opening an issue on the Hugging Face Transformers repository or the flash-attn repository if the problem is not resolved after trying the above steps.