transformers - 💡(How to fix) Fix Qwen3.5 + `flash_attention_2` crashes: 3D M-RoPE position_ids leak to `_is_packed_sequence` [2 comments, 3 participants]

Qwen3.5 + flash_attention_2 crashes: 3D M-RoPE position_ids leak to _is_packed_sequence

System Info

• transformers: 5.3.0, PyTorch: 2.6.0+cu124, flash-attn: 2.8.3, Python: 3.10, Linux

Reproduction

Fine-tuning Qwen3.5-9B with attn_implementation="flash_attention_2" crashes with CUDA error: an illegal memory access inside flash_attn_varlen_func.

Root Cause

Qwen3_5TextModel.forward passes 3D M-RoPE position_ids [3, batch, seq_len] to decoder layers. The attention layer doesn't use them (it uses position_embeddings), but they leak through **kwargs to _flash_attention_forward → _is_packed_sequence, which misinterprets the 3D shape and incorrectly routes to flash_varlen_fn with garbage cu_seqlens.

Qwen3VLTextModel.forward (same M-RoPE architecture) avoids this by passing the 2D text_position_ids to decoder layers instead:

# Qwen3-VL (correct): passes 2D text_position_ids
position_ids=text_position_ids,

# Qwen3.5 (buggy): passes 3D position_ids that leaks to flash attention
position_ids=position_ids,

Suggested Fix

Either pass text_position_ids (2D) to decoder layers in Qwen3_5TextModel.forward (matching Qwen3-VL), or add if position_ids.ndim > 2: return False to _is_packed_sequence.

Workaround

import transformers.modeling_flash_attention_utils as _fa_utils
_orig = _fa_utils._is_packed_sequence

def _patched(position_ids, **kwargs):
    if position_ids is not None and position_ids.dim() > 2:
        return False
    return _orig(position_ids, **kwargs)

_fa_utils._is_packed_sequence = _patched

Who can help?

@zucchini-nlp @ArthurZucker @Cyrilvallez