System Info

• transformers version: 5.7.0.dev0 (main, c472755e79)
• Platform: macOS-26.1-arm64, Apple M5, 32 GB
• Python version: 3.12.13
• PyTorch version: 2.11.0 (MPS available)

Who can help?

@Cyrilvallez (this is the MPS counterpart to the XPU branch you added).

Reproduction

Running test_eager_matches_sdpa_inference on the MPS backend fails for every fp16 parametrization across all SDPA-supporting models. Concretely:

$ TRANSFORMERS_TEST_DEVICE=mps python -m pytest \
    tests/models/llama/test_modeling_llama.py \
    tests/models/gemma/test_modeling_gemma.py \
    tests/models/qwen2/test_modeling_qwen2.py \
    tests/models/mistral/test_modeling_mistral.py \
    -k "test_eager_matches_sdpa_inference and fp16"
...
======== 24 failed, 779 deselected, 25 warnings in 3.40s ========

Failure pattern (representative):

LlamaModelTest::test_eager_matches_sdpa_inference_00_fp16_pad_left_sdpa_kernels
ValueError: mean relative difference for hidden_states: 2.962e-05,
            torch atol = 1e-07, torch rtol = 0.0001

The underlying numerics are correct: max_abs_diff ≈ 1.95e-3, mean_rel ≈ 9e-4, both well within the fp16 tolerance the test uses on CPU/CUDA. The same tests pass on CPU with the same model/seed/inputs.

Root cause

In tests/test_modeling_common.py, lines 462-476:

if torch_device in ["cpu", "cuda"]:
    atol = atols[torch_device, enable_kernels, dtype]
    rtol = rtols[torch_device, enable_kernels, dtype]
elif torch_device in ["hpu", "npu"]:
    atol = atols["cuda", enable_kernels, dtype]
    rtol = rtols["cuda", enable_kernels, dtype]
elif torch_device == "xpu":
    atol = atols["cuda", False, dtype]
    rtol = rtols["cuda", False, dtype]
else:
    atol = 1e-7
    rtol = 1e-4

MPS falls through to atol=1e-7, rtol=1e-4, which is fp32-tight and impossible to satisfy for fp16. MPS only supports SDPBackend.MATH (same as XPU, no flash / mem_efficient kernels), so the correct dispatch is the same one used for XPU.

The same dispatch block is duplicated in tests/models/video_llama_3/test_modeling_video_llama_3.py:233-247 and has the same defect.

Precedent

This is the MPS counterpart to #34888 / PR #34889, which fixed the identical issue for the XPU backend. The suggested fix is to add mps to the same branch as xpu:

elif torch_device in ("xpu", "mps"):
    atol = atols["cuda", False, dtype]
    rtol = rtols["cuda", False, dtype]

Verification

With that two-file change applied locally on M5 + torch 2.11.0:

• Before fix, -k "test_eager_matches_sdpa_inference and fp16" on Llama+Gemma+Qwen2+Mistral: 24 failed.
• After fix, -k "test_eager_matches_sdpa_inference" (all dtypes) on the same 4 models: 100 passed.
• After fix, same test on Llama+Gemma2+Gemma3+Qwen3: 124 passed, 1 skipped.
• After fix, test_modeling_video_llama_3.py on MPS: 24 passed, 8 skipped (fp32 output_attentions, unrelated).
• CPU regression check (same tests, default device): unchanged, 16 passed.

Expected behavior

test_eager_matches_sdpa_inference should pass on MPS for the same model/dtype combinations it passes on CPU.

Offering to PR

Happy to open the PR. Patch is around 8 lines across the two files above, mirrors PR #34889 exactly, and I have an M5 to run the suite against. Please confirm you'd like the same dispatch (XPU-style MATH-only tolerances) for MPS rather than a separate MPS-specific entry.