Issue Station
Sign In
Tools hereToolbox

vllm - 💡(How to fix) Fix [CI Failure]: mi355_1: Entrypoints Integration (Pooling) [2 comments, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#41342•Fetched 2026-05-01 05:34:06
View on GitHub
Comments
2
Participants
1
Timeline
8
Reactions
0
Author
AndreasKaratzasAndreasKaratzas
Participants
AndreasKaratzasAndreasKaratzas
Timeline (top)
added_to_project_v2 ×2commented ×2labeled ×1mentioned ×1

Root Cause

  • Flaky test
  • Can reproduce locally
  • Caused by external libraries (e.g. bug in transformers)

Code Example

FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_str[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_text_content[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_list[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_rerank_api_queries_str_documents_list[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_list_documents_list[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_str[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_text_content[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_list[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_rerank_api_queries_str_documents_list[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_list_documents_list[FLEX_ATTENTION]
RAW_BUFFERClick to expand / collapse

Name of failing test

pytest -s -v entrypoints/pooling/scoring/test_cross_encoder_online_vision.py

Basic information

  • Flaky test
  • Can reproduce locally
  • Caused by external libraries (e.g. bug in transformers)

🧪 Describe the failing test

FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_str[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_text_content[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_list[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_rerank_api_queries_str_documents_list[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_list_documents_list[ROCM_AITER_FA]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_str[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_text_content[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_str_documents_list[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_rerank_api_queries_str_documents_list[FLEX_ATTENTION]
FAILED entrypoints/pooling/scoring/test_cross_encoder_online_vision.py::test_score_api_queries_list_documents_list[FLEX_ATTENTION]

📝 History of failing test

  • Current streak start: 2026-04-27
  • First failure in 60d window: 2026-03-31
  • Last successful nightly: 2026-04-26
  • Break frequency (60d, pass↔fail flips): 6
  • Latest nightly date: 2026-04-29
  • Latest build(s): amd-ci #8058
  • Latest hardware status: mi355_1=fail

extent analysis

TL;DR

Investigate the ROCM_AITER_FA and FLEX_ATTENTION configurations to identify the cause of the test failures in test_cross_encoder_online_vision.py.

Guidance

  • Review the test code in test_cross_encoder_online_vision.py to understand the differences between the ROCM_AITER_FA and FLEX_ATTENTION configurations.
  • Check the hardware status and build logs for any errors or warnings related to the mi355_1 hardware.
  • Investigate the changes made to the code or environment between the last successful nightly build (2026-04-26) and the first failure in the 60d window (2026-03-31).
  • Run the tests with additional logging or debugging to gather more information about the failures.

Notes

The issue seems to be related to specific configurations and hardware, so the solution may depend on the details of the test environment and code changes.

Recommendation

Apply workaround: Investigate and fix the issues related to the ROCM_AITER_FA and FLEX_ATTENTION configurations, as they are the common factors in the failed tests.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#tokenizer error#prompt formatting#chain error#conversation history

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING