vllm - ✅(Solved) Fix [Bug]: `block_size=8` triggers Triton CompilationError in FlexAttention kernel; other backends correctly reject [2 pull requests, 1 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#39340Fetched 2026-04-09 07:51:46
View on GitHub
Comments
0
Participants
1
Timeline
3
Reactions
0
Participants
Timeline (top)
cross-referenced ×1labeled ×1referenced ×1

Error Message

torch._inductor.exc.InductorError: CompilationError: at 146:20: ... acc, l_i, m_i = forward_inner( ^

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 39 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 24 On-line CPU(s) list: 0-23 Vendor ID: GenuineIntel Model name: 13th Gen Intel(R) Core(TM) i7-13700F CPU family: 6 Model: 183 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 1 CPU(s) scaling MHz: 43% CPU max MHz: 5200.0000 CPU min MHz: 800.0000 BogoMIPS: 4224.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities Virtualization: VT-x L1d cache: 640 KiB (16 instances) L1i cache: 768 KiB (16 instances) L2 cache: 24 MiB (10 instances) L3 cache: 30 MiB (1 instance) NUMA node(s): 1 NUMA node0 CPU(s): 0-23 Vulnerability Gather data sampling: Not affected Vulnerability Ghostwrite: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Mitigation; Clear Register File Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

PR fix notes

PR #39385: [Bugfix] Validate block_size in FlexAttention at construction time

Description (problem / solution / changelog)

Summary

FlexAttentionBackend.get_kv_cache_shape() now validates that block_size is a multiple of 16, matching the existing checks in TreeAttentionBackend and ROCmAttentionBackend.

Without this, block_size=8 silently passes construction and crashes later with a cryptic Triton CompilationError during kernel codegen. Other backends (FLASH_ATTN, FLASHINFER) already reject unsupported block sizes cleanly at backend selection time — FlexAttention was the outlier.

Note: FlexAttention already has a runtime check that block_size == kv_block_size (line ~620), but that fires too late — after Triton compilation has already failed.

Fixes #39340

Test plan

  • LLM(model="Qwen/Qwen3-0.6B", max_model_len=512, block_size=8) with FlexAttention selected should now raise ValueError at construction with a clear message instead of crashing in Triton codegen
  • Valid block sizes (16, 32, 64, etc.) continue to work unchanged

Changed files

  • .buildkite/.pipeline_gen_v2 (added, +0/-0)
  • .buildkite/check-wheel-size.py (added, +53/-0)
  • .buildkite/ci_config.yaml (added, +25/-0)
  • .buildkite/ci_config_intel.yaml (added, +23/-0)
  • .buildkite/hardware_tests/amd.yaml (added, +22/-0)
  • .buildkite/hardware_tests/ascend_npu.yaml (added, +10/-0)
  • .buildkite/hardware_tests/cpu.yaml (added, +110/-0)
  • .buildkite/hardware_tests/gh200.yaml (added, +10/-0)
  • .buildkite/hardware_tests/intel.yaml (added, +17/-0)
  • .buildkite/image_build/image_build.sh (added, +255/-0)
  • .buildkite/image_build/image_build.yaml (added, +58/-0)
  • .buildkite/image_build/image_build_cpu.sh (added, +34/-0)
  • .buildkite/image_build/image_build_cpu_arm64.sh (added, +33/-0)
  • .buildkite/image_build/image_build_hpu.sh (added, +34/-0)
  • .buildkite/image_build/image_build_xpu.sh (added, +34/-0)
  • .buildkite/intel_jobs/test-intel.yaml (added, +64/-0)
  • .buildkite/lm-eval-harness/configs/DeepSeek-V2-Lite-Chat.yaml (added, +13/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-70B-Instruct-FBGEMM-nonuniform.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-70B-Instruct.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-Channelwise-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-FBGEMM-nonuniform.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-FP8-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-FP8.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-INT8-compressed-tensors-asym.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-INT8-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct-nonuniform-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-Instruct.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3-8B-QQQ.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3.2-1B-Instruct-FP8-compressed-tensors.yaml (added, +11/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-3.2-1B-Instruct-INT8-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-4-Maverick-17B-128E-Instruct-FP8-MM.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Meta-Llama-4-Maverick-17B-128E-Instruct-FP8.yaml (added, +14/-0)
  • .buildkite/lm-eval-harness/configs/Minitron-4B-Base-FP8.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Mixtral-8x22B-Instruct-v0.1-FP8-Dynamic.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Mixtral-8x7B-Instruct-v0.1-FP8.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Mixtral-8x7B-Instruct-v0.1.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16.yaml (added, +15/-0)
  • .buildkite/lm-eval-harness/configs/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8.yaml (added, +19/-0)
  • .buildkite/lm-eval-harness/configs/Qwen1.5-MoE-W4A16-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Qwen2-1.5B-Instruct-FP8W8.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Qwen2-1.5B-Instruct-INT8-compressed-tensors.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Qwen2-57B-A14-Instruct.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Qwen2.5-1.5B-Instruct.yaml (added, +11/-0)
  • .buildkite/lm-eval-harness/configs/Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml (added, +15/-0)
  • .buildkite/lm-eval-harness/configs/Qwen2.5-VL-7B-Instruct.yaml (added, +12/-0)
  • .buildkite/lm-eval-harness/configs/Qwen3-235B-A22B-Instruct-2507-FP8.yaml (added, +17/-0)
  • .buildkite/lm-eval-harness/configs/models-large-hopper.txt (added, +2/-0)
  • .buildkite/lm-eval-harness/configs/models-large-rocm-fp8.txt (added, +1/-0)
  • .buildkite/lm-eval-harness/configs/models-large-rocm.txt (added, +1/-0)
  • .buildkite/lm-eval-harness/configs/models-large.txt (added, +6/-0)
  • .buildkite/lm-eval-harness/configs/models-mm-large-h100.txt (added, +1/-0)
  • .buildkite/lm-eval-harness/configs/models-mm-small.txt (added, +1/-0)
  • .buildkite/lm-eval-harness/configs/models-small-rocm.txt (added, +6/-0)
  • .buildkite/lm-eval-harness/configs/models-small.txt (added, +6/-0)
  • .buildkite/lm-eval-harness/conftest.py (added, +44/-0)
  • .buildkite/lm-eval-harness/run-lm-eval-chartqa-vllm-vlm-baseline.sh (added, +44/-0)
  • .buildkite/lm-eval-harness/run-lm-eval-gsm-hf-baseline.sh (added, +46/-0)
  • .buildkite/lm-eval-harness/run-lm-eval-gsm-vllm-baseline.sh (added, +51/-0)
  • .buildkite/lm-eval-harness/run-lm-eval-mmlupro-vllm-baseline.sh (added, +47/-0)
  • .buildkite/lm-eval-harness/test_lm_eval_correctness.py (added, +145/-0)
  • .buildkite/performance-benchmarks/README.md (added, +180/-0)
  • .buildkite/performance-benchmarks/performance-benchmarks-descriptions.md (added, +65/-0)
  • .buildkite/performance-benchmarks/scripts/compare-json-results.py (added, +1327/-0)
  • .buildkite/performance-benchmarks/scripts/convert-results-json-to-markdown.py (added, +414/-0)
  • .buildkite/performance-benchmarks/scripts/launch-server.sh (added, +224/-0)
  • .buildkite/performance-benchmarks/scripts/run-performance-benchmarks.sh (added, +896/-0)
  • .buildkite/performance-benchmarks/tests/genai-perf-tests.json (added, +21/-0)
  • .buildkite/performance-benchmarks/tests/latency-tests-arm64-cpu.json (added, +26/-0)
  • .buildkite/performance-benchmarks/tests/latency-tests-cpu.json (added, +26/-0)
  • .buildkite/performance-benchmarks/tests/latency-tests-hpu.json (added, +106/-0)
  • .buildkite/performance-benchmarks/tests/latency-tests.json (added, +32/-0)
  • .buildkite/performance-benchmarks/tests/nightly-tests.json (added, +311/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests-arm64-cpu.json (added, +131/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests-cpu-asr.json (added, +38/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests-cpu-embed.json (added, +41/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests-cpu-text.json (added, +356/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests-cpu.json (added, +143/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests-hpu.json (added, +163/-0)
  • .buildkite/performance-benchmarks/tests/serving-tests.json (added, +77/-0)
  • .buildkite/performance-benchmarks/tests/throughput-tests-arm64-cpu.json (added, +27/-0)
  • .buildkite/performance-benchmarks/tests/throughput-tests-cpu.json (added, +27/-0)
  • .buildkite/performance-benchmarks/tests/throughput-tests-hpu.json (added, +123/-0)
  • .buildkite/performance-benchmarks/tests/throughput-tests.json (added, +35/-0)
  • .buildkite/release-pipeline.yaml (added, +698/-0)
  • .buildkite/scripts/annotate-release.sh (added, +119/-0)
  • .buildkite/scripts/annotate-rocm-release.sh (added, +113/-0)
  • .buildkite/scripts/cache-rocm-base-wheels.sh (added, +131/-0)
  • .buildkite/scripts/check-ray-compatibility.sh (added, +235/-0)
  • .buildkite/scripts/cherry-pick-from-milestone.sh (added, +242/-0)
  • .buildkite/scripts/ci-clean-log.sh (added, +17/-0)
  • .buildkite/scripts/cleanup-nightly-builds.sh (added, +130/-0)
  • .buildkite/scripts/generate-and-upload-nightly-index.sh (added, +84/-0)
  • .buildkite/scripts/generate-nightly-index.py (added, +468/-0)
  • .buildkite/scripts/hardware_ci/run-amd-test.sh (added, +498/-0)
  • .buildkite/scripts/hardware_ci/run-cpu-compatibility-test.sh (added, +65/-0)
  • .buildkite/scripts/hardware_ci/run-cpu-distributed-smoke-test.sh (added, +44/-0)
  • .buildkite/scripts/hardware_ci/run-cpu-test-arm.sh (added, +73/-0)
  • .buildkite/scripts/hardware_ci/run-cpu-test-ppc64le.sh (added, +54/-0)
  • .buildkite/scripts/hardware_ci/run-cpu-test-s390x.sh (added, +13/-0)
  • .buildkite/scripts/hardware_ci/run-cpu-test.sh (added, +20/-0)

PR #39807: [Bugfix] FlexAttention: reject block_size < 16

Description (problem / solution / changelog)

Summary

block_size=8 with FlexAttention currently blows up inside Triton codegen on the first forward pass. FlexAttentionBackend was missing a get_supported_kernel_block_sizes override and inherited the base [MultipleOf(1)] default, so anything smaller than the kernel's real minimum (16) slipped past backend selection and only failed deep inside torch.compile.

The minimum block_lower_bound = 16 is already hard-coded inside get_kernel_options, but only in the non-direct-build branch — the direct-build path (torch >= 2.9) skipped it. So the 16 minimum was effectively only half-applied.

Two small changes for the same root cause:

  • Override get_supported_kernel_block_sizes on FlexAttentionBackend to return [MultipleOf(16)], so unsupported block sizes are rejected up front with the usual "block_size not supported" error, matching what FA / Triton / FlashInfer already do.
  • Lift block_lower_bound = 16 out of the non-direct-build branch of get_kernel_options and apply max(..., 16) in the direct-build branch as well, so BLOCK_M / BLOCK_N can't drop below 16 even if a smaller tile is routed through.

The attention-backend doc table was regenerated by the pre-commit hook (Any -> %16 for FLEX_ATTENTION).

Fixes #39340

Duplicate check

  • gh issue view 39340 --comments — no one has claimed this issue, no in-progress work.
  • gh pr list --state open --search "39340 in:body" — no open PR references this issue.
  • gh pr list --state open --search "FlexAttention block_size" — no overlapping PR.

Test Plan

  • .venv/bin/python -m pytest tests/kernels/attention/test_attention_selector.py -v → 17 passed, 10 skipped (GPU-gated), 0 failed. Includes the new test_flex_attention_rejects_small_block_sizes unit test which asserts FlexAttentionBackend.supports_block_size returns False for 1 and 8, and True for 16/32/64. No GPU required.
  • pre-commit run --files ... all hooks green (ruff, mypy, SPDX, attention-backend-docs, etc).

Test code

def test_flex_attention_rejects_small_block_sizes():
    from vllm.v1.attention.backends.flex_attention import FlexAttentionBackend

    assert not FlexAttentionBackend.supports_block_size(1)
    assert not FlexAttentionBackend.supports_block_size(8)
    assert FlexAttentionBackend.supports_block_size(16)
    assert FlexAttentionBackend.supports_block_size(32)
    assert FlexAttentionBackend.supports_block_size(64)

Changed files

  • docs/design/attention_backends.md (modified, +1/-1)
  • tests/kernels/attention/test_attention_selector.py (modified, +14/-0)
  • vllm/v1/attention/backends/flex_attention.py (modified, +12/-3)

Code Example

Your output of `python collect_env.py` here
  Collecting environment information...
  ==============================
          System Info
  ==============================
  OS                           : Ubuntu 24.04.4 LTS (x86_64)
  GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
  Clang version                : 15.0.0 (git@github.com:llvm/llvm-project.git 4ba6a9c9f65bbc8bd06e3652cb20fd4dfc846137)
  CMake version                : version 3.28.3
  Libc version                 : glibc-2.39

  ==============================
        PyTorch Info
  ==============================
  PyTorch version              : 2.10.0+cu128
  Is debug build               : False
  CUDA used to build PyTorch   : 12.8
  ROCM used to build PyTorch   : N/A

  ==============================
        Python Environment
  ==============================
  Python version               : 3.12.13 | packaged by conda-forge | (main, Mar  5 2026, 16:50:00) [GCC 14.3.0] (64-bit runtime)
  Python platform              : Linux-6.14.0-37-generic-x86_64-with-glibc2.39

  ==============================
        CUDA / GPU Info
  ==============================
  Is CUDA available            : True
  CUDA runtime version         : Could not collect
  CUDA_MODULE_LOADING set to   : 
  GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4070
  Nvidia driver version        : 575.57.08
  cuDNN version                : Probably one of the following:
  /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
  HIP runtime version          : N/A
  MIOpen runtime version       : N/A
  Is XNNPACK available         : True

  ==============================
            CPU Info
  ==============================
  Architecture:                            x86_64
  CPU op-mode(s):                          32-bit, 64-bit
  Address sizes:                           39 bits physical, 48 bits virtual
  Byte Order:                              Little Endian
  CPU(s):                                  24
  On-line CPU(s) list:                     0-23
  Vendor ID:                               GenuineIntel
  Model name:                              13th Gen Intel(R) Core(TM) i7-13700F
  CPU family:                              6
  Model:                                   183
  Thread(s) per core:                      2
  Core(s) per socket:                      16
  Socket(s):                               1
  Stepping:                                1
  CPU(s) scaling MHz:                      43%
  CPU max MHz:                             5200.0000
  CPU min MHz:                             800.0000
  BogoMIPS:                                4224.00
  Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
  Virtualization:                          VT-x
  L1d cache:                               640 KiB (16 instances)
  L1i cache:                               768 KiB (16 instances)
  L2 cache:                                24 MiB (10 instances)
  L3 cache:                                30 MiB (1 instance)
  NUMA node(s):                            1
  NUMA node0 CPU(s):                       0-23
  Vulnerability Gather data sampling:      Not affected
  Vulnerability Ghostwrite:                Not affected
  Vulnerability Indirect target selection: Not affected
  Vulnerability Itlb multihit:             Not affected
  Vulnerability L1tf:                      Not affected
  Vulnerability Mds:                       Not affected
  Vulnerability Meltdown:                  Not affected
  Vulnerability Mmio stale data:           Not affected
  Vulnerability Reg file data sampling:    Mitigation; Clear Register File
  Vulnerability Retbleed:                  Not affected
  Vulnerability Spec rstack overflow:      Not affected
  Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
  Vulnerability Srbds:                     Not affected
  Vulnerability Tsa:                       Not affected
  Vulnerability Tsx async abort:           Not affected
  Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

  ==============================
  Versions of relevant libraries
  ==============================
  [pip3] flashinfer-python==0.6.6
  [pip3] numpy==2.2.6
  [pip3] nvidia-cublas-cu12==12.8.4.1
  [pip3] nvidia-cuda-cupti-cu12==12.8.90
  [pip3] nvidia-cuda-nvrtc-cu12==12.8.93
  [pip3] nvidia-cuda-runtime-cu12==12.8.90
  [pip3] nvidia-cudnn-cu12==9.10.2.21
  [pip3] nvidia-cudnn-frontend==1.18.0
  [pip3] nvidia-cufft-cu12==11.3.3.83
  [pip3] nvidia-cufile-cu12==1.13.1.3
  [pip3] nvidia-curand-cu12==10.3.9.90
  [pip3] nvidia-cusolver-cu12==11.7.3.90
  [pip3] nvidia-cusparse-cu12==12.5.8.93
  [pip3] nvidia-cusparselt-cu12==0.7.1
  [pip3] nvidia-cutlass-dsl==4.4.2
  [pip3] nvidia-cutlass-dsl-libs-base==4.4.2
  [pip3] nvidia-ml-py==13.595.45
  [pip3] nvidia-nccl-cu12==2.27.5
  [pip3] nvidia-nvjitlink-cu12==12.8.93
  [pip3] nvidia-nvshmem-cu12==3.4.5
  [pip3] nvidia-nvtx-cu12==12.8.90
  [pip3] pyzmq==27.1.0
  [pip3] torch==2.10.0+cu128
  [pip3] torch_c_dlpack_ext==0.1.5
  [pip3] torchaudio==2.10.0+cu128
  [pip3] torchvision==0.25.0+cu128
  [pip3] transformers==4.57.6
  [pip3] triton==3.6.0
  [conda] Could not collect

  ==============================
          vLLM Info
  ==============================
  ROCM Version                 : Could not collect
  vLLM Version                 : 0.19.0
  vLLM Build Flags:
    CUDA Archs: Not Set; ROCm: Disabled
  GPU Topology:
      GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
  GPU0	 X 	0-23	0		N/A

  Legend:

    X    = Self
    SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
    NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
    PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
    PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
    PIX  = Connection traversing at most a single PCIe bridge
    NV#  = Connection traversing a bonded set of # NVLinks

  ==============================
      Environment Variables
  ==============================
  PYTORCH_NVML_BASED_CUDA_CHECK=1
  TORCHINDUCTOR_COMPILE_THREADS=1
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_fqin2

---

import vllm

llm = vllm.LLM(
    model="Qwen/Qwen3-0.6B",
    max_model_len=512,
    block_size=8,
)
llm.generate(["hello"], vllm.SamplingParams(max_tokens=1))

---

torch._inductor.exc.InductorError: CompilationError: at 146:20:
    ...
    acc, l_i, m_i = forward_inner(
                    ^
RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>
Your output of `python collect_env.py` here
  Collecting environment information...
  ==============================
          System Info
  ==============================
  OS                           : Ubuntu 24.04.4 LTS (x86_64)
  GCC version                  : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
  Clang version                : 15.0.0 ([email protected]:llvm/llvm-project.git 4ba6a9c9f65bbc8bd06e3652cb20fd4dfc846137)
  CMake version                : version 3.28.3
  Libc version                 : glibc-2.39

  ==============================
        PyTorch Info
  ==============================
  PyTorch version              : 2.10.0+cu128
  Is debug build               : False
  CUDA used to build PyTorch   : 12.8
  ROCM used to build PyTorch   : N/A

  ==============================
        Python Environment
  ==============================
  Python version               : 3.12.13 | packaged by conda-forge | (main, Mar  5 2026, 16:50:00) [GCC 14.3.0] (64-bit runtime)
  Python platform              : Linux-6.14.0-37-generic-x86_64-with-glibc2.39

  ==============================
        CUDA / GPU Info
  ==============================
  Is CUDA available            : True
  CUDA runtime version         : Could not collect
  CUDA_MODULE_LOADING set to   : 
  GPU models and configuration : GPU 0: NVIDIA GeForce RTX 4070
  Nvidia driver version        : 575.57.08
  cuDNN version                : Probably one of the following:
  /usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.7
  /usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.7
  HIP runtime version          : N/A
  MIOpen runtime version       : N/A
  Is XNNPACK available         : True

  ==============================
            CPU Info
  ==============================
  Architecture:                            x86_64
  CPU op-mode(s):                          32-bit, 64-bit
  Address sizes:                           39 bits physical, 48 bits virtual
  Byte Order:                              Little Endian
  CPU(s):                                  24
  On-line CPU(s) list:                     0-23
  Vendor ID:                               GenuineIntel
  Model name:                              13th Gen Intel(R) Core(TM) i7-13700F
  CPU family:                              6
  Model:                                   183
  Thread(s) per core:                      2
  Core(s) per socket:                      16
  Socket(s):                               1
  Stepping:                                1
  CPU(s) scaling MHz:                      43%
  CPU max MHz:                             5200.0000
  CPU min MHz:                             800.0000
  BogoMIPS:                                4224.00
  Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv1 xsaves split_lock_detect user_shstk avx_vnni dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d arch_capabilities
  Virtualization:                          VT-x
  L1d cache:                               640 KiB (16 instances)
  L1i cache:                               768 KiB (16 instances)
  L2 cache:                                24 MiB (10 instances)
  L3 cache:                                30 MiB (1 instance)
  NUMA node(s):                            1
  NUMA node0 CPU(s):                       0-23
  Vulnerability Gather data sampling:      Not affected
  Vulnerability Ghostwrite:                Not affected
  Vulnerability Indirect target selection: Not affected
  Vulnerability Itlb multihit:             Not affected
  Vulnerability L1tf:                      Not affected
  Vulnerability Mds:                       Not affected
  Vulnerability Meltdown:                  Not affected
  Vulnerability Mmio stale data:           Not affected
  Vulnerability Reg file data sampling:    Mitigation; Clear Register File
  Vulnerability Retbleed:                  Not affected
  Vulnerability Spec rstack overflow:      Not affected
  Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
  Vulnerability Srbds:                     Not affected
  Vulnerability Tsa:                       Not affected
  Vulnerability Tsx async abort:           Not affected
  Vulnerability Vmscape:                   Mitigation; IBPB before exit to userspace

  ==============================
  Versions of relevant libraries
  ==============================
  [pip3] flashinfer-python==0.6.6
  [pip3] numpy==2.2.6
  [pip3] nvidia-cublas-cu12==12.8.4.1
  [pip3] nvidia-cuda-cupti-cu12==12.8.90
  [pip3] nvidia-cuda-nvrtc-cu12==12.8.93
  [pip3] nvidia-cuda-runtime-cu12==12.8.90
  [pip3] nvidia-cudnn-cu12==9.10.2.21
  [pip3] nvidia-cudnn-frontend==1.18.0
  [pip3] nvidia-cufft-cu12==11.3.3.83
  [pip3] nvidia-cufile-cu12==1.13.1.3
  [pip3] nvidia-curand-cu12==10.3.9.90
  [pip3] nvidia-cusolver-cu12==11.7.3.90
  [pip3] nvidia-cusparse-cu12==12.5.8.93
  [pip3] nvidia-cusparselt-cu12==0.7.1
  [pip3] nvidia-cutlass-dsl==4.4.2
  [pip3] nvidia-cutlass-dsl-libs-base==4.4.2
  [pip3] nvidia-ml-py==13.595.45
  [pip3] nvidia-nccl-cu12==2.27.5
  [pip3] nvidia-nvjitlink-cu12==12.8.93
  [pip3] nvidia-nvshmem-cu12==3.4.5
  [pip3] nvidia-nvtx-cu12==12.8.90
  [pip3] pyzmq==27.1.0
  [pip3] torch==2.10.0+cu128
  [pip3] torch_c_dlpack_ext==0.1.5
  [pip3] torchaudio==2.10.0+cu128
  [pip3] torchvision==0.25.0+cu128
  [pip3] transformers==4.57.6
  [pip3] triton==3.6.0
  [conda] Could not collect

  ==============================
          vLLM Info
  ==============================
  ROCM Version                 : Could not collect
  vLLM Version                 : 0.19.0
  vLLM Build Flags:
    CUDA Archs: Not Set; ROCm: Disabled
  GPU Topology:
      GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
  GPU0	 X 	0-23	0		N/A

  Legend:

    X    = Self
    SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
    NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
    PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
    PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
    PIX  = Connection traversing at most a single PCIe bridge
    NV#  = Connection traversing a bonded set of # NVLinks

  ==============================
      Environment Variables
  ==============================
  PYTORCH_NVML_BASED_CUDA_CHECK=1
  TORCHINDUCTOR_COMPILE_THREADS=1
  TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_fqin2
</details>

🐛 Describe the bug

block_size=8 causes vLLM to select FLEX_ATTENTION (without the user requesting it). Init succeeds. The first generate() crashes. The EngineCore stderr shows torch._inductor.exc.InductorError: CompilationError deep inside the FlexAttention forward_inner Triton kernel.

In addition, FLASH_ATTN, TRITON_ATTN, FLASHINFER all reject block_size=8 cleanly with a ValueError at backend selection time. FLEX_ATTENTION silently accepts it and crashes later in Triton codegen.

Reproduce

import vllm

llm = vllm.LLM(
    model="Qwen/Qwen3-0.6B",
    max_model_len=512,
    block_size=8,
)
llm.generate(["hello"], vllm.SamplingParams(max_tokens=1))

EngineCore stderr:

torch._inductor.exc.InductorError: CompilationError: at 146:20:
    ...
    acc, l_i, m_i = forward_inner(
                    ^

Expected

LLM(...) raises ValueError at construction time when block_size is below the FlexAttention kernel's minimum. For example, add a block_size >= {minimal value} to the flexattention backend selector.

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

The issue can be fixed by adding a validation check to ensure the block_size is greater than or equal to the minimum required value for the FlexAttention kernel.

Guidance

  • Identify the minimum block_size required by the FlexAttention kernel and add a validation check in the LLM constructor to raise a ValueError if the provided block_size is less than the minimum.
  • Modify the FlexAttention backend selector to reject block_size=8 and instead raise a ValueError at construction time.
  • Update the documentation to reflect the minimum block_size requirement for the FlexAttention kernel.
  • Test the updated code with different block_size values to ensure the validation check is working correctly.

Example

class LLM:
    def __init__(self, model, max_model_len, block_size):
        # Add validation check for block_size
        min_block_size = 16  # Replace with the actual minimum block size required by FlexAttention kernel
        if block_size < min_block_size:
            raise ValueError(f"block_size must be greater than or equal to {min_block_size}")
        # ... rest of the constructor code ...

Notes

The actual minimum block_size required by the FlexAttention kernel is not specified in the issue, so it needs to be determined and replaced in the example code.

Recommendation

Apply the workaround by adding a validation check to the LLM constructor to ensure the block_size is valid for the FlexAttention kernel. This will prevent the crash and provide a clear error message to the user.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING