Motivation.

Purpose

This addresses the FlashInfer GDN prefill kernel JIT compilation hang issue affecting Qwen3.5 and Qwen3-Next models, and seeks community input on the best approach.

Problem Statement

Users loading models with GDN attention (e.g., Qwen3.5) may experience:

• Engine hangs during initialization with no clear error message
• JIT compilation timeout after 20+ minutes
• OOM due to excessive parallel compilation (50+ concurrent cicc processes consuming 150-200GB RAM)
• Library loading failures (cannot find -lcuda, GLIBCXX_3.4.32 not found)

Root Cause

The issue occurs during FlashInfer's JIT compilation of GDN prefill kernels at:

# vllm/model_executor/layers/mamba/gdn_linear_attn.py:94
from flashinfer.gdn_prefill import chunk_gated_delta_rule

This import triggers FlashInfer's JIT compilation, which may fail at three stages:

1. Compilation Phase: 30-50 kernel variants are compiled concurrently

   • Each cicc process consumes ~3-4GB RAM
   • Default parallelism causes OOM (150-200GB total)

2. Linking Phase: CUDA runtime library not found

   • Error: cannot find -lcuda
   • Cause: conda environment doesn't have libcuda.so path

3. Loading Phase: C++ standard library version mismatch

   • Error: GLIBCXX_3.4.32 not found
   • Cause: System libstdc++ is too old, conda's version not used

Proposed Change.

Two Proposed Approaches

I'm seeking community feedback on which approach is better:

Approach A: Simple Error Handling (My Preference)

Catch the import error, log a detailed error message with suggested fixes, and re-raise as a clear RuntimeError.

Implementation (see code diff):

• Wrap the import in try-except
• Detect specific error types (libcuda not found, GLIBCXX mismatch, etc.)
• Log actionable error messages with suggested fixes
• Re-raise with clear guidance

Example error message:

ERROR: FlashInfer GDN prefill kernel compilation/load failed: cannot find -lcuda

Suggested fixes:
  1. Use Triton backend (recommended): --gdn-prefill-backend triton
  2. CUDA library not found. Try:
     export LD_LIBRARY_PATH=/opt/conda/lib:$LD_LIBRARY_PATH
  3. Check diagnostic script: python test_flashinfer_jit.py

RuntimeError: FlashInfer GDN prefill kernel failed to load. 
Quick fix: use --gdn-prefill-backend triton

Advantages:

• ✅ Minimal code changes (~30 lines)
• ✅ Clear, actionable error messages
• ✅ Users can make informed decisions
• ✅ Preserves FlashInfer performance when it works
• ✅ No silent fallback (users know which backend they're using)

Disadvantages:

• ❌ Still requires manual intervention
• ❌ Doesn't prevent the long compilation attempt
• ❌ User needs to restart after fixing environment

Approach B: Comprehensive Error Handling

More detailed error handling that:

• Checks for specific failure modes before they occur
• Provides context-specific suggestions
• Potentially caches failure state to avoid repeated attempts
• May include pre-flight environment validation

Key differences from Approach A:

• More granular error detection
• Potentially faster failure (don't wait for timeout)
• More code complexity
• May require additional dependencies

I'm not providing a concrete implementation here as I'd like to gauge community interest first. If there's strong preference for this approach, I can develop a full implementation.

My Position

I prefer Approach A (Simple Error Handling) because:

1. Simplicity: ~30 lines vs potentially ~100+ lines
2. Reliability: No false positives from pre-flight checks
3. User autonomy: Lets users choose their fix
4. Maintenance: Less code to maintain
5. Graceful degradation: Compilation may still succeed even if some checks would "fail"

However, I may be missing some considerations. Would love community feedback.

Questions for the Community

Please share your thoughts on:

1. Which approach is better? Simple error handling (A) or comprehensive handling (B)?
2. Should there be an automatic fallback to Triton? (With a warning log)
3. Is automated testing worthwhile? Or is this too environment-specific?
4. Error message content? Are the suggested fixes clear and actionable?
5. Any missing failure modes? What other edge cases should we handle?
6. Should we add environment variable defaults? (e.g., set MAX_JOBS=2 by default for FlashInfer)

Test Plan

Manual Testing (Completed)

Tested on H100 with conda environment under various failure scenarios:

Scenario 1: Missing libcuda symlink

• Before: Hangs for 20+ minutes, then timeout with no helpful error
• After (Approach A): Immediate error with fix instructions

Scenario 2: GLIBCXX version mismatch

• Before: Hangs, then GLIBCXX_3.4.32 not found
• After (Approach A): Clear error message with LD_LIBRARY_PATH fix

Scenario 3: OOM during compilation

• Before: System becomes unresponsive
• After (Approach A): Still possible, but error message suggests MAX_JOBS=2

Scenario 4: Successful compilation (regression test)

• Verified that successful compilation still works normally
• No performance impact on the happy path

Diagnostic Script

I've also created test_flashinfer_jit.py to help users diagnose these issues. This script:

• Checks for required libraries
• Validates environment configuration
• Tests FlashInfer GDN kernel compilation
• Provides actionable error messages

Should this be:

• Included in this PR?
• Submitted as a separate PR?
• Kept as external tool?

Additional Context

Affected Models:

• Qwen3.5 series
• Qwen3-Next series
• Any model using GDN attention

Current Workaround (without this fix):

vllm serve Qwen/Qwen3.5-0.8B --gdn-prefill-backend triton

Environment Setup (for users encountering this issue):

# Limit compilation concurrency
export MAX_JOBS=2
export NVCC_THREADS=1

# Fix library paths
export LD_LIBRARY_PATH=/opt/conda/lib:$LD_LIBRARY_PATH

# Create CUDA library symlink (if needed)
sudo ln -s /usr/lib/x86_64-linux-gnu/libcuda.so /opt/conda/lib64/libcuda.so

Files Changed

• vllm/model_executor/layers/mamba/gdn_linear_attn.py - Error handling in fi_chunk_gated_delta_rule()
• test_flashinfer_jit.py - Diagnostic script (optional, for discussion)

Acknowledgments

I may not have a complete understanding of all the trade-offs here. Any feedback from maintainers and community members with FlashInfer experience would be greatly appreciated!

Special thanks to those who helped diagnose this issue through the various stages (compilation concurrency, library paths, GLIBCXX versions).

TL;DR: Seeking community input on error handling for FlashInfer GDN compilation failures. My preference is simple error handling with clear error messages (Approach A), but want to hear other perspectives.

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.