Issue Station
Sign In
Tools hereToolbox

vllm - ✅(Solved) Fix [RFC]: Deprecate bitsandbytes and GGUF quantization support [5 pull requests, 4 comments, 4 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
vllm-project/vllm#39583•Fetched 2026-04-12 13:24:36
View on GitHub
Comments
4
Participants
4
Timeline
22
Reactions
5
Author
mgoinmgoin
Participants
dongbo910220dongbo910220Isotr0pyIsotr0pylishunyang12lishunyang12mgoinmgoin
Timeline (top)
mentioned ×6subscribed ×6commented ×4cross-referenced ×3
bitsandbytesGGUF
Dedicated Python~1,426 lines~1,464 lines
CUDA kernels0~6,000 lines
Shared code branches~95 lines in 6 locations~75 lines in 5 locations
weight_loader_v2not supportednot supported
TP supportlimited (pre-quant doesn't work)full
CUDA graph support8-bit forces eagerfull
External depbitsandbytes pip packagegguf pip package
Model-specific hacks3 models8+ models

Both formats add ~3,100 lines of dedicated Python, ~170 lines of branching in shared weight loading code, and block migration to weight_loader_v2. GGUF additionally carries ~6,000 lines of CUDA kernels.

The primary benefit of removal isn't the line count; it's making linear.py's weight loading methods readable and refactorable again, and unblocking the weight_loader_v2 migration.

Error Message

  • Pre-quantized bnb models don't work with tensor parallelism at all (hard error at line 551-555).

Root Cause

  • GGUF: is_gguf_weight_type direct copy in weight_loader, bypassing normal shard logic
  • GGUF: tie_weights() returns embed_tokens instead of self because quantized embeddings can't share raw weight tensors

Fix Action

Fix / Workaround

FileLinesPurpose
quantization/bitsandbytes.py609Config, LinearMethod (4bit/8bit), MoEMethod
model_loader/bitsandbytes_loader.py817Full model loader with TP sharding, quant state mgmt, on-the-fly quantization
quantization/gguf.py691Config, LinearMethod, MoEMethod, EmbeddingMethod, kernel dispatch
model_loader/gguf_loader.py437Model loader, GGUF file discovery, tensor name mapping
transformers_utils/gguf_utils.py336GGUF detection, remote download, config patching
Total~2,890
  • gguf_loader.py instantiates a dummy HuggingFace model on meta device to extract parameter names for tensor mapping (lines 219-227). This is fragile and breaks when HF model classes change.
  • The loader has ~70 lines of hardcoded model-type name remapping (deepseek_v2/v3, qwen2/3_moe, minimax_m2, cohere, gemma3) that must be updated for each new MoE architecture.
  • transformers_utils/gguf_utils.py adds config patching (maybe_patch_hf_config_from_gguf) and tokenizer extraction from the GGUF container.
  • ~8 model files (llama, llama4, gemma3, exaone, etc.) have GGUF-specific RoPE style detection branches.

Remove 2 of ~6 loader classes. The dispatch logic in model_loader/__init__.py gets simpler.

PR fix notes

PR #1528: [Feature]: Bitsandbytes Quantization Support for Diffusion Pipelines

Description (problem / solution / changelog)

Purpose

This PR implements Bitsandbytes (BNB) 4-bit quantization support for Diffusion Pipelines in vllm-omni, as proposed in RFC #1527.

By integrating BNB quantization, we significantly reduce the peak VRAM requirements for diffusion model inference, enabling high-quality image generation on GPUs with limited memory (e.g., consumer-grade cards) and allowing for larger batch sizes in production environments.

Co-author: @Michael-Zzq

Users can enable quantization by specifying the backend:

vllm serve Tongyi-MAI/Z-Image-Turbo --omni --quantization bitsandbytes

Test Plan

pytest tests/diffusion/test_bitsandbytes_quantization.py`

Test Result

passed

Qualitative Comparison (VRAM Profile)

Baseline OutputBNB 4bit Output
<img width="1024" height="1024" alt="baseline_vram" src="https://github.com/user-attachments/assets/b048018b-a551-44c7-a89f-19f1bc7c266e" /><img width="1024" height="1024" alt="bnb4bit_vram" src="https://github.com/user-attachments/assets/095e393b-27e3-4f64-83c9-01b1f179ed15" />
Peak: ~24.5 GiBPeak: ~17.1 GiB
<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan. Please providing the test scripts & test commands. Please state the reasons if your codes don't require additional test scripts. For test file guidelines, please check the test style doc
  • The test results. Please pasting the results comparison before and after, or e2e results.
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model. Please run mkdocs serve to sync the documentation editions to ./docs.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft.
</details>

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Changed files

  • docs/user_guide/diffusion/quantization/fp8.md (modified, +2/-2)
  • pyproject.toml (modified, +4/-0)
  • tests/diffusion/test_bitsandbytes_quantization.py (added, +275/-0)
  • tests/diffusion/test_offload_bnb_interaction.py (added, +37/-0)
  • tests/entrypoints/test_omni_stage_diffusion_config.py (modified, +4/-0)
  • vllm_omni/diffusion/data.py (modified, +35/-2)
  • vllm_omni/diffusion/model_loader/diffusers_loader.py (modified, +77/-11)
  • vllm_omni/diffusion/models/z_image/pipeline_z_image.py (modified, +6/-0)
  • vllm_omni/diffusion/offloader/layerwise_backend.py (modified, +19/-3)
  • vllm_omni/diffusion/offloader/sequential_backend.py (modified, +24/-2)
  • vllm_omni/diffusion/quantization/__init__.py (modified, +57/-4)
  • vllm_omni/diffusion/quantization/bitsandbytes.py (added, +811/-0)
  • vllm_omni/diffusion/worker/diffusion_model_runner.py (modified, +33/-0)
  • vllm_omni/entrypoints/async_omni.py (modified, +2/-0)
  • vllm_omni/entrypoints/cli/serve.py (modified, +20/-0)
  • vllm_omni/entrypoints/omni.py (modified, +33/-0)

PR #39612: [DRAFT] Remove GGUF quantization

Description (problem / solution / changelog)

Purpose

Test Plan

Test Result

<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.
</details>

Changed files

  • .github/dependabot.yml (modified, +0/-1)
  • .pre-commit-config.yaml (modified, +1/-1)
  • CMakeLists.txt (modified, +0/-1)
  • csrc/ops.h (modified, +1/-23)
  • csrc/quantization/gguf/dequantize.cuh (removed, +0/-571)
  • csrc/quantization/gguf/ggml-common.h (removed, +0/-1150)
  • csrc/quantization/gguf/gguf_kernel.cu (removed, +0/-542)
  • csrc/quantization/gguf/mmq.cuh (removed, +0/-610)
  • csrc/quantization/gguf/mmvq.cuh (removed, +0/-212)
  • csrc/quantization/gguf/moe.cuh (removed, +0/-739)
  • csrc/quantization/gguf/moe_vec.cuh (removed, +0/-338)
  • csrc/quantization/gguf/vecdotq.cuh (removed, +0/-1812)
  • csrc/torch_bindings.cpp (modified, +0/-33)
  • docs/features/quantization/README.md (modified, +0/-2)
  • docs/features/quantization/gguf.md (removed, +0/-87)
  • docs/mkdocs/hooks/generate_examples.py (modified, +0/-1)
  • requirements/common.txt (modified, +0/-1)
  • requirements/test/rocm.txt (modified, +0/-8)
  • tests/compile/fullgraph/test_full_graph.py (modified, +0/-6)
  • tests/kernels/quantization/test_ggml.py (removed, +0/-54)
  • tests/kernels/quantization/test_gguf.py (removed, +0/-207)
  • tests/models/multimodal/generation/test_multimodal_gguf.py (removed, +0/-180)
  • tests/models/quantization/test_gguf.py (removed, +0/-204)
  • tests/models/test_gguf_download.py (removed, +0/-221)
  • tests/transformers_utils/test_utils.py (modified, +0/-210)
  • vllm/_custom_ops.py (modified, +0/-128)
  • vllm/config/load.py (modified, +0/-2)
  • vllm/config/model.py (modified, +1/-24)
  • vllm/engine/arg_utils.py (modified, +2/-4)
  • vllm/model_executor/layers/fused_moe/layer.py (modified, +8/-5)
  • vllm/model_executor/layers/linear.py (modified, +1/-96)
  • vllm/model_executor/layers/quantization/__init__.py (modified, +0/-3)
  • vllm/model_executor/layers/quantization/base_config.py (modified, +26/-0)
  • vllm/model_executor/layers/quantization/gguf.py (removed, +0/-691)
  • vllm/model_executor/layers/vocab_parallel_embedding.py (modified, +3/-18)
  • vllm/model_executor/model_loader/__init__.py (modified, +0/-4)
  • vllm/model_executor/model_loader/gguf_loader.py (removed, +0/-436)
  • vllm/model_executor/model_loader/weight_utils.py (modified, +2/-166)
  • vllm/model_executor/models/apertus.py (modified, +4/-3)
  • vllm/model_executor/models/exaone.py (modified, +4/-2)
  • vllm/model_executor/models/exaone4.py (modified, +4/-2)
  • vllm/model_executor/models/gemma3.py (modified, +4/-8)
  • vllm/model_executor/models/jais2.py (modified, +4/-2)
  • vllm/model_executor/models/llama.py (modified, +4/-3)
  • vllm/model_executor/models/llama4.py (modified, +4/-3)
  • vllm/model_executor/models/openpangu.py (modified, +12/-9)
  • vllm/model_executor/models/siglip.py (modified, +2/-13)
  • vllm/platforms/rocm.py (modified, +0/-1)
  • vllm/tokenizers/registry.py (modified, +0/-22)
  • vllm/transformers_utils/config.py (modified, +10/-97)
  • vllm/transformers_utils/gguf_utils.py (removed, +0/-336)
  • vllm/transformers_utils/processor.py (modified, +4/-26)
  • vllm/v1/metrics/perf.py (modified, +0/-1)

PR #39559: [Model] Add GGUF support for Qwen 3.5 dense and MoE models

Description (problem / solution / changelog)

Purpose

Add GGUF support for Qwen 3.5 dense and MoE models

Fixes: #39198, #36456, #38122

Test Plan

# Qwen 3.5 Dense
vllm serve unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS --tokenizer Qwen/Qwen3.5-0.8B --hf-config-path Qwen/Qwen3.5-0.8B
# Qwen 3.5 MoE
vllm serve unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS --tokenizer Qwen/Qwen3.5-35B-A3B-GGUF

Test Result

Qwen3.5 Dense

<details> <summary>before</summary> 
vllm serve unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS --tokenizer Qwen/Qwen3.5-0.8B --hf-config-path Qwen/Qwen3.5-0.8B
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:299]
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.1rc1.dev122+g83aea2147
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:299]   █▄█▀ █     █     █     █  model   unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:299]
(APIServer pid=2639330) INFO 04-11 18:49:05 [utils.py:233] non-default args: {'model_tag': 'unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS', 'model': 'unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS', 'tokenizer': 'Qwen/Qwen3.5-0.8B', 'hf_config_path': 'Qwen/Qwen3.5-0.8B'}
(APIServer pid=2639330) WARNING 04-11 18:49:05 [gguf_utils.py:60] Non-standard GGUF quant type 'UD-IQ2_XXS' detected.
(APIServer pid=2639330) INFO 04-11 18:49:07 [model.py:554] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=2639330) INFO 04-11 18:49:07 [model.py:1684] Using max model len 262144
(APIServer pid=2639330) INFO 04-11 18:49:07 [vllm.py:799] Asynchronous scheduling is enabled.
(APIServer pid=2639330) INFO 04-11 18:49:07 [kernel.py:199] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(EngineCore pid=2639990) INFO 04-11 18:49:23 [core.py:107] Initializing a V1 LLM engine (v0.19.1rc1.dev122+g83aea2147) with config: model='unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS', speculative_config=None, tokenizer='Qwen/Qwen3.5-0.8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=gguf, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gguf, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=2639990) INFO 04-11 18:49:25 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.16.1.10:59959 backend=nccl
(EngineCore pid=2639990) INFO 04-11 18:49:25 [parallel_state.py:1713] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=2639990) WARNING 04-11 18:49:25 [gguf_utils.py:60] Non-standard GGUF quant type 'UD-IQ2_XXS' detected.
(EngineCore pid=2639990) INFO 04-11 18:49:34 [gpu_model_runner.py:4735] Starting to load model unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS...
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112] EngineCore failed to start.
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112] Traceback (most recent call last):
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 1086, in run_engine_core
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     return func(*args, **kwargs)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 850, in __init__
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     super().__init__(
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 116, in __init__
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     self.model_executor = executor_class(vllm_config)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     return func(*args, **kwargs)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     self._init_executor()
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/executor/uniproc_executor.py", line 52, in _init_executor
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     self.driver_worker.load_model()
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/worker/gpu_worker.py", line 323, in load_model
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     return func(*args, **kwargs)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/v1/worker/gpu_model_runner.py", line 4751, in load_model
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     self.model = model_loader.load_model(
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/model_executor/model_loader/gguf_loader.py", line 406, in load_model
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     gguf_weights_map = self._get_gguf_weights_map(model_config)
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]   File "/home/name/.test/.gpu/vllm/vllm/model_executor/model_loader/gguf_loader.py", line 204, in _get_gguf_weights_map
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112]     raise RuntimeError(f"Unknown gguf model_type: {model_type}")
(EngineCore pid=2639990) ERROR 04-11 18:49:35 [core.py:1112] RuntimeError: Unknown gguf model_type: qwen3_5
(EngineCore pid=2639990) Process EngineCore:
(EngineCore pid=2639990) Traceback (most recent call last):
(EngineCore pid=2639990)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=2639990)     self.run()
(EngineCore pid=2639990)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=2639990)     self._target(*self._args, **self._kwargs)
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 1116, in run_engine_core
(EngineCore pid=2639990)     raise e
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 1086, in run_engine_core
(EngineCore pid=2639990)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=2639990)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2639990)     return func(*args, **kwargs)
(EngineCore pid=2639990)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 850, in __init__
(EngineCore pid=2639990)     super().__init__(
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core.py", line 116, in __init__
(EngineCore pid=2639990)     self.model_executor = executor_class(vllm_config)
(EngineCore pid=2639990)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2639990)     return func(*args, **kwargs)
(EngineCore pid=2639990)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=2639990)     self._init_executor()
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/executor/uniproc_executor.py", line 52, in _init_executor
(EngineCore pid=2639990)     self.driver_worker.load_model()
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/worker/gpu_worker.py", line 323, in load_model
(EngineCore pid=2639990)     self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2639990)     return func(*args, **kwargs)
(EngineCore pid=2639990)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/v1/worker/gpu_model_runner.py", line 4751, in load_model
(EngineCore pid=2639990)     self.model = model_loader.load_model(
(EngineCore pid=2639990)                  ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/model_executor/model_loader/gguf_loader.py", line 406, in load_model
(EngineCore pid=2639990)     gguf_weights_map = self._get_gguf_weights_map(model_config)
(EngineCore pid=2639990)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2639990)   File "/home/name/.test/.gpu/vllm/vllm/model_executor/model_loader/gguf_loader.py", line 204, in _get_gguf_weights_map
(EngineCore pid=2639990)     raise RuntimeError(f"Unknown gguf model_type: {model_type}")
(EngineCore pid=2639990) RuntimeError: Unknown gguf model_type: qwen3_5
[rank0]:[W411 18:49:36.785516042 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=2639330) Traceback (most recent call last):
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/.venv/bin/vllm", line 10, in <module>
(APIServer pid=2639330)     sys.exit(main())
(APIServer pid=2639330)              ^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/entrypoints/cli/main.py", line 75, in main
(APIServer pid=2639330)     args.dispatch_function(args)
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=2639330)     uvloop.run(run_server(args))
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=2639330)     return __asyncio.run(
(APIServer pid=2639330)            ^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=2639330)     return runner.run(main)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=2639330)     return self._loop.run_until_complete(task)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=2639330)     return await main
(APIServer pid=2639330)            ^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/entrypoints/openai/api_server.py", line 686, in run_server
(APIServer pid=2639330)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/entrypoints/openai/api_server.py", line 700, in run_server_worker
(APIServer pid=2639330)     async with build_async_engine_client(
(APIServer pid=2639330)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=2639330)     return await anext(self.gen)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=2639330)     async with build_async_engine_client_from_engine_args(
(APIServer pid=2639330)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=2639330)     return await anext(self.gen)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=2639330)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=2639330)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=2639330)     return cls(
(APIServer pid=2639330)            ^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=2639330)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=2639330)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=2639330)     return func(*args, **kwargs)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=2639330)     return AsyncMPClient(*client_args)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=2639330)     return func(*args, **kwargs)
(APIServer pid=2639330)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core_client.py", line 890, in __init__
(APIServer pid=2639330)     super().__init__(
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/core_client.py", line 551, in __init__
(APIServer pid=2639330)     with launch_core_engines(
(APIServer pid=2639330)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=2639330)   File "/home/name/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=2639330)     next(self.gen)
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/utils.py", line 1094, in launch_core_engines
(APIServer pid=2639330)     wait_for_engine_startup(
(APIServer pid=2639330)   File "/home/name/.test/.gpu/vllm/vllm/v1/engine/utils.py", line 1153, in wait_for_engine_startup
(APIServer pid=2639330)     raise RuntimeError(
(APIServer pid=2639330) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
</details> <details> <summary>after</summary> 
vllm serve unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS --tokenizer Qwen/Qwen3.5-0.8B --hf-config-path Qwen/Qwen3.5-0.8B
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:299] 
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.1rc1.dev164+g55d037e2e.d20260410
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:299]   █▄█▀ █     █     █     █  model   unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:299] 
(APIServer pid=1622311) INFO 04-13 22:08:32 [utils.py:233] non-default args: {'model_tag': 'unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS', 'model': 'unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS', 'tokenizer': 'Qwen/Qwen3.5-0.8B', 'hf_config_path': 'Qwen/Qwen3.5-0.8B'}
(APIServer pid=1622311) WARNING 04-13 22:08:32 [gguf_utils.py:62] Non-standard GGUF quant type 'UD-IQ2_XXS' detected.
(APIServer pid=1622311) INFO 04-13 22:08:34 [model.py:554] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1622311) INFO 04-13 22:08:34 [model.py:1684] Using max model len 262144
(APIServer pid=1622311) INFO 04-13 22:08:34 [vllm.py:809] Asynchronous scheduling is enabled.
(APIServer pid=1622311) INFO 04-13 22:08:34 [kernel.py:199] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=1622311) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=1622311) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1623438) INFO 04-13 22:08:55 [core.py:107] Initializing a V1 LLM engine (v0.19.1rc1.dev164+g55d037e2e.d20260410) with config: model='unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS', speculative_config=None, tokenizer='Qwen/Qwen3.5-0.8B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=gguf, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gguf, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=1623438) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=1623438) INFO 04-13 22:08:58 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.16.1.10:39833 backend=nccl
(EngineCore pid=1623438) INFO 04-13 22:08:58 [parallel_state.py:1713] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=1623438) WARNING 04-13 22:08:58 [gguf_utils.py:62] Non-standard GGUF quant type 'UD-IQ2_XXS' detected.
(EngineCore pid=1623438) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1623438) INFO 04-13 22:09:11 [gpu_model_runner.py:4750] Starting to load model unsloth/Qwen3.5-0.8B-GGUF:UD-IQ2_XXS...
(EngineCore pid=1623438) The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
(EngineCore pid=1623438) INFO 04-13 22:09:24 [gguf_loader.py:443] Loading extra mm_proj weights from /home/name/.cache/huggingface/hub/models--unsloth--Qwen3.5-0.8B-GGUF/snapshots/6ab461498e2023f6e3c1baea90a8f0fe38ab64d0/mmproj-BF16.gguf...
(EngineCore pid=1623438) INFO 04-13 22:09:24 [cuda.py:422] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=1623438) INFO 04-13 22:09:24 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=1623438) INFO 04-13 22:09:24 [gdn_linear_attn.py:155] Using Triton/FLA GDN prefill kernel
(EngineCore pid=1623438) INFO 04-13 22:09:25 [cuda.py:366] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=1623438) INFO 04-13 22:09:25 [flash_attn.py:637] Using FlashAttention version 2
(EngineCore pid=1623438) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore pid=1623438) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=1623438) INFO 04-13 22:09:32 [gpu_model_runner.py:4835] Model loading took 0.95 GiB memory and 20.688811 seconds
(EngineCore pid=1623438) INFO 04-13 22:09:32 [interface.py:606] Setting attention block size to 544 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=1623438) INFO 04-13 22:09:32 [interface.py:630] Padding mamba page size by 2.64% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=1623438) INFO 04-13 22:09:32 [gpu_model_runner.py:5784] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=1623438) INFO 04-13 22:09:33 [backends.py:1070] Using cache directory: /home/name/.cache/vllm/torch_compile_cache/3ff83d0cde/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=1623438) INFO 04-13 22:09:33 [backends.py:1130] Dynamo bytecode transform time: 0.60 s
(EngineCore pid=1623438) INFO 04-13 22:09:34 [backends.py:286] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 0.904 s
(EngineCore pid=1623438) INFO 04-13 22:09:34 [decorators.py:305] Directly load AOT compilation from path /home/name/.cache/vllm/torch_compile_cache/torch_aot_compile/910686eaa28fa02dabfb763dc29f80d7cc4efce33d0e6008f20f0e94258b227f/rank_0_0/model
(EngineCore pid=1623438) INFO 04-13 22:09:34 [monitor.py:48] torch.compile took 1.61 s in total
(EngineCore pid=1623438) INFO 04-13 22:09:35 [monitor.py:76] Initial profiling/warmup run took 0.11 s
(EngineCore pid=1623438) INFO 04-13 22:09:35 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(EngineCore pid=1623438) INFO 04-13 22:09:35 [gpu_model_runner.py:5914] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=35 (largest=256)
(EngineCore pid=1623438) INFO 04-13 22:09:36 [gpu_model_runner.py:5993] Estimated CUDA graph memory: 0.73 GiB total
(EngineCore pid=1623438) INFO 04-13 22:09:36 [gpu_worker.py:436] Available KV cache memory: 18.76 GiB
(EngineCore pid=1623438) INFO 04-13 22:09:36 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9312 to maintain the same effective KV cache size.
(EngineCore pid=1623438) INFO 04-13 22:09:36 [kv_cache_utils.py:1319] GPU KV cache size: 409,632 tokens
(EngineCore pid=1623438) INFO 04-13 22:09:36 [kv_cache_utils.py:1324] Maximum concurrency for 262,144 tokens per request: 6.21x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|███████████████████████| 51/51 [00:01<00:00, 47.86it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████████████████████████████████████| 35/35 [00:00<00:00, 49.09it/s]
(EngineCore pid=1623438) INFO 04-13 22:09:38 [gpu_model_runner.py:6084] Graph capturing finished in 2 secs, took 0.72 GiB
(EngineCore pid=1623438) INFO 04-13 22:09:38 [gpu_worker.py:597] CUDA graph pool memory: 0.72 GiB (actual), 0.73 GiB (estimated), difference: 0.01 GiB (1.3%).
(EngineCore pid=1623438) INFO 04-13 22:09:38 [core.py:285] init engine (profile, create kv cache, warmup model) took 6.29 seconds
(EngineCore pid=1623438) INFO 04-13 22:09:38 [vllm.py:809] Asynchronous scheduling is enabled.
(EngineCore pid=1623438) INFO 04-13 22:09:38 [kernel.py:199] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=1622311) INFO 04-13 22:09:38 [api_server.py:600] Supported tasks: ['generate']
(APIServer pid=1622311) INFO 04-13 22:09:47 [hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1622311) INFO 04-13 22:09:56 [base.py:245] Multi-modal warmup completed in 8.255s
(APIServer pid=1622311) INFO 04-13 22:09:57 [api_server.py:604] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:37] Available routes are:
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1622311) INFO 04-13 22:09:57 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=1622311) INFO:     Started server process [1622311]
(APIServer pid=1622311) INFO:     Waiting for application startup.
(APIServer pid=1622311) INFO:     Application startup complete.
(APIServer pid=1622311) INFO:     127.0.0.1:40480 - "POST /v1/chat/completions HTTP/1.1" 200 OK
</details>

Qwen3.5 MoE

<details> <summary>after</summary> 
vllm serve unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS --tokenizer Qwen/Qwen3.5-35B-A3B
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:299]
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:299]        █     █     █▄   ▄█
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.19.1rc1.dev164+g55d037e2e.d20260410
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:299]   █▄█▀ █     █     █     █  model   unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:299]
(APIServer pid=1258756) INFO 04-13 19:49:54 [utils.py:233] non-default args: {'model_tag': 'unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS', 'model': 'unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS', 'tokenizer': 'Qwen/Qwen3.5-35B-A3B'}
(APIServer pid=1258756) WARNING 04-13 19:49:54 [gguf_utils.py:62] Non-standard GGUF quant type 'UD-IQ2_XXS' detected.
(APIServer pid=1258756) INFO 04-13 19:49:56 [gguf_utils.py:334] Forced Qwen3.5 multimodal architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1258756) INFO 04-13 19:49:56 [model.py:554] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1258756) INFO 04-13 19:49:56 [model.py:1684] Using max model len 262144
(APIServer pid=1258756) INFO 04-13 19:49:56 [vllm.py:809] Asynchronous scheduling is enabled.
(APIServer pid=1258756) INFO 04-13 19:49:56 [kernel.py:199] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=1258756) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=1258756) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1259857) INFO 04-13 19:50:17 [core.py:107] Initializing a V1 LLM engine (v0.19.1rc1.dev164+g55d037e2e.d20260410) with config: model='unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS', speculative_config=None, tokenizer='Qwen/Qwen3.5-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=gguf, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=gguf, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=1259857) `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=1259857) INFO 04-13 19:50:19 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.16.1.10:51161 backend=nccl
(EngineCore pid=1259857) INFO 04-13 19:50:20 [parallel_state.py:1713] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=1259857) WARNING 04-13 19:50:20 [gguf_utils.py:62] Non-standard GGUF quant type 'UD-IQ2_XXS' detected.
(EngineCore pid=1259857) The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=1259857) INFO 04-13 19:50:32 [gpu_model_runner.py:4750] Starting to load model unsloth/Qwen3.5-35B-A3B-GGUF:UD-IQ2_XXS...
(EngineCore pid=1259857) The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
(EngineCore pid=1259857) INFO 04-13 19:50:47 [gguf_loader.py:456] Loading extra mm_proj weights from /home/name/.cache/huggingface/hub/models--unsloth--Qwen3.5-35B-A3B-GGUF/snapshots/bc014a17be43adabd7066b7a86075ff935c6a4e2/mmproj-BF16.gguf...
(EngineCore pid=1259857) INFO 04-13 19:50:47 [cuda.py:422] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=1259857) INFO 04-13 19:50:47 [mm_encoder_attention.py:230] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=1259857) INFO 04-13 19:50:47 [gdn_linear_attn.py:155] Using Triton/FLA GDN prefill kernel
(EngineCore pid=1259857) INFO 04-13 19:50:47 [cuda.py:366] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=1259857) INFO 04-13 19:50:47 [flash_attn.py:637] Using FlashAttention version 2
(EngineCore pid=1259857) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(EngineCore pid=1259857) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(EngineCore pid=1259857) INFO 04-13 19:50:57 [gpu_model_runner.py:4835] Model loading took 12.21 GiB memory and 23.596208 seconds
(EngineCore pid=1259857) INFO 04-13 19:50:57 [interface.py:606] Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=1259857) INFO 04-13 19:50:57 [interface.py:630] Padding mamba page size by 0.76% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=1259857) INFO 04-13 19:50:57 [gpu_model_runner.py:5784] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=1259857) INFO 04-13 19:51:01 [backends.py:1070] Using cache directory: /home/name/.cache/vllm/torch_compile_cache/2e45e00b32/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=1259857) INFO 04-13 19:51:01 [backends.py:1130] Dynamo bytecode transform time: 3.86 s
(EngineCore pid=1259857) INFO 04-13 19:51:03 [backends.py:373] Cache the graph of compile range (1, 2048) for later use
(EngineCore pid=1259857) INFO 04-13 19:51:15 [backends.py:391] Compiling a graph for compile range (1, 2048) takes 13.76 s
(EngineCore pid=1259857) INFO 04-13 19:51:18 [decorators.py:655] saved AOT compiled function to /home/name/.cache/vllm/torch_compile_cache/torch_aot_compile/703a7fd7463c0d2c8352e4b42e821c7d228ba254269fa8c7310bbeda6ae7ffa0/rank_0_0/model
(EngineCore pid=1259857) INFO 04-13 19:51:18 [monitor.py:48] torch.compile took 20.01 s in total
(EngineCore pid=1259857) INFO 04-13 19:51:20 [monitor.py:76] Initial profiling/warmup run took 2.18 s
(EngineCore pid=1259857) INFO 04-13 19:51:20 [kv_cache_utils.py:829] Overriding num_gpu_blocks=0 with num_gpu_blocks_override=512
(EngineCore pid=1259857) INFO 04-13 19:51:20 [gpu_model_runner.py:5914] Profiling CUDA graph memory: PIECEWISE=51 (largest=512), FULL=35 (largest=256)
(EngineCore pid=1259857) INFO 04-13 19:51:23 [gpu_model_runner.py:5993] Estimated CUDA graph memory: 1.62 GiB total
(EngineCore pid=1259857) INFO 04-13 19:51:24 [gpu_worker.py:436] Available KV cache memory: 6.9 GiB
(EngineCore pid=1259857) INFO 04-13 19:51:24 [gpu_worker.py:470] In v0.19, CUDA graph memory profiling will be enabled by default (VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1), which more accurately accounts for CUDA graph memory during KV cache allocation. To try it now, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 and increase --gpu-memory-utilization from 0.9000 to 0.9690 to maintain the same effective KV cache size.
(EngineCore pid=1259857) INFO 04-13 19:51:24 [kv_cache_utils.py:1319] GPU KV cache size: 89,760 tokens
(EngineCore pid=1259857) INFO 04-13 19:51:24 [kv_cache_utils.py:1324] Maximum concurrency for 262,144 tokens per request: 1.36x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|████████████████████████████████████████| 51/51 [00:10<00:00,  4.86it/s]
Capturing CUDA graphs (decode, FULL): 100%|███████████████████████████████████████████████████████████| 35/35 [00:04<00:00,  7.02it/s]
(EngineCore pid=1259857) INFO 04-13 19:51:40 [gpu_model_runner.py:6084] Graph capturing finished in 16 secs, took 1.71 GiB
(EngineCore pid=1259857) INFO 04-13 19:51:40 [gpu_worker.py:597] CUDA graph pool memory: 1.71 GiB (actual), 1.62 GiB (estimated), difference: 0.08 GiB (4.8%).
(EngineCore pid=1259857) INFO 04-13 19:51:40 [core.py:285] init engine (profile, create kv cache, warmup model) took 43.22 seconds
(EngineCore pid=1259857) INFO 04-13 19:51:40 [vllm.py:809] Asynchronous scheduling is enabled.
(EngineCore pid=1259857) INFO 04-13 19:51:40 [kernel.py:199] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=1258756) INFO 04-13 19:51:40 [api_server.py:600] Supported tasks: ['generate']
(APIServer pid=1258756) INFO 04-13 19:51:47 [hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1258756) INFO 04-13 19:51:55 [base.py:245] Multi-modal warmup completed in 8.099s
(APIServer pid=1258756) INFO 04-13 19:51:55 [api_server.py:604] Starting vLLM server on http://0.0.0.0:8000/
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:37] Available routes are:
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=1258756) INFO 04-13 19:51:55 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=1258756) INFO:     Started server process [1258756]
(APIServer pid=1258756) INFO:     Waiting for application startup.
(APIServer pid=1258756) INFO:     Application startup complete.
(APIServer pid=1258756) INFO:     127.0.0.1:33702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
</details>
<details> <summary> Essential Elements of an Effective PR Description Checklist </summary>
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.
</details>

Changed files

  • tests/models/multimodal/generation/test_multimodal_gguf.py (modified, +73/-2)
  • vllm/model_executor/layers/linear.py (modified, +26/-6)
  • vllm/model_executor/model_loader/gguf_loader.py (modified, +233/-12)
  • vllm/transformers_utils/gguf_utils.py (modified, +25/-0)

PR #1920: [ROCm] Optimize kgemm_4bit_inference_naive for ROCm, use it for batch sizes other than 1

Description (problem / solution / changelog)

Based on issues raised in #1842 and pytorch#171687.

Summary

  • Optimize kgemm_4bit_inference_naive on ROCm, following the suggestions discussed in #1842.
  • On gfx1201 this improves the kernel microbenchmark by up to 2.39x and end-to-end inference by up to 1.98x. On gfx1151 vLLM serving throughput improves by up to 5.13x at Req = 2.
  • The kernel optimizations don't improve or regress the performance on Nvidia GPUs. Nvidia however still benefits from the vLLM serving optimizations.
  • Extend the fused 4-bit inference path to support small multi-row (M > 1) inputs instead of only the vector case.
  • Update the Python and backend dispatch path so compatible small-batch inference uses the fused kernel automatically, improving decode throughput for multi-request vLLM serving with 4-bit quantization.

Technical details

This PR makes two related changes.

Kernel optimization

  • Rework the hot loop in kgemm_4bit_inference_naive to reduce overhead in the fused dequantize + matmul path on ROCm.
  • Improve the packed-weight load and dequant flow so the kernel uses memory bandwidth more effectively on AMD GPUs.
  • Specific details about the kernel optimizations can be found in #1842

Fused path support for M > 1

  • Remove the vector-only restriction in the 4-bit fused inference path.
  • Pass the real number of input rows through the Python/backend interface.
  • Update the ROCm launch configuration so the fused kernel is used for small multi-row inputs, not just M == 1.

Up to a platform-specific crossover point, launching the fused kernel is substantially faster than falling back to split dequantize + GEMM. This matters most for serving workloads, where decode steps regularly hit small M > 1 batches.

Example measured on Strix Halo:

MSplitFusedSpeedup
1741 us741 us1.00x
25512 us998 us5.52x
45498 us1630 us3.37x
85514 us3001 us1.84x

At larger M, the fused path eventually converges with and then regresses against split dequantize + GEMM. The crossover differs by GPU:

GPUCrossover M
gfx115116
RTX 50908-12
gfx120110-12
MI308X4-6

For this PR, the dispatch threshold is set to M=8 as a cross-GPU compromise. That still leaves some regressions on MI308X once reqs >= 6, but avoids the larger regressions seen at higher thresholds on other GPUs.

Testing plan

  • Run the gemm_4bit unit tests to validate correctness of the updated kernel path.
  • Run end-to-end Transformers inference in 4-bit format to validate single-user decode behavior.
  • Run end-to-end vLLM serving with multiple concurrent requests to verify the M > 1 fused-path performance gain.

Testing results

gemv_4bit unit-tests

  • All tests pass on all of the tested configurations.

kgemm_4bit_inference_naive benchmark

In this table, A denotes the baseline kernel and B denotes the optimized kernel.

GPUTime ATime BBW ABW BPeak BW Reference% Peak A% Peak BSpeedup (B vs A)
gfx11511133 us740 us117 GB/s178 GB/s~210 GB/s (measured)56%85%1.53x
RTX 509086 us84 us1361 GB/s1394 GB/s~1,790 GB/s76%78%1.02x
gfx1201539 us226 us218 GB/s519 GB/s640 GB/s34%81%2.39x
MI308X656 us246 us179 GB/s477 GB/s~3,277 GB/s5.5%14.6%2.67x

End-to-end Transformers Throughput

Strix Halo (gfx1151):

ModelA (tok/s)B (tok/s)Speedup
Llama-3.3-70B-Instruct2.453.861.58x
Mistral-7B-Instruct-v0.318.327.91.53x
Phi-4 (14B)10.715.61.46x
DeepSeek-R1-Distill-Qwen-14B9.612.51.30x
DeepSeek-R1-Distill-Qwen-7B17.422.31.28x

RTX 5090:

ModelPhase A (tok/s)Phase B (tok/s)Speedup
Mistral-7B85.5784.320.99x
Llama-8B82.4380.760.98x
Qwen3.5-9B59.5158.950.99x

Radeon AI Pro R9700 (gfx1201):

ModelPhase A (tok/s)Phase B (tok/s)Speedup
Mistral-7B38.4264.321.67x
Llama-8B31.3146.271.48x
Qwen3.5-9B23.8332.001.34x

MI308X (gfx942):

ModelPhase A (tok/s)Phase B (tok/s)Speedup
Mistral-7B31.2840.511.30x
Llama-3.1-8B30.9740.461.31x
Qwen3.5-9B23.4529.031.24x
Qwen3.5-35B-A3B (MoE)15.6215.941.02x
Llama-3.3-70B-Instruct4.5910.602.31x
Llama-3.2-90B-Vision (text-only)4.5910.612.31x

End-to-end vLLM Serving Throughput for Reqs > 1

Strix Halo (gfx1151)

Mistral-7B
ReqsBaselineL=1L=8L=16L=16 vs Baseline
122.5 fused34.7 fused35.3 fused35.7 fused1.59x
210.4 split10.4 split51.8 fused53.4 fused5.13x
420.3 split20.3 split67.6 fused67.3 fused3.32x
630.4 split30.5 split72.4 fused71.9 fused2.37x
840.4 split40.4 split75.2 fused75.1 fused1.86x
1050.5 split50.6 split50.6 split76.7 fused1.52x
1260.2 split60.4 split60.4 split77.9 fused1.29x
1469.9 split70.1 split70.1 split78.7 fused1.13x
1680.1 split80.2 split80.3 split79.2 fused0.99x
Llama-8B
ReqsBaselineL=1L=8L=16L=16 vs Baseline
120.7 fused32.0 fused32.0 fused32.0 fused1.55x
210.4 split10.4 split48.9 fused47.3 fused4.55x
420.3 split20.3 split63.7 fused63.6 fused3.13x
630.3 split30.2 split68.8 fused68.5 fused2.26x
840.2 split40.1 split72.4 fused72.4 fused1.80x
1050.2 split50.2 split50.2 split74.6 fused1.49x
1260.0 split60.0 split59.9 split75.8 fused1.26x
1469.5 split69.6 split69.6 split76.9 fused1.11x
1679.5 split79.5 split79.5 split77.2 fused0.97x
Qwen3.5-9B
ReqsBaselineL=1L=8L=16L=16 vs Baseline
117.5 fused23.1 fused22.9 fused22.9 fused1.31x
29.4 split9.4 split40.4 fused39.5 fused4.20x
418.4 split18.4 split55.8 fused57.0 fused3.10x
626.9 split27.0 split61.8 fused62.0 fused2.30x
835.4 split35.5 split65.9 fused66.0 fused1.86x
1044.5 split44.6 split44.7 split68.7 fused1.54x
1252.0 split52.3 split52.3 split70.6 fused1.36x
1460.1 split60.3 split60.4 split72.1 fused1.20x
1669.0 split69.3 split69.5 split72.9 fused1.06x
Llama-3.3-70B
ReqsBaselineL=1L=8L=16L=16 vs Baseline
12.5 fused4.1 fused4.1 fused4.0 fused1.60x
21.2 split1.2 split5.9 fused5.8 fused4.83x
42.4 split-7.4 fused7.4 fused3.08x
63.6 split-7.8 fused7.8 fused2.17x
84.8 split-8.0 fused8.0 fused1.67x
106.0 split-6.0 split8.1 fused1.35x
127.2 split--8.2 fused1.14x
148.4 split--8.3 fused0.99x
169.5 split--8.3 fused0.87x

RTX 5090

Mistral-7B
ReqsBaselineL=1L=8L=16L=8 vs BaselineL=16 vs Baseline
1134.9 fused136.0 fused135.3 fused131.4 fused1.00x0.97x
2104.0 split103.9 split255.5 fused243.8 fused2.46x2.34x
4204.9 split204.6 split347.0 fused343.1 fused1.69x1.67x
6283.0 split283.0 split385.1 fused382.0 fused1.36x1.35x
8375.8 split376.0 split404.5 fused401.9 fused1.08x1.07x
9422.4 split422.4 split420.9 split407.0 fused1.00x0.96x
10469.1 split468.4 split469.3 split411.9 fused1.00x0.88x
12558.4 split559.8 split558.6 split415.9 fused1.00x0.74x
16736.8 split736.7 split737.3 split425.5 fused1.00x0.58x
Llama-8B
ReqsBaselineL=1L=8L=16L=8 vs BaselineL=16 vs Baseline
1136.6 fused134.1 fused133.0 fused133.3 fused0.97x0.98x
2101.5 split101.4 split251.4 fused245.6 fused2.48x2.42x
4200.0 split199.5 split333.6 fused330.7 fused1.67x1.65x
6275.7 split275.7 split373.8 fused374.3 fused1.36x1.36x
8365.9 split365.0 split394.3 fused395.4 fused1.08x1.08x
9410.7 split410.8 split411.0 split399.3 fused1.00x0.97x
10456.0 split456.0 split456.8 split404.5 fused1.00x0.89x
12544.9 split545.2 split545.4 split410.1 fused1.00x0.75x
16720.7 split720.5 split720.7 split420.5 fused1.00x0.58x
Qwen3.5-9B
ReqsBaselineL=1L=8L=16L=8 vs BaselineL=16 vs Baseline
172.3 fused72.6 fused73.4 fused72.2 fused1.02x1.00x
2100.0 split100.0 split135.3 fused132.4 fused1.35x1.32x
4188.7 split188.5 split271.1 fused264.8 fused1.44x1.40x
6280.0 split280.0 split344.4 fused343.2 fused1.23x1.23x
8370.4 split370.4 split369.3 fused368.2 fused1.00x0.99x
9415.6 split415.7 split415.6 split375.0 fused1.00x0.90x
10462.1 split462.1 split462.4 split382.2 fused1.00x0.83x
12545.8 split545.8 split545.9 split390.6 fused1.00x0.72x
16737.9 split738.1 split738.1 split400.8 fused1.00x0.54x

Radeon AI Pro R9700 (gfx1201)

Mistral-7B
ReqsBaselineL=1L=8L=16L=8 vs BaselineL=16 vs Baseline
145.5 fused87.2 fused90.0 fused88.1 fused1.98x1.94x
234.1 split34.2 split127.9 fused119.6 fused3.75x3.51x
468.1 split67.7 split150.4 fused147.1 fused2.21x2.16x
8134.2 split134.2 split166.6 fused163.9 fused1.24x1.22x
9151.2 split150.7 split151.0 split167.2 fused1.00x1.11x
10167.7 split167.0 split167.1 split167.3 fused1.00x1.00x
11184.1 split183.3 split183.6 split166.9 fused1.00x0.91x
12199.0 split198.7 split199.7 split169.3 fused1.00x0.85x
16263.2 split261.5 split262.6 split170.2 fused1.00x0.65x
Llama-8B
ReqsBaselineL=1L=8L=16L=8 vs BaselineL=16 vs Baseline
144.1 fused80.9 fused80.9 fused79.7 fused1.84x1.81x
233.5 split33.4 split117.8 fused112.1 fused3.52x3.35x
466.6 split66.4 split142.1 fused140.7 fused2.13x2.11x
8132.4 split132.0 split159.7 fused160.6 fused1.21x1.21x
9147.3 split147.1 split147.1 split162.5 fused1.00x1.10x
10163.1 split162.9 split162.8 split162.4 fused1.00x1.00x
11179.2 split178.4 split178.7 split160.4 fused1.00x0.90x
12195.1 split194.9 split194.8 split165.5 fused1.00x0.85x
16256.1 split255.8 split256.3 split167.5 fused1.00x0.65x
Qwen3.5-9B
ReqsBaselineL=1L=8L=16L=8 vs AL=16 vs Baseline
19.4 fused10.8 fused10.8 fused10.8 fused1.15x1.15x
213.4 split13.5 split19.8 fused19.7 fused1.48x1.47x
426.8 split26.7 split36.3 fused36.3 fused1.35x1.35x
852.7 split52.7 split61.3 fused61.1 fused1.16x1.16x
959.3 split59.2 split59.3 split64.1 fused1.00x1.08x
1065.4 split65.4 split65.4 split69.2 fused1.00x1.06x
1172.3 split72.3 split72.2 split73.6 fused1.00x1.02x
1278.7 split78.6 split78.6 split78.0 fused1.00x0.99x
16103.2 split103.1 split103.2 split92.1 fused1.00x0.89x

MI308X (gfx942)

Mistral-7B
ReqsBaselineL=1L=8L=16L=1 vs BaseL=8 vs BaselineL=16 vs Baseline
137.6 fused61.3 fused64.2 fused61.7 fused1.63x1.71x1.64x
247.3 split48.0 split98.8 fused92.1 fused1.01x2.09x1.95x
494.4 split95.1 split112.8 fused111.1 fused1.01x1.19x1.18x
6141.3 split142.0 split120.2 fused118.8 fused1.00x0.85x0.84x
8188.5 split189.2 split124.0 fused123.2 fused1.00x0.66x0.65x
9214.0 split215.6 split192.7 split124.6 fused1.01x0.90x0.58x
10237.5 split239.0 split238.9 split125.9 fused1.01x1.01x0.53x
12284.8 split287.1 split285.6 split128.9 fused1.01x1.00x0.45x
16379.2 split382.0 split381.3 split130.1 fused1.01x1.01x0.34x
Llama-8B
ReqsBaselineL=1L=8L=16L=1 vs BaseL=8 vs BaselineL=16 vs Baseline
137.3 fused63.9 fused64.9 fused62.1 fused1.71x1.74x1.66x
247.1 split47.6 split96.3 fused91.5 fused1.01x2.04x1.94x
490.8 split93.7 split111.7 fused110.3 fused1.03x1.23x1.21x
6139.8 split139.0 split117.9 fused117.6 fused0.99x0.84x0.84x
8186.9 split185.7 split122.6 fused121.6 fused0.99x0.66x0.65x
9212.8 split213.2 split211.8 split123.7 fused1.00x1.00x0.58x
10235.6 split237.0 split236.9 split125.3 fused1.01x1.01x0.53x
12283.2 split283.3 split284.5 split127.0 fused1.00x1.00x0.45x
16376.5 split377.2 split377.6 split129.8 fused1.00x1.00x0.34x
Qwen3.5-9B
ReqsBaselineL=1L=8L=16L=1 vs BaseL=8 vs BaselineL=16 vs Baseline
130.2 fused30.0 fused30.1 fused29.8 fused0.99x1.00x0.99x
240.0 split39.8 split57.6 fused58.8 fused1.00x1.44x1.47x
479.7 split78.9 split96.9 fused96.5 fused0.99x1.22x1.21x
6119.4 split119.0 split107.0 fused106.2 fused1.00x0.90x0.89x
8159.4 split159.0 split112.2 fused112.0 fused1.00x0.70x0.70x
9179.7 split178.9 split177.8 split114.2 fused1.00x0.99x0.64x
10199.3 split182.9 split197.7 split115.9 fused0.92x0.99x0.58x
12239.6 split236.7 split224.1 split118.5 fused0.99x0.94x0.49x
16317.6 split318.1 split315.2 split122.0 fused1.00x0.99x0.38x
Llama-3.3-70B
ReqsBaselineL=1L=8L=16L=1 vs BaseL=8 vs BaselineL=16 vs Baseline
14.7 fused11.3 fused11.3 fused10.7 fused2.40x2.40x2.28x
25.4 split5.4 split12.6 fused11.9 fused1.00x2.33x2.20x
410.7 split10.7 split13.8 fused13.5 fused1.00x1.29x1.26x
616.1 split16.1 split14.3 fused14.1 fused1.00x0.89x0.88x
821.4 split21.4 split14.6 fused14.4 fused1.00x0.68x0.67x
924.2 split24.2 split24.1 split14.7 fused1.00x1.00x0.61x
1026.9 split26.9 split26.9 split14.8 fused1.00x1.00x0.55x
1232.1 split32.1 split32.1 split14.9 fused1.00x1.00x0.46x
1642.7 split42.7 split42.7 split14.9 fused1.00x1.00x0.35x

Changed files

  • bitsandbytes/_ops.py (modified, +0/-2)
  • bitsandbytes/autograd/_functions.py (modified, +6/-1)
  • bitsandbytes/backends/cuda/ops.py (modified, +3/-2)
  • csrc/kernels.cu (modified, +136/-85)
  • csrc/ops.cu (modified, +11/-5)

PR #5547: Update vLLM version support to 0.18.0

Description (problem / solution / changelog)

it seems to work:

$ pytest tests/test_vllm_client_server.py
=============================================================================== test session starts ================================================================================
platform linux -- Python 3.13.11, pytest-8.4.2, pluggy-1.6.0
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
rootdir: /fsx/qgallouedec/trl
configfile: pyproject.toml
plugins: rerunfailures-15.1, order-1.3.0, random-order-1.2.0, asyncio-1.3.0, anyio-4.12.1, env-1.2.0, hypothesis-6.151.13, xdist-3.8.0, datadir-1.8.0, cov-7.0.0, timeout-2.4.0, rich-0.2.0
asyncio: mode=Mode.STRICT, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 51 items                                                                                                                                                                 

tests/test_vllm_client_server.py ..................x..................sssssssss.....                                                                                         [100%]

=============================================================== 41 passed, 9 skipped, 1 xfailed in 583.95s (0:09:43) ===============================================================

nothing special from the https://github.com/vllm-project/vllm/releases/tag/v0.18.0

<!-- CURSOR_SUMMARY -->

[!NOTE] Low Risk Low risk: updates version bounds and user-facing warnings/docs only, with no behavioral changes beyond allowing installs up to vLLM 0.18.0.

Overview Updates TRL’s vLLM support window to include vLLM 0.18.0 by raising the max version in the trl[vllm] optional dependency and aligning the runtime is_vllm_available compatibility check/warning message.

Documentation and a related compatibility patch comment are updated to reflect the new supported range (0.11.0–0.18.0).

<sup>Reviewed by Cursor Bugbot for commit 9a4e62fdb9b25963b090723f64ca13ee764108a5. Bugbot is set up for automated code reviews on this repo. Configure here.</sup>

<!-- /CURSOR_SUMMARY -->

Changed files

  • docs/source/vllm_integration.md (modified, +1/-1)
  • pyproject.toml (modified, +1/-1)
  • trl/_compat.py (modified, +1/-1)
  • trl/import_utils.py (modified, +2/-2)
RAW_BUFFERClick to expand / collapse

Motivation.

bitsandbytes and GGUF are two quantization/format backends in vLLM that see very low usage relative to the maintenance burden they impose (roughly 0.5% and 0.1% respectively from what I can tell).

Both predate the current weight loading architecture (weight_loader_v2) and have not been migrated to it. They inject conditional branches throughout the critical weight-loading path in shared code (linear.py, fused_moe/layer.py, vocab_parallel_embedding.py) in ways that make the codebase harder to maintain and refactor.

In addition, performance is not great when using these methods, with users often citing running GGUF models with llamacpp to be faster due to different priorities wrt bs=1 performance on consumer GPUs.

This RFC proposes deprecating both backends and eventually removing them, to simplify the core weight loading infrastructure and unblock further cleanup.

If we were to choose one over the other, I think removing GGUF would take priority due to the greater usage of BNB. Another option is to propose moving these methods to be OOT quantization plugins, but I doubt the feasibility due to the current need to modify internal structures in vLLM.

Summary

bitsandbytesGGUF
Dedicated Python~1,426 lines~1,464 lines
CUDA kernels0~6,000 lines
Shared code branches~95 lines in 6 locations~75 lines in 5 locations
weight_loader_v2not supportednot supported
TP supportlimited (pre-quant doesn't work)full
CUDA graph support8-bit forces eagerfull
External depbitsandbytes pip packagegguf pip package
Model-specific hacks3 models8+ models

Both formats add ~3,100 lines of dedicated Python, ~170 lines of branching in shared weight loading code, and block migration to weight_loader_v2. GGUF additionally carries ~6,000 lines of CUDA kernels.

The primary benefit of removal isn't the line count; it's making linear.py's weight loading methods readable and refactorable again, and unblocking the weight_loader_v2 migration.

Codebase cost

Dedicated files

These are self-contained and could be deleted as units:

FileLinesPurpose
quantization/bitsandbytes.py609Config, LinearMethod (4bit/8bit), MoEMethod
model_loader/bitsandbytes_loader.py817Full model loader with TP sharding, quant state mgmt, on-the-fly quantization
quantization/gguf.py691Config, LinearMethod, MoEMethod, EmbeddingMethod, kernel dispatch
model_loader/gguf_loader.py437Model loader, GGUF file discovery, tensor name mapping
transformers_utils/gguf_utils.py336GGUF detection, remote download, config patching
Total~2,890

Also ~6,000 lines of GGUF-specific CUDA kernels in csrc/quantization/gguf/ (a partial port of ggml ops).

Conditional branches in shared code

This is the real problem. Both formats add if branches in the hot path of weight loading that every other quantization method has to read around.

linear.py — the worst offender

bitsandbytes adds branches in 6 locations (~95 lines):

  • adjust_bitsandbytes_4bit_shard() — a top-level helper that only exists for bnb
  • ColumnParallelLinear.weight_loader — overloads is_sharded_weight with use_bitsandbytes_4bit
  • MergedColumnParallelLinear.weight_loader — builds an offsets dict and calls adjust_bitsandbytes_4bit_shard(), duplicated for both the fused and per-shard paths
  • QKVParallelLinear.weight_loader — same pattern again, duplicated for both paths
  • RowParallelLinear.weight_loader — overloads is_sharded_weight again

The bnb pattern is essentially copy-pasted 4 times: build an offsets dict mapping shard IDs to original sizes, call adjust_bitsandbytes_4bit_shard() to recompute the offset in packed uint8 space.

GGUF adds branches in 5 locations (~75 lines):

  • ReplicatedLinear.weight_loader — is_gguf_weight / is_gguf_weight_type checks + materialize UninitializedParameter
  • ColumnParallelLinear.weight_loader — same pattern
  • MergedColumnParallelLinear.weight_loader — weight type dict, shard_id tracking, data_container append
  • QKVParallelLinear.weight_loader — same with q/k/v index map
  • RowParallelLinear.weight_loader — same materialize pattern

GGUF uses UninitializedParameter + a data_container list + shard_id_map — a lazy-init approach that forces every weight_loader to have special materialization logic.

fused_moe/layer.py

The weight_loader method has two early-return blocks before the normal loading path:

  • GGUF (~10 lines): is_gguf_weight_type check + UninitializedParameter materialization for MoE experts
  • bnb (~35 lines): flat-packed BNB tensor handling with special w1/w2/w3 logic

vocab_parallel_embedding.py

  • GGUF: is_gguf_weight_type direct copy in weight_loader, bypassing normal shard logic
  • GGUF: tie_weights() returns embed_tokens instead of self because quantized embeddings can't share raw weight tensors

config/model.py

  • _verify_bnb_config(): 25 lines to force eager mode because bnb 8-bit doesn't support CUDA graphs

engine/arg_utils.py

  • Auto-detection overrides for both formats: if is_gguf(self.model): self.quantization = self.load_format = "gguf" and the equivalent for bnb

Neither supports weight_loader_v2

linear.py has a WEIGHT_LOADER_V2_SUPPORTED allowlist. Neither BitsAndBytesLinearMethod nor GGUFLinearMethod is on it — they both use the legacy weight_loader path. This means any effort to migrate the codebase to the cleaner v2 API has to keep the old code path alive for these two backends.

Additional GGUF-specific complexity

  • gguf_loader.py instantiates a dummy HuggingFace model on meta device to extract parameter names for tensor mapping (lines 219-227). This is fragile and breaks when HF model classes change.
  • The loader has ~70 lines of hardcoded model-type name remapping (deepseek_v2/v3, qwen2/3_moe, minimax_m2, cohere, gemma3) that must be updated for each new MoE architecture.
  • transformers_utils/gguf_utils.py adds config patching (maybe_patch_hf_config_from_gguf) and tokenizer extraction from the GGUF container.
  • ~8 model files (llama, llama4, gemma3, exaone, etc.) have GGUF-specific RoPE style detection branches.

Additional bnb-specific complexity

  • bitsandbytes_loader.py has its own TP sharding logic in _unquantized_generator (110 lines) that reimplements what the linear layer weight loaders already do.
  • The loader attaches runtime state as parameter attributes (bnb_quant_state, bnb_shard_offsets, matmul_state) which the quantization method reads during inference. This attribute-passing pattern is unique to bnb and forces checks in every weight loading path.
  • MoE quant state fusion (_fuse_moe_quant_states, 80 lines) manually merges per-expert quant states into fused w13/w2 representations.
  • Pre-quantized bnb models don't work with tensor parallelism at all (hard error at line 551-555).

Proposed Change.

linear.py weight_loader cleanup

Remove ~170 lines of conditional branching across the 4 parallel linear classes. The weight_loader methods become straightforward: determine output/input dim, narrow, copy. No more adjust_bitsandbytes_4bit_shard(), no more UninitializedParameter materialization, no more data_container tracking.

This is the biggest win — these methods are read and modified by anyone working on a new quantization backend, and the bnb/GGUF branches are confusing because they work completely differently from every other quant method.

weight_loader_v2 migration

With bnb and GGUF gone, the legacy weight_loader path could potentially be removed entirely (or at least simplified), since the remaining quant methods are all on the v2 allowlist or could be migrated.

fused_moe/layer.py simplification

Remove ~45 lines of early-return branches from the weight_loader. The control flow becomes linear.

Model loader factory

Remove 2 of ~6 loader classes. The dispatch logic in model_loader/__init__.py gets simpler.

Config / arg_utils

Remove auto-detection branches, CUDA graph workarounds, and bnb/GGUF-specific validation.

Build system

Drop ~6,000 lines of CUDA kernels from csrc/quantization/gguf/ and the corresponding CMakeLists entry. Faster builds.

Dependencies

Drop bitsandbytes and gguf as pip dependencies.

Feedback Period.

Two weeks

CC List.

@robertgshaw2-redhat @simon-mo @Isotr0py @DarkLight1337

Any Other Things.

No response

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

extent analysis

TL;DR

Remove the bitsandbytes and GGUF quantization backends to simplify the core weight loading infrastructure and unblock further cleanup.

Guidance

  • Identify and remove the dedicated files for bitsandbytes and GGUF, including quantization/bitsandbytes.py, model_loader/bitsandbytes_loader.py, quantization/gguf.py, model_loader/gguf_loader.py, transformers_utils/gguf_utils.py, and the CUDA kernels in csrc/quantization/gguf/.
  • Remove the conditional branches in shared code, including the if branches in linear.py, fused_moe/layer.py, vocab_parallel_embedding.py, config/model.py, and engine/arg_utils.py.
  • Simplify the weight_loader methods in linear.py by removing the adjust_bitsandbytes_4bit_shard() and UninitializedParameter materialization logic.
  • Remove the auto-detection branches, CUDA graph workarounds, and bnb/GGUF-specific validation in config/model.py and engine/arg_utils.py.

Example

No code snippet is provided as the issue does not contain specific code that needs to be modified.

Notes

The removal of bitsandbytes and GGUF backends may break existing models that rely on these backends. It is essential to test and verify that the removal does not introduce any regressions.

Recommendation

Apply the workaround of removing the bitsandbytes and GGUF backends, as the benefits of simplifying the core weight loading infrastructure and unblocking further cleanup outweigh the potential costs of breaking existing models.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…
#api#dependency conflict#environment setup#docker error#permission error

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING