`model.load_weights(weights=...)` should be idempotent on a live model with respect to the source weights. Calling it twice with the same HF safetensors source should leave the model state unchanged, not silently corrupt every forward after the second `load_weights` call. vLLM's own [`reload_weights` RPC](https://github.com/vllm-project/vllm/blob/v0.20.2/vllm/v1/worker/gpu_model_runner.py#L4972-L5021) avoids this bug by wrapping `model.load_weights` with `initialize_layerwise_reload` / `finalize_layerwise_reload`, but the raw `model.load_weights` entrypoint is the one external callers use (e.g. [SkyRL's `WorkerWrap.load_weights`](https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/inference_servers/vllm_worker.py#L74-L93)). A few possible fixes: 1. `process_weights_after_loading` could re-run on every `load_weights` invocation, re-applying the kernel-layout transform. 2. `load_weights` could detect that the layer has already been initialized and either: - `raise RuntimeError("load_weights called on already-initialized model; use reload_weights or initialize_layerwise_reload instead.")`. - Route through `initialize_layerwise_reload`/`finalize_layerwise_reload` internally.

vllm - 💡(How to fix) Fix [Bug]: `model.load_weights` silently corrupts MoE forward on second call

vllm2026-05-16 05:44:05

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

Fix Action

Fix / Workaround

============================== CPU Info

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 104 On-line CPU(s) list: 0-103 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8470 CPU family: 6 Model: 143 Thread(s) per core: 1 Core(s) per socket: 52 Socket(s): 2 Stepping: 8 CPU max MHz: 3800.0000 CPU min MHz: 800.0000 BogoMIPS: 4000.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user L1d cache: 4.9 MiB (104 instances) L1i cache: 3.3 MiB (104 instances) L2 cache: 208 MiB (104 instances) L3 cache: 210 MiB (2 instances) NUMA node(s): 8 NUMA node0 CPU(s): 0-12 NUMA node1 CPU(s): 13-25 NUMA node2 CPU(s): 26-38 NUMA node3 CPU(s): 39-51 NUMA node4 CPU(s): 52-64 NUMA node5 CPU(s): 65-77 NUMA node6 CPU(s): 78-90 NUMA node7 CPU(s): 91-103 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

Code Example

INFO 05-14 23:02:56 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
INFO 05-14 23:02:57 [nixl_utils.py:32] NIXL is available
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Apr  7 2026, 20:45:25) [Clang 22.1.1 ] (64-bit runtime)
Python platform              : Linux-6.8.0-1043-nvidia-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration :
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 575.57.08
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               104
On-line CPU(s) list:                  0-103
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8470
CPU family:                           6
Model:                                143
Thread(s) per core:                   1
Core(s) per socket:                   52
Socket(s):                            2
Stepping:                             8
CPU max MHz:                          3800.0000
CPU min MHz:                          800.0000
BogoMIPS:                             4000.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user
L1d cache:                            4.9 MiB (104 instances)
L1i cache:                            3.3 MiB (104 instances)
L2 cache:                             208 MiB (104 instances)
L3 cache:                             210 MiB (2 instances)
NUMA node(s):                         8
NUMA node0 CPU(s):                    0-12
NUMA node1 CPU(s):                    13-25
NUMA node2 CPU(s):                    26-38
NUMA node3 CPU(s):                    39-51
NUMA node4 CPU(s):                    52-64
NUMA node5 CPU(s):                    65-77
NUMA node6 CPU(s):                    78-90
NUMA node7 CPU(s):                    91-103
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Vulnerability Vmscape:                Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvcc-cu12==12.9.86
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.17.1.4
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.28.9
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] sentence-transformers==5.4.1
[pip3] torch==2.11.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu129
[pip3] torchdata==0.11.0
[pip3] torchvision==0.26.0+cu129
[pip3] transformers==5.8.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.2
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	NIC9	NIC10	NIC11	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	PIX	PIX	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	0-12	0		N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	26-38	2		N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	39-51	3		N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	13-25	1		N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	PIX	PIX	PIX	SYS	SYS	SYS	52-64	4		N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	78-90	6		N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	91-103	7		N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	65-77	5		N/A
NIC0	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC1	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC2	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	PIX	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC3	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC4	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC5	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS
NIC6	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	PIX	SYS	SYS	SYS
NIC7	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	PIX	SYS	SYS	SYS
NIC8	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	PIX	 X 	SYS	SYS	SYS
NIC9	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS
NIC10	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS
NIC11	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/data/users/user/code/repo/.venv/lib/python3.12/site-packages/cv2/../../lib64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/opt/hpcx/ompi/lib:/opt/hpcx/hcoll/lib:/opt/hpcx/sharp/lib:/opt/hpcx/ucx/lib:/opt/hpcx/ucx/lib/ucx:/opt/hpcx/ucc/lib:/opt/hpcx/ucc/lib/ucc:/opt/hpcx/nccl_rdma_sharp_plugin/lib:/usr/local/cuda/lib64:
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_user
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

---

...Sot remis二十四节气ttsueEil Извест彻底s [data)Gilr)fam)Aam shcuiteg...

---

import asyncio
import glob
import os
import sys
import time
from pathlib import Path

import vllm
from huggingface_hub import snapshot_download
from vllm.model_executor.model_loader.weight_utils import (
    safetensors_weights_iterator,
)
from vllm.sampling_params import SamplingParams

MODEL = "Qwen/Qwen3.6-35B-A3B"  # Any unquantized MoE that lands on FlashInfer
PROMPT = "The capital of France is"
MAX_TOKENS = 64  # Keep the repro under a minute of generation


class WeightRoundTripExt:
    """
    vLLM worker extension that re-invokes `model.load_weights` against HF disk.

    Emulate https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/inference_servers/vllm_worker.py#L74-L93.
    """

    def reload_from_hf_disk(self, model_path: str) -> None:
        files = sorted(glob.glob(str(Path(model_path) / "*.safetensors")))
        self.model_runner.model.load_weights(
            weights=safetensors_weights_iterator(files, use_tqdm_on_load=False)
        )


async def _generate(
    engine: vllm.AsyncLLMEngine, sampling_params: SamplingParams, label: str
) -> str:
    async for output in engine.generate(
        PROMPT, sampling_params, request_id=f"{label}-{time.time_ns()}"
    ):
        if output.finished:
            return output.outputs[0].text
    assert False, "Not part of this repro"


async def main() -> int:
    engine = vllm.AsyncLLMEngine.from_engine_args(
        vllm.AsyncEngineArgs(
            model=MODEL,
            # Any TP that fits the unquantized ~70 GiB weights works
            tensor_parallel_size=8,
            max_model_len=4096,
            gpu_memory_utilization=0.4,
            enforce_eager=True,
            trust_remote_code=True,
            dtype="bfloat16",
            # Qwen3.6 has Gated DeltaNet layers; route prefill through vLLM's bundled
            # Triton/FLA path so engine init does not need FlashInfer GDN dev headers
            additional_config={"gdn_prefill_backend": "triton"},
            worker_extension_cls=f"{_SCRIPT_MODULE}.WeightRoundTripExt",
        )
    )

    # Greedy decoding makes generation deterministic on the model's logits,
    # so any post-reload divergence is forward-output corruption (not sampler noise)
    sampling = SamplingParams(temperature=0, max_tokens=MAX_TOKENS)

    text_pre = await _generate(engine, sampling, "pre")
    print(f"PRE  ({len(text_pre)} chars): {text_pre!r}", flush=True)

    # Round-trip the HF safetensors back through `model.load_weights` inside
    # every vLLM worker via `collective_rpc`
    await engine.collective_rpc(
        "reload_from_hf_disk",
        args=(
            snapshot_download(repo_id=MODEL, allow_patterns=["*.safetensors", "*.json"]),
        ),
    )

    text_post = await _generate(engine, sampling, "post")
    print(f"POST ({len(text_post)} chars): {text_post!r}", flush=True)
    print(f"VERDICT: identical={text_pre == text_post}", flush=True)
    return 0 if text_pre == text_post else 1


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

---

PRE  (308 chars): ' Paris, a city renowned for its rich history, culture, and iconic landmarks. Paris is situated in the north-central part of France, along the Seine River. It is the most populous city in France and one of the most visited cities in the world, attracting millions of tourists each year.\nParis is known for its'
POST (93 chars): ' not yet SH\xa0 1tARSEPn 19..51.2ons.\n  1AudANGEANGEE The 45udANGEPF iner  1 1ad 6 1u 2uh9e 1 1u'
VERDICT: identical=False

RAW_BUFFERClick to expand / collapse

Your current environment

<details> <summary>The output of <code>python collect_env.py</code></summary>

INFO 05-14 23:02:56 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
INFO 05-14 23:02:57 [nixl_utils.py:32] NIXL is available
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Apr  7 2026, 20:45:25) [Clang 22.1.1 ] (64-bit runtime)
Python platform              : Linux-6.8.0-1043-nvidia-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   :
GPU models and configuration :
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3

Nvidia driver version        : 575.57.08
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               104
On-line CPU(s) list:                  0-103
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8470
CPU family:                           6
Model:                                143
Thread(s) per core:                   1
Core(s) per socket:                   52
Socket(s):                            2
Stepping:                             8
CPU max MHz:                          3800.0000
CPU min MHz:                          800.0000
BogoMIPS:                             4000.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user
L1d cache:                            4.9 MiB (104 instances)
L1i cache:                            3.3 MiB (104 instances)
L2 cache:                             208 MiB (104 instances)
L3 cache:                             210 MiB (2 instances)
NUMA node(s):                         8
NUMA node0 CPU(s):                    0-12
NUMA node1 CPU(s):                    13-25
NUMA node2 CPU(s):                    26-38
NUMA node3 CPU(s):                    39-51
NUMA node4 CPU(s):                    52-64
NUMA node5 CPU(s):                    65-77
NUMA node6 CPU(s):                    78-90
NUMA node7 CPU(s):                    91-103
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected
Vulnerability Vmscape:                Mitigation; IBPB before exit to userspace

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.8.post1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvcc-cu12==12.9.86
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.17.1.4
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.4.2
[pip3] nvidia-cutlass-dsl-libs-base==4.4.2
[pip3] nvidia-ml-py==13.595.45
[pip3] nvidia-nccl-cu12==2.28.9
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.4.5
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] sentence-transformers==5.4.1
[pip3] torch==2.11.0+cu129
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0+cu129
[pip3] torchdata==0.11.0
[pip3] torchvision==0.26.0+cu129
[pip3] transformers==5.8.0
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.20.2
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	NIC0	NIC1	NIC2	NIC3	NIC4	NIC5	NIC6	NIC7	NIC8	NIC9	NIC10	NIC11	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	PIX	PIX	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	0-12	0		N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	26-38	2		N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	39-51	3		N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	13-25	1		N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	PIX	PIX	PIX	SYS	SYS	SYS	52-64	4		N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	78-90	6		N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	91-103	7		N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	65-77	5		N/A
NIC0	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC1	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC2	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	PIX	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC3	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC4	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS
NIC5	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS
NIC6	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	PIX	PIX	SYS	SYS	SYS
NIC7	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	 X 	PIX	SYS	SYS	SYS
NIC8	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	PIX	 X 	SYS	SYS	SYS
NIC9	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS
NIC10	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS
NIC11	SYS	SYS	SYS	SYS	SYS	SYS	SYS	PIX	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7
  NIC8: mlx5_8
  NIC9: mlx5_9
  NIC10: mlx5_10
  NIC11: mlx5_11

==============================
     Environment Variables
==============================
LD_LIBRARY_PATH=/data/users/user/code/repo/.venv/lib/python3.12/site-packages/cv2/../../lib64:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/targets/x86_64-linux/lib:/opt/hpcx/ompi/lib:/opt/hpcx/hcoll/lib:/opt/hpcx/sharp/lib:/opt/hpcx/ucx/lib:/opt/hpcx/ucx/lib/ucx:/opt/hpcx/ucc/lib:/opt/hpcx/ucc/lib/ucc:/opt/hpcx/nccl_rdma_sharp_plugin/lib:/usr/local/cuda/lib64:
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_user
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

</details>

🐛 Describe the bug

A loaded model's load_weights method (e.g. Qwen3MoeForCausalLM.load_weights) is not idempotent on a live AsyncLLMEngine for unquantized MoE models that land on the FlashInfer-CUTLASS or FlashInfer-TRTLLM backend.

Calling load_weights a second time, even with the same HF safetensors source the engine was originally loaded from, silently corrupts every subsequent forward into a stable multi-language subword soup:

...Sot remis二十四节气ttsueEil Извест彻底s [data)Gilr)fam)Aam shcuiteg...

Mechanism

I ran an extensive bisect on this between v0.19.1 and v0.20.2, and arrived at https://github.com/vllm-project/vllm/pull/36286 causing this regression for the MoE. backends FLASHINFER_CUTLASS and FLASHINFER_TRTLLM.

https://github.com/vllm-project/vllm/pull/36286 moves every per-backend kernel-layout transform (e.g. FLASHINFER_CUTLASS, FLASHINFER_TRTLLM) out of UnquantizedFusedMoEMethod.process_weights_after_loading's body and routes it through _setup_kernel → convert_to_unquantized_kernel_format.
Therein, convert_to_unquantized_kernel_format applies a MoE-backend-specific layout mutation once at engine init:
- FLASHINFER_CUTLASS (gated MoE): swap_w13_to_w31(w13) (swaps the [w1; w3] halves to [w3; w1]).
  - Before #36286, CUTLASS did no swap, so reload was safe on this path.
- FLASHINFER_TRTLLM: swap_w13_to_w31(w13) plus convert_moe_weights_to_flashinfer_trtllm_block_layout(...).
  - #36286 now routes the install through replace_parameter, which preserves weight_loader, making the corruption happen on TRTLLM too (as it no-longer falls through to default_weight_loader).
Next in _setup_kernel, the transformed tensor is installed via replace_parameter, so layer.w13_weight now lives in kernel layout, not checkpoint layout. Critically, replace_parameter also copies the per-expert weight_loader attribute onto the new parameter, so the next call into load_weights will still route through FusedMoE._load_w13.
When external code calls model.load_weights(weights=...) a second time, FusedMoE._load_w13 (still attached to the new parameter, per step 3) writes raw checkpoint-format [w1; w3] bytes into layer.w13_weight.
But the buffer is in [w3; w1] (or, for TRTLLM, block-permuted) layout. The write hits the wrong half-slot; the kernel reads the half it expects to be w1 but finds w3 (and vice versa); the resulting forward output is a multilingual subword soup.

Reproducer

import asyncio
import glob
import os
import sys
import time
from pathlib import Path

import vllm
from huggingface_hub import snapshot_download
from vllm.model_executor.model_loader.weight_utils import (
    safetensors_weights_iterator,
)
from vllm.sampling_params import SamplingParams

MODEL = "Qwen/Qwen3.6-35B-A3B"  # Any unquantized MoE that lands on FlashInfer
PROMPT = "The capital of France is"
MAX_TOKENS = 64  # Keep the repro under a minute of generation


class WeightRoundTripExt:
    """
    vLLM worker extension that re-invokes `model.load_weights` against HF disk.

    Emulate https://github.com/NovaSky-AI/SkyRL/blob/skyrl-v0.2.0/skyrl/backends/skyrl_train/inference_servers/vllm_worker.py#L74-L93.
    """

    def reload_from_hf_disk(self, model_path: str) -> None:
        files = sorted(glob.glob(str(Path(model_path) / "*.safetensors")))
        self.model_runner.model.load_weights(
            weights=safetensors_weights_iterator(files, use_tqdm_on_load=False)
        )


async def _generate(
    engine: vllm.AsyncLLMEngine, sampling_params: SamplingParams, label: str
) -> str:
    async for output in engine.generate(
        PROMPT, sampling_params, request_id=f"{label}-{time.time_ns()}"
    ):
        if output.finished:
            return output.outputs[0].text
    assert False, "Not part of this repro"


async def main() -> int:
    engine = vllm.AsyncLLMEngine.from_engine_args(
        vllm.AsyncEngineArgs(
            model=MODEL,
            # Any TP that fits the unquantized ~70 GiB weights works
            tensor_parallel_size=8,
            max_model_len=4096,
            gpu_memory_utilization=0.4,
            enforce_eager=True,
            trust_remote_code=True,
            dtype="bfloat16",
            # Qwen3.6 has Gated DeltaNet layers; route prefill through vLLM's bundled
            # Triton/FLA path so engine init does not need FlashInfer GDN dev headers
            additional_config={"gdn_prefill_backend": "triton"},
            worker_extension_cls=f"{_SCRIPT_MODULE}.WeightRoundTripExt",
        )
    )

    # Greedy decoding makes generation deterministic on the model's logits,
    # so any post-reload divergence is forward-output corruption (not sampler noise)
    sampling = SamplingParams(temperature=0, max_tokens=MAX_TOKENS)

    text_pre = await _generate(engine, sampling, "pre")
    print(f"PRE  ({len(text_pre)} chars): {text_pre!r}", flush=True)

    # Round-trip the HF safetensors back through `model.load_weights` inside
    # every vLLM worker via `collective_rpc`
    await engine.collective_rpc(
        "reload_from_hf_disk",
        args=(
            snapshot_download(repo_id=MODEL, allow_patterns=["*.safetensors", "*.json"]),
        ),
    )

    text_post = await _generate(engine, sampling, "post")
    print(f"POST ({len(text_post)} chars): {text_post!r}", flush=True)
    print(f"VERDICT: identical={text_pre == text_post}", flush=True)
    return 0 if text_pre == text_post else 1


if __name__ == "__main__":
    sys.exit(asyncio.run(main()))

Output with Python 3.12, vllm==0.20.2, and huggingface-hub==1.11.0:

PRE  (308 chars): ' Paris, a city renowned for its rich history, culture, and iconic landmarks. Paris is situated in the north-central part of France, along the Seine River. It is the most populous city in France and one of the most visited cities in the world, attracting millions of tourists each year.\nParis is known for its'
POST (93 chars): ' not yet SH\xa0 1tARSEPn 19..51.2ons.\n  1AudANGEANGEE The 45udANGEPF iner  1 1ad 6 1u 2uh9e 1 1u'
VERDICT: identical=False

Both generations use the same prompt with temperature 0. The only state change between PRE and POST is the collective_rpc("reload_from_hf_disk", ...) call, which inside each vLLM worker invokes model.load_weights(safetensors_weights_iterator(...)) against the same safetensors files vLLM originally loaded.

Expected behavior

model.load_weights(weights=...) should be idempotent on a live model with respect to the source weights. Calling it twice with the same HF safetensors source should leave the model state unchanged, not silently corrupt every forward after the second load_weights call.

vLLM's own reload_weights RPC avoids this bug by wrapping model.load_weights with initialize_layerwise_reload / finalize_layerwise_reload, but the raw model.load_weights entrypoint is the one external callers use (e.g. SkyRL's WorkerWrap.load_weights).

A few possible fixes:

process_weights_after_loading could re-run on every load_weights invocation, re-applying the kernel-layout transform.
load_weights could detect that the layer has already been initialized and either:
- raise RuntimeError("load_weights called on already-initialized model; use reload_weights or initialize_layerwise_reload instead.").
- Route through initialize_layerwise_reload/finalize_layerwise_reload internally.

Before submitting a new issue...

Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

FAQ

Expected behavior

A few possible fixes:

process_weights_after_loading could re-run on every load_weights invocation, re-applying the kernel-layout transform.
load_weights could detect that the layer has already been initialized and either:
- raise RuntimeError("load_weights called on already-initialized model; use reload_weights or initialize_layerwise_reload instead.").
- Route through initialize_layerwise_reload/finalize_layerwise_reload internally.

#api #environment variable #parallel task #integration issue #memory leak

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

vllm - 💡(How to fix) Fix [Bug]: `model.load_weights` silently corrupts MoE forward on second call

Recommended Tools

GitHub issue graph ai analysis

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

Mechanism

Reproducer

Expected behavior

Before submitting a new issue...

FAQ

Expected behavior

Still need to ship something?

TRENDING

vllm - 💡(How to fix) Fix [Bug]: `model.load_weights` silently corrupts MoE forward on second call

Recommended Tools

GitHub issue graph ai analysis

Fix Action

Fix / Workaround

============================== CPU Info

Code Example

Your current environment

🐛 Describe the bug

Mechanism

Reproducer

Expected behavior

Before submitting a new issue...

FAQ

Expected behavior

Still need to ship something?

RELATED_DISCOVERY

TRENDING