ERROR 03-12 20:19:38 [multiproc_executor.py:844] WorkerProc failed to start. ERROR 03-12 20:19:38 [multiproc_executor.py:844] Traceback (most recent call last): ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/v1/executor/multiproc_executor.py", line 813, in worker_main ERROR 03-12 20:19:38 [multiproc_executor.py:844] worker = WorkerProc(*args, **kwargs) ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/v1/executor/multiproc_executor.py", line 619, in init ERROR 03-12 20:19:38 [multiproc_executor.py:844] self.worker.load_model() ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/v1/worker/gpu_worker.py", line 337, in load_model ERROR 03-12 20:19:38 [multiproc_executor.py:844] self.model_runner.load_model(load_dummy_weights=dummy_weights) ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/v1/worker/gpu_model_runner.py", line 4277, in load_model ERROR 03-12 20:19:38 [multiproc_executor.py:844] self.model = model_loader.load_model( ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/model_loader/base_loader.py", line 62, in load_model ERROR 03-12 20:19:38 [multiproc_executor.py:844] self.load_weights(model, model_config) ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/model_loader/default_loader.py", line 290, in load_weights ERROR 03-12 20:19:38 [multiproc_executor.py:844] loaded_weights = model.load_weights(self.get_all_weights(model_config, model)) ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/models/nemotron_h.py", line 957, in load_weights ERROR 03-12 20:19:38 [multiproc_executor.py:844] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper) ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights ERROR 03-12 20:19:38 [multiproc_executor.py:844] return original_load_weights(self, weights, *args, **kwargs) ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/models/utils.py", line 340, in load_weights ERROR 03-12 20:19:38 [multiproc_executor.py:844] autoloaded_weights = set(self._load_module("", self.module, weights)) ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/models/utils.py", line 287, in _load_module ERROR 03-12 20:19:38 [multiproc_executor.py:844] yield from self._load_module( ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/models/utils.py", line 260, in _load_module ERROR 03-12 20:19:38 [multiproc_executor.py:844] loaded_params = module_load_weights(weights) ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/models/nemotron_h.py", line 756, in load_weights ERROR 03-12 20:19:38 [multiproc_executor.py:844] success = weight_loader( ERROR 03-12 20:19:38 [multiproc_executor.py:844] ^^^^^^^^^^^^^^ ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/layers/fused_moe/layer.py", line 1323, in weight_loader ERROR 03-12 20:19:38 [multiproc_executor.py:844] self._load_model_weight_or_group_weight_scale( ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/layers/fused_moe/layer.py", line 915, in _load_model_weight_or_group_weight_scale ERROR 03-12 20:19:38 [multiproc_executor.py:844] self._load_w2( ERROR 03-12 20:19:38 [multiproc_executor.py:844] File "/home/bnellnm/nm-vllm-new/vllm/model_executor/layers/fused_moe/layer.py", line 1001, in load_w2 ERROR 03-12 20:19:38 [multiproc_executor.py:844] expert_data.copy(loaded_weight) ERROR 03-12 20:19:38 [multiproc_executor.py:844] RuntimeError: The size of tensor a (3072) must match the size of tensor b (2688) at non-singleton dimension 0

Fix Plan

The fix involves modifying the weight_loader function in layer.py to handle the rounded dimension correctly.

• Modify the weight_loader function to check if the loaded weight dimension matches the expected dimension.
• If the dimensions do not match, adjust the loaded weight to match the expected dimension.

Example code:

def weight_loader(self, loaded_weight):
    # Check if the loaded weight dimension matches the expected dimension
    if loaded_weight.shape[0] != self.expected_dim:
        # Adjust the loaded weight to match the expected dimension
        loaded_weight = loaded_weight[:self.expected_dim]
    self._load_model_weight_or_group_weight_scale(loaded_weight)

Alternatively, you can also modify the load_weights function in nemotron_h.py to pass the correct dimension to the weight_loader function.

• Modify the load_weights function to calculate the correct dimension and pass it to the weight_loader function.

def load_weights(self, weights):
    # Calculate the correct dimension
    correct_dim = self.calculate_correct_dim()
    # Pass the correct dimension to the weight_loader function
    weight_loader(correct_dim, weights)

Verification

To verify that the fix worked, run the repro steps again and check if the error is resolved.

• Run the command:

python3 examples/offline_inference/data_parallel.py \
        --model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" \
        --all2all-backend deepep_low_latency \
        --trust-remote-code \
        -dp=2 \
        -tp=1 \
        --enforce-eager

• Check if the error message is resolved and the model loads correctly.

Extra Tips

• Make sure to test the fix thoroughly to ensure that it does not introduce any new issues.
• Consider adding additional error handling to handle cases where the loaded weight dimension does not match the expected dimension.
• Review the code changes to ensure that they are consistent with the overall architecture and design of the project.