expert_data.copy_(loaded_weight)

RuntimeError: The size of tensor a (2048) must match the size of tensor b (16)

TL;DR

Modify the deepseek_v4.py loader to support the Base model's FP8 weights by adjusting the tensor shapes, packing, and quantization scales.

Guidance

• Identify the specific lines of code in fused_moe/layer.py where the RuntimeError occurs and adjust the tensor copy operation to accommodate the FP8 weights.
• Update the weight loader in deepseek_v4.py to handle the differences in tensor shapes and quantization scales between the Instruct and Base models.
• Consider adding a model-type check to use the correct weight loading logic based on the model being used.
• Verify that the modified loader correctly loads the Base model weights and resolves the RuntimeError.

Example

# Example of modified weight loading logic
if model_type == "Base":
    # Load FP8 weights with correct tensor shape and quantization scale
    expert_data.copy_(loaded_weight_fp8)
else:
    # Load FP4 weights with correct tensor shape and quantization scale
    expert_data.copy_(loaded_weight_fp4)

Notes

The solution requires modifying the existing code to support the Base model's FP8 weights. The exact changes will depend on the specific implementation details of the deepseek_v4.py loader and the fused_moe/layer.py module.

Recommendation

Apply workaround: Modify the deepseek_v4.py loader to support the Base model's FP8 weights, as this will allow the Base model to be used with the vLLM.