vllm - 💡(How to fix) Fix [Bug]: Docker Model Runner vLLM ignores tensor parallel config and starts with world_size=1 on 4x RTX 3060 [2 comments, 1 participants]

Fix Plan

To fix the issue, you need to configure the Docker Model Runner to use tensor parallelism across 4 GPUs.

• Stop the runner: docker model stop-runner
• Uninstall the runner: docker model uninstall-runner --images
• Install the runner with vLLM + CUDA and specify the number of GPUs:

  docker model install-runner --backend vllm --gpu cuda --num-gpus 4

• Update the model configuration to use tensor parallelism:

  import torch
  
  # Set the device (4 GPUs)
  device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
  torch.cuda.set_device(device)
  
  # Set the tensor parallel size
  tensor_parallel_size = 4
  
  # Load the model with tensor parallelism
  from transformers import AutoModel
  model = AutoModel.from_pretrained("huggingface.co/qwen/qwen3.5-9b", 
                                    device_map={0: [0, 1, 2, 3]})  # Map the model to 4 GPUs

Verification

To verify that the fix worked, check the model configuration and GPU usage:

• Check the model configuration: docker model show
• Check the GPU usage: nvidia-smi

Extra Tips

• Make sure you have the latest version of the Docker Model Runner and vLLM installed.
• Check the documentation for the docker model install-runner command for more options and configuration parameters.
• You can also use the --data-parallel-size and --pipeline-parallel-size options to configure the data parallelism and pipeline parallelism.