vllm bench serve
--host localhost --port 8000
--tokenizer nvidia/DeepSeek-R1-0528-NVFP4
--trust-remote-code --dataset-name random
--random-input-len 2000 --random-output-len 1000
--ignore-eos --max-concurrency 128
--request-rate inf --num-prompts 640

Server crashes with: MemoryError: CUDA out of memory. Tried to allocate 3.94 GiB. GPU has 184.00 GiB total, 2.38 GiB free.

Fix Plan

To resolve the MemoryError: CUDA out of memory issue, we need to optimize the memory utilization of the VLLM server. Here are the steps:

• Reduce the --max-num-seqs and --max-num-batched-tokens parameters to decrease the memory requirements:
  • --max-num-seqs 512
  • --max-num-batched-tokens 4096
• Decrease the --gpu-memory-utilization to prevent over-allocation of GPU memory:
  • --gpu-memory-utilization 0.7
• Consider reducing the --random-input-len and --random-output-len parameters in the benchmarking command to reduce memory usage:
  • --random-input-len 1000
  • --random-output-len 500
• Update the compilation_config to reduce the max_cudagraph_capture_size:
  • --compilation_config.max_cudagraph_capture_size 1024

Example updated command:

python3 -m vllm.entrypoints.openai.api_server \
  --host 0.0.0.0 --port 8000 \
  --model nvidia/DeepSeek-R1-0528-NVFP4 \
  --trust-remote-code --no-enable-prefix-caching \
  --dtype auto --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 --data-parallel-size 4 \
  --max-num-seqs 512 --max-model-len 10240 \
  --gpu-memory-utilization 0.7 \
  --max-num-batched-tokens 4096 \
  --enable-expert-parallel --async-scheduling \
  --compilation_config.max_cudagraph_capture_size 1024 \
  --compilation_config.cudagraph_mode FULL_DECODE_ONLY

Verification

Run the updated command and monitor the GPU memory usage to ensure it does not exceed the available memory. You can use tools like nvidia-smi to monitor the GPU memory usage.

Extra Tips

• Regularly monitor the GPU memory usage and adjust the parameters as needed to prevent out-of-memory errors.
• Consider using a more efficient model or optimizing the model architecture to reduce memory requirements.
• If the issue persists, try reducing the --num-prompts parameter in the benchmarking command to decrease the load on the server.