vllm - 💡(How to fix) Fix [Usage]: How to use Env in yaml [2 comments, 2 participants]

Fix: Using Environment Variables to Control GPU Memory Utilization

To control GPU memory utilization through environment variables, you can modify the server-config.yaml file to reference environment variables.

Fix Plan

• Modify the server-config.yaml file to use environment variables for GPU memory utilization:

gpu-memory-utilization: ${GPU_MEMORY_UTILIZATION}

• Set the GPU_MEMORY_UTILIZATION environment variable before starting the model:

export GPU_MEMORY_UTILIZATION=16G

• Alternatively, you can use a command-line argument to set the environment variable when starting the model:

GPU_MEMORY_UTILIZATION=16G vllm-server --config server-config.yaml

• In your code, you can access the environment variable using your programming language of choice. For example, in Python:

import os

gpu_memory_utilization = os.environ.get('GPU_MEMORY_UTILIZATION')

Verification

• Verify that the environment variable is set correctly by checking the server-config.yaml file or printing the value of the variable in your code.
• Check the GPU memory utilization to ensure it matches the value set by the environment variable.

Extra Tips

• Make sure to update the server-config.yaml file to reference the environment variable correctly.
• You can also use other environment variables to control other parameters in the server-config.yaml file.