vllm - 💡(How to fix) Fix [Bug]: Non-monotonic latency at ctx=8192: concurrency 1 slower than concurrency 2 (local vLLM) [3 comments, 2 participants]

Fix Plan

The fix involves modifying the batching strategy to handle long-context requests more efficiently at low concurrency.

Step-by-Step Solution:

1. Update vLLM configuration: Modify the vllm_config.json file to adjust the batching parameters for long-context requests.
2. Implement custom batching logic: Create a custom batching function that prioritizes long-context requests at low concurrency.

Example code snippet:

import torch

def custom_batching_function(requests, ctx_size):
    # Prioritize long-context requests at low concurrency
    if ctx_size == 8192 and len(requests) == 1:
        # Process the request immediately
        return [requests]
    else:
        # Default batching logic
        batches = []
        batch = []
        for request in requests:
            batch.append(request)
            if len(batch) == 2:
                batches.append(batch)
                batch = []
        if batch:
            batches.append(batch)
        return batches

# Update the vLLM configuration to use the custom batching function
vllm_config = {
    'batching_function': custom_batching_function,
    # Other configuration parameters...
}

1. Apply the updated configuration: Restart the vLLM server with the updated configuration.

Verification

To verify the fix, re-run the experiment with the updated configuration and measure the p95 latency at concurrency=1 and concurrency=2.

Example command:

vllm serve Qwen/Qwen2-0.5B-Instruct --host 0.0.0.0 --port 8000 --config vllm_config.json

Monitor the latency metrics and compare the results with the previous measurements.

Extra Tips

• Monitor the system resources and adjust the batching parameters accordingly to avoid overloading the system.
• Consider implementing a more sophisticated batching strategy that takes into account the context size and concurrency level.