vllm - 💡(How to fix) Fix [vLLM IR] Op test & benchmark infra [1 comments, 2 participants]

TL;DR

Create infrastructure and utilities to simplify adding tests and benchmarks for new operations, including tests for provider semantics and benchmarks with a standardized driver file.

Guidance

• Develop a test framework that checks the semantics of each operation across all supported providers, ensuring consistency with native implementations.
• Design a utility for providers to write custom tests, particularly for validating supports_args functionality, to reduce duplication.
• Establish a benchmarking system with a single driver file, where each operation registers a get_inputs function for generating benchmark inputs based on BenchmarkCase parameters.
• Define a set of standard BenchmarkCase parameters, such as hidden_size, to ensure consistent benchmarking across operations.

Example

class BenchmarkCase:
    def __init__(self, num_tokens: int, hidden_size: int):
        self.num_tokens = num_tokens
        self.hidden_size = hidden_size

def get_inputs(case: BenchmarkCase) -> list[torch.Tensor]:
    # Operation-specific implementation to generate inputs based on BenchmarkCase
    pass

Notes

The proposed solution focuses on creating a structured approach to testing and benchmarking operations. However, the exact implementation details, such as how to reduce duplication in provider tests, are left to be determined.

Recommendation

Apply workaround by starting with the proposed infrastructure and utilities, and iteratively refine them based on the needs of each operation and provider, as the issue suggests a phased approach to developing these tests and benchmarks.