🚀 The feature, motivation and pitch

RFC: Auto-select and run Operator Performance Benchmarks from PR Diffs

Summary

This RFC explores automatically mapping code changes in a PR to the operator performance benchmarks that should run against those changes. Today this mapping does not exist. The benchmarks exist, the comparison workflow exists, but nothing connects a diff to the right benchmarks. This RFC presents a script that does that mapping, discusses adding missing benchmark tests over time, and outlines how CI integration could work as a next step. Feedback on the approach is welcome.

Problem

When a PR modifies operator kernel code, reviewers have no performance signal before merging. Regressions show up later in model-level benchmarks or user reports. By then, finding the responsible PR means bisecting across hundreds of commits.

The infrastructure to catch this already exists. benchmarks/operator_benchmark/ has test files covering some of operators, and operator_microbenchmark_compare.yml can build two wheels, run benchmarks on both, and post a comparison as a PR comment. But the workflow is manual: someone picks an operator from a dropdown, picks a GPU, and triggers it. For most PRs, nobody does this.

The missing piece is the mapping: given a diff, which benchmarks should run? And when a benchmark doesn't exist for an affected operator, how do we know its missing?

The script

The RFC includes a single Python script (tools/testing/perfmap.py, ~250 lines) that takes a base ref, maps the diff to operators and benchmarks, runs the benchmarks, and produces a comparison across branches:

# On your feature branch
python tools/testing/perfmap.py run --label my-feature

# Switch to baseline, build, run again
python tools/testing/perfmap.py run --label my-feature

# Compare
python tools/testing/perfmap.py compare --label my-feature

Internally the script walks this chain:

1. Parse git diff to get changed files and line ranges
2. Find which functions enclose those changed lines using an AST parser
3. Look up those function names in native_functions.yaml to get operator names
4. Match operator names to benchmark test files in operator_benchmark/pt/
5. Run the matched benchmarks and save results
6. On compare, produce a markdown table

The comparison output looks like this and can be pasted directly into a PR description or GitHub comment to give reviewers a performance signal:

For function detection, the script currently uses tree-sitter with C++ and Python grammars. This gives exact function boundaries rather than relying on git's hunk headers. tree-sitter adds pip dependencies (tree-sitter, tree-sitter-cpp, tree-sitter-python). Alternatives include ctags, libclang, or a hand-written parser. Input on the right choice here is welcome.

Adding missing benchmarks

The script reports "NO BENCHMARK" when an operator is affected but has no benchmark test file. This is useful on its own: it tells us exactly which operators need benchmarks and prioritizes them by how often their kernels are changed.

Currently a small fraction of operators have benchmarks. Growing this coverage is an incremental effort, independent of the mapping script itself. Each new benchmark test file is automatically discovered by the script on its next run.

Future goals

• CI integration. If this approach proves useful, the script can plug into the existing operator_microbenchmark_compare.yml workflow by adding a pull_request trigger and a detect-operators job that feeds the results to the existing benchmark runner. This would automate the process so reviewers get a performance comparison comment on every PR that touches operator code.
• Increase benchmark coverage. Most operators registered through native_functions.yaml don't have benchmarks yet. The script identifies which operators are missing coverage, which helps prioritize adding new benchmark test files. Each new file is automatically discovered.
• Cover operators outside native_functions.yaml. Code not reachable through native_functions.yaml needs to be benchmarked. Extending the mapping to cover these would improve recall.
• Eager + compiled benchmarks. The benchmark runner supports --use-compile. Changes to inductor/dynamo only affect the compile path, so triggering both modes selectively would be valuable.
• File-level fallback. When function-level matching fails, falling back to all operators in the changed file would catch more regressions at the cost of running more benchmarks.
• Canary set for shared infrastructure. Changes to code like CUDACachingAllocator or TensorIterator affect many operators but can't map to specific ones. A fixed set of representative benchmarks for these paths would catch broad regressions.

Comments and suggestions welcome.

Code changes can be found here https://github.com/pytorch/pytorch/pull/186835/changes

Alternatives

No response

Additional context

No response

cc @malfet @pytorch/pytorch-dev-infra