pytorch - ✅(Solved) Fix Multi-level Skip Mechanism for backends(test class or test method level) [1 pull requests, 5 comments, 2 participants]

pytorch2026-03-12 11:40:48

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

pytorch/pytorch#177253•Fetched 2026-04-08 00:42:42

View on GitHub

Comments

Participants

Timeline

Reactions

Author

fffrog

Participants

fffrog

mikaylagawarecki

Timeline (top)

cross-referenced ×7commented ×5labeled ×4mentioned ×4

Fix Action

Fixed

Fixed by PR: [Test] Add device type test support for op decorators and skips in DeviceTypeTestBase (https://github.com/pytorch/pytorch/pull/176264)

PR fix notes

PR #176264: [Test] Add device type test support for op decorators and skips in `DeviceTypeTestBase`

Repository: pytorch/pytorch
Author: KarhouTam
State: open | merged: False
Link: https://github.com/pytorch/pytorch/pull/176264

Description (problem / solution / changelog)

https://github.com/pytorch/pytorch/issues/174469 Fixes https://github.com/pytorch/pytorch/issues/177256

Summary

This pull request enhances the device type test infrastructure to allow device-specific decorators and skips for operator tests, and demonstrates this with a new test for the Open Registration (PrivateUse1) extension. The main improvements are the introduction of op_decorators and op_skips hooks, and a new test file that exercises these features for custom backend scenarios.

Device-specific test customization:

Added op_decorators and op_skips class attributes to DeviceTypeTestBase, enabling device-specific decorators (like expectedFailure) and skips to be applied to operator tests.
Implemented update_op_list class method to apply these decorators and skips to the relevant operator tests, handling device type mapping (including custom backends like PrivateUse1).
Integrated update_op_list into the test parametrization process to ensure operator test configurations are updated before test instantiation.

Testing and demonstration:

Added a new test file test/cpp_extensions/open_registration_extension/torch_openreg/tests/test_testing.py that demonstrates the use of op_decorators and op_skips for custom backend operator tests, including examples with dummy ops and expected failure/skipped tests.

(pytorch-dev-cuda) ➜  pytorch-cuda git:(testing/op-filter) ✗ python test/cpp_extensions/open_registration_extension/torch_openreg/tests/test_testing.py --verbose
test_normal_openreg (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_normal_openreg) ... ok
test_op_dummy_op1_openreg_bfloat16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_bfloat16) ... ok
test_op_dummy_op1_openreg_bool (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_bool) ... ok
test_op_dummy_op1_openreg_complex128 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_complex128) ... ok
test_op_dummy_op1_openreg_complex64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_complex64) ... ok
test_op_dummy_op1_openreg_float16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_float16) ... ok
test_op_dummy_op1_openreg_float32 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_float32) ... ok
test_op_dummy_op1_openreg_float64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_float64) ... ok
test_op_dummy_op1_openreg_int16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_int16) ... ok
test_op_dummy_op1_openreg_int32 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_int32) ... ok
test_op_dummy_op1_openreg_int64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_int64) ... ok
test_op_dummy_op1_openreg_int8 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_int8) ... ok
test_op_dummy_op1_openreg_uint8 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op1_openreg_uint8) ... ok
test_op_dummy_op2_openreg_bfloat16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_bfloat16) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_bool (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_bool) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_complex128 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_complex128) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_complex64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_complex64) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_float16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_float16) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_float32 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_float32) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_float64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_float64) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_int16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_int16) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_int32 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_int32) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_int64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_int64) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_int8 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_int8) ... skipped 'skip dummy_op2'
test_op_dummy_op2_openreg_uint8 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op2_openreg_uint8) ... skipped 'skip dummy_op2'
test_op_dummy_op3_openreg_bfloat16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_bfloat16) ... expected failure
test_op_dummy_op3_openreg_bool (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_bool) ... expected failure
test_op_dummy_op3_openreg_complex128 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_complex128) ... expected failure
test_op_dummy_op3_openreg_complex64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_complex64) ... expected failure
test_op_dummy_op3_openreg_float16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_float16) ... expected failure
test_op_dummy_op3_openreg_float32 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_float32) ... expected failure
test_op_dummy_op3_openreg_float64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_float64) ... expected failure
test_op_dummy_op3_openreg_int16 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_int16) ... expected failure
test_op_dummy_op3_openreg_int32 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_int32) ... expected failure
test_op_dummy_op3_openreg_int64 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_int64) ... expected failure
test_op_dummy_op3_openreg_int8 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_int8) ... expected failure
test_op_dummy_op3_openreg_uint8 (__main__.TestDeviceTypeOpenRegPRIVATEUSE1.test_op_dummy_op3_openreg_uint8) ... expected failure

----------------------------------------------------------------------
Ran 37 tests in 0.281s

OK (skipped=12, expected failures=12)

Changed files

test/cpp_extensions/open_registration_extension/torch_openreg/tests/test_testing.py (added, +50/-0)
torch/testing/_internal/common_device_type.py (modified, +32/-0)

Code Example

class DeviceTypeTestBase(TestCase):
    skipped_testcases = None

    @classmethod
    def get_skipped_testcases(cls, test_class_name):
        if cls.skipped_testcases and test_class_name in cls.skipped_testcases:
            return cls.skipped_testcases[test_class_name]
        return []

def instantiate_device_type_tests(...):
    for base in get_desired_device_type_test_bases(...):
        skipped = base.get_skipped_testcases(generic_test_class.__name__)
        if skipped and skipped[0] == "*":
            continue  
        for name in generic_tests:
            if name in skipped:
                continue
            ...

RAW_BUFFERClick to expand / collapse

A mechanism built on top of instantiate_device_type_tests and DeviceTypeTestBase to skip specific test classes or test methods.

This allows out-of-tree backends to skip tests that fail due to implementation progress or hardware limitations, without modifying any upstream test files.

class DeviceTypeTestBase(TestCase):
    skipped_testcases = None

    @classmethod
    def get_skipped_testcases(cls, test_class_name):
        if cls.skipped_testcases and test_class_name in cls.skipped_testcases:
            return cls.skipped_testcases[test_class_name]
        return []

def instantiate_device_type_tests(...):
    for base in get_desired_device_type_test_bases(...):
        skipped = base.get_skipped_testcases(generic_test_class.__name__)
        if skipped and skipped[0] == "*":
            continue  
        for name in generic_tests:
            if name in skipped:
                continue
            ...

cc @mruberry

extent analysis

Fix Plan

To implement the mechanism for skipping specific test classes or test methods, we will modify the DeviceTypeTestBase class and the instantiate_device_type_tests function.

Steps

Modify the DeviceTypeTestBase class to include a class variable skipped_testcases that stores the test cases to be skipped.
Update the get_skipped_testcases method to return the skipped test cases for a given test class name.
Modify the instantiate_device_type_tests function to check for skipped test cases and skip them accordingly.

Example Code

class DeviceTypeTestBase(TestCase):
    skipped_testcases = {
        'TestClassName': ['test_method1', 'test_method2'],
        'AnotherTestClassName': ['*']  # skip all test methods in this class
    }

    @classmethod
    def get_skipped_testcases(cls, test_class_name):
        if cls.skipped_testcases and test_class_name in cls.skipped_testcases:
            return cls.skipped_testcases[test_class_name]
        return []

def instantiate_device_type_tests(...):
    for base in get_desired_device_type_test_bases(...):
        skipped = base.get_skipped_testcases(generic_test_class.__name__)
        if skipped and skipped[0] == "*":
            # skip all test methods in this class
            continue  
        for name in generic_tests:
            if name in skipped:
                # skip this test method
                continue
            # run the test method
            ...

Verification

To verify that the fix worked, run the tests and check that the skipped test cases are indeed skipped. You can add print statements or use a debugger to verify that the skipped test cases are correctly identified and skipped.

Extra Tips

Make sure to update the skipped_testcases dictionary in the DeviceTypeTestBase class to include the test cases that need to be skipped.
Use the * wildcard to skip all test methods in a class.
You can also add more complex logic to the get_skipped_testcases method to dynamically determine which test cases to skip based on various conditions.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #autograd error #dependency conflict #environment setup #docker error #permission error #memory optimization

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

pytorch - ✅(Solved) Fix Multi-level Skip Mechanism for backends(test class or test method level) [1 pull requests, 5 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Fix Action

Fixed

PR fix notes

PR #176264: [Test] Add device type test support for op decorators and skips in `DeviceTypeTestBase`

Description (problem / solution / changelog)

Summary

Changed files

Code Example

extent analysis

Fix Plan

Steps

Example Code

Verification

Extra Tips

Still need to ship something?

TRENDING

pytorch - ✅(Solved) Fix Multi-level Skip Mechanism for backends(test class or test method level) [1 pull requests, 5 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Fix Action

Fixed

PR fix notes

PR #176264: [Test] Add device type test support for op decorators and skips in DeviceTypeTestBase

Description (problem / solution / changelog)

Summary

Changed files

Code Example

extent analysis

Fix Plan

Steps

Example Code

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING

PR #176264: [Test] Add device type test support for op decorators and skips in `DeviceTypeTestBase`