pytorch - 💡(How to fix) Fix [Windows] test_meta_device failed with CPP wrapper on windows [2 comments, 2 participants]

proTests.test_meta_device __________________________________________________________________________ Traceback (most recent call last): File "C:\Users\yucheng3.conda\envs\yucheng\lib\unittest\case.py", line 59, in testPartExecutor yield File "C:\Users\yucheng3.conda\envs\yucheng\lib\unittest\case.py", line 591, in run self.callTestMethod(testMethod) File "C:\Users\yucheng3.conda\envs\yucheng\lib\unittest\case.py", line 549, in callTestMethod method() File "D:\yucheng\pytorch\torch\testing_internal\common_utils.py", line 3444, in wrapper method(*args, **kwargs) File "D:\yucheng\pytorch\test\inductor\test_cpu_repro.py", line 2595, in test_meta_device self.assertEqual(fn().shape, [1024, 128, 128]) File "D:\yucheng\pytorch_\torch_dynamo\eval_frame.py", line 1055, in compile_wrapper raise e.remove_dynamo_frames() from None # see TORCHDYNAMO_VERBOSE=1 File "D:\yucheng\pytorch_\torch_inductor\compile_fx.py", line 1069, in compile_fx_inner raise InductorError(e, currentframe()).with_traceback( File "D:\yucheng\pytorch\torch_inductor\compile_fx.py", line 1049, in compile_fx_inner mb_compiled_graph = fx_codegen_and_compile( File "D:\yucheng\pytorch\torch_inductor\compile_fx.py", line 1832, in fx_codegen_and_compile return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs) File "D:\yucheng\pytorch_\torch_inductor\compile_fx.py", line 1593, in codegen_and_compile compiled_module = graph.compile_to_module() File "D:\yucheng\pytorch_\torch_inductor\graph.py", line 2612, in compile_to_module return self.compile_to_module() File "D:\yucheng\pytorch\torch_inductor\graph.py", line 2622, in compile_to_module mod = self.compile_to_module_lines(wrapper_code) File "D:\yucheng\pytorch\torch_inductor\graph.py", line 2697, in compile_to_module_lines mod = PyCodeCache.load_by_key_path( File "D:\yucheng\pytorch\torch_inductor\codecache.py", line 3961, in load_by_key_path mod = reload_python_module(key, path, set_sys_modules=in_toplevel) File "D:\yucheng\pytorch\torch_inductor\runtime\compile_tasks.py", line 35, in reload_python_module exec(code, mod.dict, mod.dict) File "C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo\ck\ccknyp3ri5ay4ljhnk2kc3s7fgywgyp4rkaam2z5fqieq4vg6pmv.py", line 79, in <module> File "D:\yucheng\pytorch\torch_inductor\codecache.py", line 3438, in load_pybinding return cls.load_pybinding_async(*args, **kwargs)() File "D:\yucheng\pytorch\torch_inductor\codecache.py", line 3430, in future result = get_result() File "D:\yucheng\pytorch_\torch_inductor\codecache.py", line 3217, in load_fn result = worker_fn() File "D:\yucheng\pytorch_\torch_inductor\codecache.py", line 3246, in worker_compile_cpp builder.build() File "D:\yucheng\pytorch\torch_inductor\cpp_builder.py", line 2261, in build run_compile_cmd(build_cmd, cwd=build_tmp_dir) File "D:\yucheng\pytorch\torch_inductor\cpp_builder.py", line 688, in run_compile_cmd run_compile_cmd(cmd_line, cwd) File "D:\yucheng\pytorch\torch_inductor\cpp_builder.py", line 683, in _run_compile_cmd raise exc.CppCompileError(cmd, output) from e torch._inductor.exc.InductorError: CppCompileError: C++ compile error

Command: cl /I C:/Users/yucheng3/.conda/envs/yucheng/Include /I D:/yucheng/pytorch_/torch/include /I D:/yucheng/pytorch_/torch/include/torch/csrc/api/include /D NOMINMAX /D TORCH_INDUCTOR_CPP_WRAPPER /D STANDALONE_TORCH_HEADER /D TORCH_INDUCTOR_PRECOMPILE_HEADERS /D C10_USING_CUSTOM_GENERATED_MACROS /D CPU_CAPABILITY_AVX512 /O2 /DLL /MD /std:c++20 /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /wd4624 /wd4067 /wd4068 /EHsc /Zc:_cplusplus /permissive- /openmp /openmp:experimental C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp /arch:AVX512 /FeC:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.pyd /LD /link /LIBPATH:C:/Users/yucheng3/.conda/envs/yucheng/libs /LIBPATH:D:/yucheng/pytorch/torch/lib torch.lib torch_cpu.lib c10.lib torch_python.lib sleef.lib

Output: Microsoft (R) C/C++ Optimizing Compiler Version 19.42.34435 for x64 Copyright (C) Microsoft Corporation. All rights reserved.

cl : Command line warning D9025 : overriding '/openmp' with '/openmp:experimental' c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(2): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(2): error C2065: 'float16': undeclared identifier C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(3): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(3): error C2065: 'meta': undeclared identifier C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(5): error C2182: 'inductor_entry_impl': this use of 'void' is not valid C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(6): error C2065: 'AtenTensorHandle': undeclared identifier
C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(7): error C2065: 'input_handles': undeclared identifier C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(9): error C2065: 'AtenTensorHandle': undeclared identifier
C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(10): error C2065: 'output_handles': undeclared identifier
C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(13): error C2143: syntax error: missing ';' before '{' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(14): error C2653: 'py': is not a class or namespace name C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(14): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(14): error C2146: syntax error: missing ';' before identifier 'release' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(16): error C2065: 'input_handles': undeclared identifier C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(16): error C3861: 'steal_from_raw_handles_to_raii_handles': identifier not found C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(18): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(18): error C2146: syntax error: missing ';' before identifier 'custom_op_wrapper' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(19): error C2447: '{': missing function header (old-style formal list?) C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(31): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(31): error C2146: syntax error: missing ';' before identifier 'buf1' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(32): error C2447: '{': missing function header (old-style formal list?) C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(68): error C2466: cannot allocate an array of constant size 0
C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(68): error C4430: missing type specifier - int assumed. Note: C++ does not support default-int C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(68): error C2065: 'buf1': undeclared identifier C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(69): error C2059: syntax error: '}' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(69): error C2143: syntax error: missing ';' before '}' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\sal.h(2361): error C2143: syntax error: missing ';' before '{' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\sal.h(2361): error C2447: '{': missing function header (old-style formal list?) C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(108): error C2039: 'vector': is not a member of 'std' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\complex(62): note: see declaration of 'std' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(108): error C7568: argument list missing after assumed function template 'vector' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(108): error C2062: type 'unknown-type' unexpected C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(108): error C2143: syntax error: missing ';' before '{' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(108): error C2447: '{': missing function header (old-style formal list?) C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(124): error C2676: binary '[': 'const std::array<AtenTensorHandle,1>' does not define this operator or a conversion to a type acceptable to the predefined operator C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(127): error C2676: binary '[': 'const std::array<AtenTensorHandle,1>' does not define this operator or a conversion to a type acceptable to the predefined operator C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C2039: 'vector': is not a member of 'std' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\complex(62): note: see declaration of 'std' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C7568: argument list missing after assumed function template 'vector' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C2062: type 'unknown-type' unexpected C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C2039: 'vector': is not a member of 'std' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\complex(62): note: see declaration of 'std' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C2275: 'AtenTensorHandle': expected an expression instead of a type C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C2143: syntax error: missing ';' before '{' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(133): error C2447: '{': missing function header (old-style formal list?) C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C2039: 'vector': is not a member of 'std' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\complex(62): note: see declaration of 'std' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C7568: argument list missing after assumed function template 'vector' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C2988: unrecognizable template declaration/definitionC:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C2143: syntax error: missing ';' before '<parameter-list>' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C2059: syntax error: '<parameter-list>' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C2143: syntax error: missing ';' before '{' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(137): error C2447: '{': missing function header (old-style formal list?) C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(163): error C2039: 'vector': is not a member of 'std' C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.42.34433\include\complex(62): note: see declaration of 'std' C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(163): error C2275: 'AtenTensorHandle': expected an expression instead of a type C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(163): error C2563: mismatch in formal parameter list C:/Users/yucheng3/AppData/Local/Temp/3/tmpfx2rc2vo/57/c57vi5ro5t7jna6rdezfse3ev5h7m6ozf3y4i2o3nlshsot6ylkr.main.cpp(163): error C2064: term does not evaluate to a function taking 1 arguments

Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"

To execute this test, run the following from the base repo dir: python test\inductor\test_cpu_repro.py CPUReproTests.test_meta_device

TL;DR

The issue is likely due to a C++ compilation error caused by missing type specifiers, undeclared identifiers, and syntax errors in the generated C++ code.

Guidance

1. Check the C++ code generation: Verify that the C++ code generated by PyTorch is correct and does not contain any syntax errors.
2. Verify the compiler flags: Ensure that the compiler flags used to compile the C++ code are correct and compatible with the PyTorch version.
3. Update PyTorch version: Consider updating PyTorch to a newer version, as the issue might be resolved in a later version.
4. Check the environment variables: Verify that the environment variables, such as TORCHINDUCTOR_CPP_WRAPPER, are set correctly.

Example

No code snippet is provided as the issue is related to C++ compilation errors and not a specific code snippet.

Notes

The issue is likely specific to the PyTorch version and the environment used to compile the C++ code. Further investigation is required to determine the root cause of the issue.

Recommendation

Apply a workaround by setting TORCHINDUCTOR_CPP_WRAPPER=0 to disable the C++ wrapper and see if the issue persists. If the issue is resolved, it may indicate a problem with the C++ code generation or compilation.