pytorch - 💡(How to fix) Fix [vllm] [triton 3.7] AOTAutogradCache.save: _pickle.PicklingError on triton launcher function

File ".../torch/_functorch/_aot_autograd/graph_compile.py", line 466, in aot_stage2_inference entry = _cache_inference_info( File ".../torch/_functorch/_aot_autograd/graph_compile.py", line 525, in _cache_inference_info AOTAutogradCache.save(...) File ".../torch/_functorch/_aot_autograd/autograd_cache.py", line 1257, in save AOTAutogradCache._handle_save_error(e, remote, is_bypass=False) File ".../torch/_functorch/_aot_autograd/autograd_cache.py", line 1229, in _handle_save_error raise e File ".../torch/_functorch/_aot_autograd/autograd_cache.py", line 1237, in save content = AOTAutogradCache._pickle_entry(entry, remote) File ".../torch/_functorch/_aot_autograd/autograd_cache.py", line 1195, in _pickle_entry return pickle.dumps(entry) _pickle.PicklingError: Can't pickle <function launcher at 0x7f3605c8f880>: attribute lookup launcher on main failed

TL;DR

The most likely fix is to modify AOTAutograd to treat pickling errors inside AOT cache save as a bypass for triton-produced artifacts.

Guidance

• Investigate modifying AOTAutograd to catch and handle _pickle.PicklingError exceptions specifically for triton-generated launchers, allowing the compilation to continue with a warning.
• Consider adding a check for the type of error and the origin of the launcher function to ensure that only triton-produced artifacts are bypassed.
• Review the ExecutableMetadata and CompiledKernel classes in triton to see if making them pickleable via __reduce__ is a viable alternative solution.
• Verify that the proposed fix does not introduce any security vulnerabilities related to pickling and unpickling of arbitrary functions.

Example

try:
    content = AOTAutogradCache._pickle_entry(entry, remote)
except _pickle.PicklingError as e:
    if "attribute lookup launcher on __main__ failed" in str(e):
        # Handle triton-generated launcher pickling error as a bypass
        warnings.warn("Pickling error occurred for triton-generated launcher. Continuing with compilation.")
        # Continue with compilation or provide an alternative solution
    else:
        raise

Notes

The provided solution focuses on handling the pickling error as a bypass within AOTAutograd. However, it's essential to ensure that this fix does not compromise the security or functionality of the system. Further investigation into making triton-generated launchers pickleable or finding an alternative serialization method might be necessary for a more robust solution.

Recommendation

Apply the workaround by modifying AOTAutograd to treat pickling errors for triton-produced artifacts as a bypass, as it seems to be the most direct approach to resolve the immediate issue blocking the torch 2.12 upgrade.