pytorch - 💡(How to fix) Fix [vllm] [triton 3.7] PassManager::run failed in make_ttgir when compiling vLLM _penalties_kernel

File ".../vllm/v1/worker/gpu/sample/penalties.py", line 207, in apply_penalties _penalties_kernel(num_tokens, num_blocks) File ".../triton/runtime/jit.py", line 739, in run kernel = self._do_compile(key, signature, device, constexprs, options, attrs, warmup) File ".../triton/runtime/jit.py", line 884, in _do_compile kernel = self.compile(src, target=target, options=options.dict) File ".../triton/compiler/compiler.py", line 327, in compile next_module = compile_ir(module, metadata) File ".../triton/backends/nvidia/compiler.py", line 549, in <lambda> stages["ttgir"] = lambda src, metadata: self.make_ttgir(src, metadata, options, capability) File ".../triton/backends/nvidia/compiler.py", line 321, in make_ttgir pm.run(mod, 'make_ttgir') RuntimeError: PassManager::run failed

TL;DR

Downgrade triton to version 3.6.0 to potentially resolve the compilation failure of the _penalties_kernel in vLLM.

Guidance

• Verify that the issue is indeed caused by the triton version by testing with triton 3.6.0, as the same kernel compiles successfully with this version.
• Check the TRITON_* environment variables to see if there are any options to increase the verbosity of the PassManager to get more diagnostic information about the failing MLIR pass.
• Extract the offending TTIR dump using the correct TRITON_* env var to further investigate the issue.
• Review the changes between triton 3.6.0 and 3.7.0 to identify potential breaking changes that could be causing the compilation failure.

Example

No code snippet is provided as the issue is related to a specific version of triton and its interaction with the vLLM codebase.

Notes

The root cause of the issue is unclear, but it seems to be related to the make_ttgir pass in triton 3.7.0. Downgrading to triton 3.6.0 may be a temporary workaround, but a more permanent solution would require further investigation into the changes between the two versions.

Recommendation

Apply the workaround by downgrading triton to version 3.6.0, as this has been shown to resolve the compilation failure in the past. This will allow the vLLM project to proceed with the torch 2.12 upgrade while a more permanent solution is investigated.