pytorch - 💡(How to fix) Fix DISABLED test_quiesce_repeatedly (__main__.TestCompileWorker) [1 comments, 1 participants]

Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/inductor/test_compile_worker.py", line 91, in test_quiesce_repeatedly pool.shutdown() ~~~~~~~~~~~~~^^ File "/opt/conda/envs/py_3.14/lib/python3.14/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 342, in shutdown self.process.wait(300) ~~~~~~~~~~~~~~~~~^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14/subprocess.py", line 1278, in wait return self._wait(timeout=timeout) ~~~~~~~~~~^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/py_3.14/lib/python3.14/subprocess.py", line 2075, in _wait raise TimeoutExpired(self.args, timeout) subprocess.TimeoutExpired: Command '['/opt/conda/envs/py_3.14/bin/python', '/opt/conda/envs/py_3.14/lib/python3.14/site-packages/torch/_inductor/compile_worker/main.py', '--pickler=torch._inductor.compile_worker.subproc_pool.SubprocPickler', '--kind=fork', '--workers=2', '--parent=83459', '--read-fd=13', '--write-fd=18', '--torch-key=pL6Xe0JkLeLgHAc13TNbCRT+Wl8bnHXEGU8vNIVSLaM=']' timed out after 300 seconds

Fix Plan

Increase Timeout for pool.shutdown()

The root cause of the issue is a timeout error when shutting down the process pool. To fix this, we need to increase the timeout value.

Code Changes

1. Update the test_quiesce_repeatedly function in inductor/test_compile_worker.py to increase the timeout value:

   def test_quiesce_repeatedly(self):
       # ...
       pool.shutdown(timeout=600)  # Increase timeout to 10 minutes
       # ...

2. Alternatively, you can also increase the timeout value globally for the subproc_pool module:

   import torch._inductor.compile_worker.subproc_pool
   
   torch._inductor.compile_worker.subproc_pool.SubprocPool.timeout = 600  # Increase timeout to 10 minutes

Verification

1. Run the test_quiesce_repeatedly test again in CI to verify that the fix works.
2. Check the logs to ensure that the test completes successfully without any timeout errors.

Extra Tips

• Consider adding more logging statements to the test_quiesce_repeatedly function to help diagnose any future issues.
• If the test still fails after increasing the timeout value, you may need to investigate further to determine the root cause of the issue.