vllm - 💡(How to fix) Fix [Bug]: ImportError: cannot import name 'GenerationConfig' from 'transformers' [2 comments, 3 participants]

vllm-qwen35 | Traceback (most recent call last): vllm-qwen35 | File "/usr/local/bin/vllm", line 10, in <module> vllm-qwen35 | sys.exit(main()) vllm-qwen35 | ^^^^^^ vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 87, in main vllm-qwen35 | import vllm.entrypoints.cli.benchmark.main vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/benchmark/main.py", line 10, in <module> vllm-qwen35 | from vllm.entrypoints.utils import VLLM_SUBCMD_PARSER_EPILOG vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/utils.py", line 19, in <module> vllm-qwen35 | from vllm.engine.arg_utils import EngineArgs vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 35, in <module> vllm-qwen35 | from vllm.config import ( vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/config/init.py", line 20, in <module> vllm-qwen35 | from vllm.config.model import ( vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/config/model.py", line 30, in <module> vllm-qwen35 | from vllm.transformers_utils.config import ( vllm-qwen35 | File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/config.py", line 18, in <module> vllm-qwen35 | from transformers import GenerationConfig, PretrainedConfig vllm-qwen35 | ImportError: cannot import name 'GenerationConfig' from 'transformers' (/usr/local/lib/python3.12/dist-packages/transformers/init.py) vllm-qwen35 exited with code 1

TL;DR

The issue is likely due to an incompatible version of the transformers library, which is missing the GenerationConfig class.

Guidance

• The error message indicates that the GenerationConfig class is not found in the transformers library, suggesting a version mismatch.
• The fact that it works with an older version of the container (vllm/vllm-openai:cu130-nightly-55e1a8e1035bddb0b5b63f9ddecc8b4e16fc3ef6) implies that the issue is related to a change in the transformers library.
• To mitigate the issue, you can try downgrading the transformers library to a version that is compatible with the vllm library.
• You can verify the version of the transformers library being used by checking the container's dependencies or by running a command like pip show transformers inside the container.

Notes

The exact version of the transformers library that is compatible with the vllm library is not specified, so some trial and error may be required to find the correct version.

Recommendation

Apply workaround: Downgrade the transformers library to a version that is known to work with the vllm library, such as the version used in the working container (vllm/vllm-openai:cu130-nightly-55e1a8e1035bddb0b5b63f9ddecc8b4e16fc3ef6).