vllm - 💡(How to fix) Fix [Bug]: Multi-LoRA Initialization Failure on Cloud TPU v6e-4 (AssertionError: LoRA is not enabled) [1 comments, 1 participants]

(EngineCore_DP0 pid=305) ERROR 04-02 07:26:44 [core.py:866] AssertionError: LoRA is not enabled (EngineCore_DP0 pid=305) Traceback (most recent call last): (EngineCore_DP0 pid=305) File "/workspace/tpu_inference/tpu_inference/worker/tpu_worker.py", line 369, in compile_or_warm_up_model (EngineCore_DP0 pid=305) self.model_runner.capture_model() (EngineCore_DP0 pid=305) File "/workspace/tpu_inference/tpu_inference/runner/tpu_runner.py", line 556, in capture_model (EngineCore_DP0 pid=305) self.compilation_manager.capture_model() (EngineCore_DP0 pid=305) File "/workspace/tpu_inference/tpu_inference/runner/compilation_manager.py", line 92, in capture_model (EngineCore_DP0 pid=305) with self.runner.maybe_setup_dummy_loras(self.runner.lora_config): (EngineCore_DP0 pid=305) File "/workspace/vllm/vllm/v1/worker/lora_model_runner_mixin.py", line 97, in maybe_setup_dummy_loras (EngineCore_DP0 pid=305) assert self.lora_manager is not None, "LoRA is not enabled" (EngineCore_DP0 pid=305) AssertionError: LoRA is not enabled

TL;DR

The most likely fix is to verify the LoRA configuration and ensure it is correctly enabled and configured for the TPU JAX worker.

Guidance

• Review the --enable-lora flag and its associated options (--max-loras, --max-lora-rank, --lora-modules) to ensure they are correctly set and compatible with the TPU JAX worker.
• Check the lora_model_runner_mixin.py file to understand the maybe_setup_dummy_loras method and the lora_manager attribute, as the error occurs in this context.
• Verify that the vllm/vllm-tpu:latest image (v0.13.0) supports Multi-LoRA on Cloud TPU v6e-4 and that there are no known issues or limitations.
• Test the model serving with a simpler LoRA configuration to isolate the issue and determine if it's specific to the current setup.

Example

No code snippet is provided as the issue is more related to configuration and compatibility rather than a specific code error.

Notes

The provided error stack and logs suggest a configuration or compatibility issue rather than a code bug. The AssertionError: LoRA is not enabled error indicates that the LoRA configuration is not being recognized or applied correctly by the TPU JAX worker.

Recommendation

Apply a workaround by simplifying the LoRA configuration and testing it with a smaller model or a different TPU setup to isolate the issue and determine the root cause. This approach can help identify if the problem is specific to the current configuration or a more general compatibility issue.