vllm - 💡(How to fix) Fix [Bug]: V1 engine prefix caching was causing non-deterministic outputs during greedy decoding T=0 [1 comments, 2 participants]

TL;DR

• The difference in behavior between enable_prefix_caching=False and enable_prefix_caching=True when using the LLM model suggests that caching may be influencing the generation results.

Guidance

• Investigate the impact of enable_prefix_caching on the model's generation behavior by comparing results with caching enabled and disabled.
• Review the documentation for LLM and enable_prefix_caching to understand the intended effects of this parameter.
• Test the generation with different inputs to see if the difference in results is consistent or dependent on specific prompts.
• Consider the version of the LLM model and library being used, as behavior may change between versions.

Example

No specific code example is provided due to the nature of the issue, but comparing the results of llm.generate() with enable_prefix_caching=True and enable_prefix_caching=False is crucial.

Notes

The exact cause of the difference in generation results is not specified, and without more details about the expected behavior, it's challenging to provide a definitive solution. The influence of enable_prefix_caching on the model's performance and output needs further investigation.

Recommendation

• Apply workaround: Temporarily disable enable_prefix_caching to observe if it stabilizes or changes the generation results in a predictable manner, as this might help in understanding the caching mechanism's impact on the model's behavior.

FAIL-SAFE

Given the information, focusing on the enable_prefix_caching parameter and its effects seems the safest approach without making assumptions about the model or its intended use.