openclaw - 💡(How to fix) Fix Compaction max_tokens exceeds model output limit, crashes gateway [1 comments, 2 participants]

When compaction is triggered on long conversations, the gateway calculates max_tokens = Math.ceil(targetLength / 2), which can produce values far exceeding the model's maximum output token limit. This causes the API to return a 400 error and crashes the gateway.

TL;DR

Capping max_tokens at the model's maximum output token limit is likely to fix the gateway crash issue.

Guidance

• Verify that the max_tokens calculation is the root cause by checking the API error message for the max_tokens limit exceeded error.
• Update the max_tokens calculation to use Math.min to cap the value at the model's maximum output token limit, as shown in the expected behavior code snippet.
• Test the updated calculation with different conversation lengths and model output limits to ensure the fix works as expected.
• Consider adding a check to handle cases where the conversation length is extremely large, to prevent potential issues with the Math.ceil calculation.

Example

max_tokens = Math.min(Math.ceil(targetLength / 2), modelMaxOutputTokens)

Notes

This fix assumes that the modelMaxOutputTokens value is correctly set for each model. If this value is not available or is incorrect, additional work may be needed to retrieve or calculate the correct maximum output token limit.

Recommendation

Apply the workaround by updating the max_tokens calculation to cap the value at the model's maximum output token limit, as this directly addresses the root cause of the issue and prevents the gateway crash.