When a context overflow error occurs (model_context_window_exceeded), the error is logged but the UI (webchat) shows either an infinite spinner or no output at all, instead of displaying an error message to the user. 3. UI shows infinite spinner or nothing, no error is displayed When context overflow occurs, the user should see a clear error message such as:

• Error is logged server-side but not surfaced to the user When the model hits a context overflow error, it produces 0 output payloads. The UI has nothing to render, so it just shows a spinner indefinitely. When payloads=0 and isError=true, the gateway should generate a synthetic error payload (e.g., a text message containing the error description) so the UI has something to render and display to the user.

TL;DR

Implement a synthetic error payload generation in the gateway when payloads=0 and an error occurs, to display an error message to the user.

Guidance

• Check the gateway logic to handle cases where payloads=0 and isError=true, to ensure it generates a synthetic error payload.
• Verify that the UI can correctly render and display the synthetic error payload as an error message to the user.
• Review the error logging mechanism to ensure it provides sufficient information for debugging and monitoring purposes.
• Consider adding a timeout or fallback mechanism to handle cases where the UI is stuck in an infinite loading state.

Example

if payloads == 0 and isError:
    synthetic_error_payload = {
        "text": "Context overflow: prompt too large for the model. Try /reset to start a fresh session."
    }
    # Return the synthetic error payload to the UI for rendering

Notes

This solution assumes that the gateway has the capability to generate and return synthetic payloads. Additional modifications may be required to handle edge cases or specific UI requirements.

Recommendation

Apply workaround: Implement synthetic error payload generation in the gateway to display error messages to the user, as this directly addresses the issue of the UI showing an infinite spinner or no output when a context overflow error occurs.