hermes - 💡(How to fix) Fix [Feature]: Enable Anthropic prompt caching for Claude models on Nous Portal [1 participants]

TL;DR

Enable prompt caching for Claude models on Nous Portal by exposing the Anthropic wire protocol or forwarding cache control markers.

Guidance

• Expose the https://inference-api.nousresearch.com/v1/anthropic endpoint using the native Anthropic wire protocol to leverage Hermes' auto-detection feature.
• Alternatively, confirm that cache control markers are forwarded to Anthropic on the existing endpoint and add a caching policy for the Nous provider.
• Verify that the caching infrastructure is fully implemented in Hermes and that PR #12846 has prepared the client-side code for caching.
• Test the caching functionality with multi-turn conversations to ensure significant unnecessary token spend is reduced for Nous subscribers.

Example

No code snippet is provided as the issue does not require a specific code change, but rather a configuration or exposure of an endpoint.

Notes

The proposed solution assumes that exposing the Anthropic wire protocol or forwarding cache control markers will enable prompt caching without requiring additional changes to the client-side code.

Recommendation

Apply the workaround by exposing the Anthropic wire protocol, as it seems to be the most straightforward solution that leverages existing infrastructure and auto-detection features.