hermes - 💡(How to fix) Fix End-to-end agent smoke test in CI [1 participants]

TL;DR

Add a CI smoke test for hermes chat that installs hermes, creates a test profile, sends a heartbeat-style ping message, and verifies a non-empty response and exit code 0.

Guidance

• Identify a suitable mock or cheap provider to minimize costs during the test.
• Use the --max-turns 1 option to limit the number of turns and reduce costs.
• Create a test script that automates the steps: installing hermes from git, creating a test profile, sending a ping message, and verifying the response.
• Consider using a CI/CD framework's built-in support for running shell commands or scripts to execute the test.

Example

# Install hermes from git
git clone https://github.com/hermes/hermes.git
cd hermes
# Create a test profile
hermes profile create --name test-profile
# Send a heartbeat-style ping message with --max-turns 1
hermes chat --profile test-profile --max-turns 1 --message "ping"
# Verify non-empty response and exit code 0
if [ $? -eq 0 ] && [ -n "$(hermes chat --profile test-profile --max-turns 1 --message 'ping' | grep -v '^$')" ]; then
  echo "Test passed"
else
  echo "Test failed"
  exit 1
fi

Notes

This solution assumes that the hermes command is available and properly configured on the CI environment. The test script may need to be adapted based on the specific CI/CD framework and environment being used.

Recommendation

Apply workaround: Implement the proposed CI smoke test to ensure hermes chat functionality.