Issue Station
Sign In
Tools hereToolbox

crewai - ✅(Solved) Fix [BUG]: HITL pre-review fails open and silently bypasses automated safeguards [1 pull requests, 2 comments, 3 participants]

Official PRs (…)
ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

Helpful · Quick feedback

Loading…
GitHub stats
crewAIInc/crewAI#5725•Fetched 2026-05-07 03:39:56
View on GitHub
Comments
2
Participants
3
Timeline
4
Reactions
0
Author
hello-argshello-args
Participants
hanselhanselhanselhanselhello-argshello-argsJairoohJairooh
Timeline (top)
commented ×2cross-referenced ×1labeled ×1

When @human_feedback(..., learn=True) is used and lessons are recalled, _pre_review_with_lessons attempts to apply those lessons via an LLM before presenting output for human feedback.

If any exception occurs in this path (for example network or auth failure, LLM error, or structured-output issues), the implementation catches all exceptions and returns the original method_output.

As a result:

  • No exception is propagated
  • No warning or error is logged (in this except block)
  • The flow continues normally
  • Callers cannot distinguish between:
    • pre-reviewed output
    • raw model output

This creates a silent fallback: the pre-review step may not have executed, but downstream components are not informed.

Error Message

except Exception: return method_output # fallback to raw output on any failure

Root Cause

When @human_feedback(..., learn=True) is used and lessons are recalled, _pre_review_with_lessons attempts to apply those lessons via an LLM before presenting output for human feedback.

If any exception occurs in this path (for example network or auth failure, LLM error, or structured-output issues), the implementation catches all exceptions and returns the original method_output.

As a result:

  • No exception is propagated
  • No warning or error is logged (in this except block)
  • The flow continues normally
  • Callers cannot distinguish between:
    • pre-reviewed output
    • raw model output

This creates a silent fallback: the pre-review step may not have executed, but downstream components are not informed.

Fix Action

Fix / Workaround

  1. Define a Flow with memory enabled and a method decorated with @human_feedback(learn=True, ...).
  2. Ensure at least one lesson exists in memory (for example run once with learn=True to store feedback).
  3. On a subsequent run, force the pre-review LLM call to fail, for example by patching the LLM call used inside _pre_review_with_lessons to raise: 
    RuntimeError("simulated pre-review failure")
  4. Run kickoff() or kickoff_async().
  5. Observe:
    • No exception is raised
    • The value passed to human feedback is the raw method_output
    • No visible signal that pre-review failed

PR fix notes

PR #5726: fix(hitl): log pre-review failures and add learn_strict mode (closes #5725)

Description (problem / solution / changelog)

Summary

Closes #5725.

@human_feedback(..., learn=True) runs _pre_review_with_lessons to apply past HITL lessons via an LLM before the human sees the output. Today, that helper wraps the entire pre-review path in a broad except Exception: return method_output, which swallows LLM/network/auth/structured-output errors with no log, no event, and no return signal — callers cannot distinguish a pre-reviewed output from a raw one.

This PR makes the failure observable by default, and adds opt-in fail-closed behavior:

lib/crewai/src/crewai/flow/human_feedback.py

  • Add a module logger (logging.getLogger(__name__)).
  • Narrow the try/except in _pre_review_with_lessons so the mem is None and not matches short-circuits stay outside the failure path (those are not errors).
  • On memory.recall failure or pre-review LLM failure, log WARNING with exc_info=True so the silent fallback is detectable. By default the flow still falls back to the raw method_output and continues — we don't change the default fail-open contract.
  • Add learn_strict: bool = False to the human_feedback decorator and HumanFeedbackConfig. When learn_strict=True, recall/pre-review failures are re-raised instead of falling back, for callers that need fail-closed behavior.

docs/en/learn/human-feedback-in-flows.mdx

  • Update the "Graceful degradation" / Configuration section to document that pre-review failures are now logged at WARNING with traceback and to introduce the learn_strict parameter.

lib/crewai/tests/test_human_feedback_decorator.py

  • New TestHumanFeedbackPreReviewFailure class with 7 tests:
    • sync pre-review LLM failure → WARNING logged with exc_info, raw output shown to human;
    • sync memory.recall failure → WARNING logged with exc_info, raw output shown to human;
    • sync learn_strict=True → pre-review LLM RuntimeError propagates;
    • sync learn_strict=True → recall RuntimeError propagates;
    • async pre-review LLM failure → WARNING logged with exc_info, raw output shown to human;
    • async learn_strict=True → pre-review LLM RuntimeError propagates;
    • learn_strict defaults to False and round-trips through HumanFeedbackConfig.

Behavior outside the pre-review step (distillation, outcome collapse, resume emit fallback, listener failures) is intentionally untouched — issue #5725 is scoped only to pre-review.

All 40 tests in test_human_feedback_decorator.py pass locally. ruff check, ruff format --check, and mypy on the modified file are clean.

Review & Testing Checklist for Human

  • Confirm the new learn_strict decorator argument name matches your preferred public API surface (the issue suggested it as one option; alternatives like pre_review_strict or a separate strict= flag would be one-line renames).
  • Confirm fail-open + WARNING is the right default. The PR keeps the documented "Graceful degradation" contract but makes it observable. If you want fail-closed by default, learn_strict would just need to be flipped (and one test inverted).
  • Sanity-check the WARNING log strings ("HITL pre-review failed for %s; falling back to raw output." and "HITL pre-review: memory recall failed for %s; falling back to raw output.") — these will show up in user logs.
  • Spot-check the docs update at docs/en/learn/human-feedback-in-flows.mdx — only the English doc was updated; the pt-BR/ko/ar mirrors still describe silent degradation. Happy to follow up with translations if you'd like them in this PR.

Notes

  • Repro from the issue (@human_feedback(learn=True) + seeded memory + LLM raising during pre-review) now produces a single WARNING under the crewai.flow.human_feedback logger with full traceback, instead of silent passthrough.
  • The change is intentionally minimal and scoped to _pre_review_with_lessons. Distillation's existing except Exception: pass remains untouched per the issue's "out of scope" note, but it would be a natural follow-up to give it the same WARNING-on-failure treatment.

Link to Devin session: https://app.devin.ai/sessions/8615574d124449a5a1b09ea4d1fd88bd

Changed files

  • docs/en/learn/human-feedback-in-flows.mdx (modified, +3/-1)
  • lib/crewai/src/crewai/flow/human_feedback.py (modified, +59/-8)
  • lib/crewai/tests/test_human_feedback_decorator.py (modified, +272/-0)

Code Example

RuntimeError("simulated pre-review failure")

---

except Exception:
    return method_output  # fallback to raw output on any failure

---

if mem is None:
    return method_output

if not matches:
    return method_output
RAW_BUFFERClick to expand / collapse

Description

When @human_feedback(..., learn=True) is used and lessons are recalled, _pre_review_with_lessons attempts to apply those lessons via an LLM before presenting output for human feedback.

If any exception occurs in this path (for example network or auth failure, LLM error, or structured-output issues), the implementation catches all exceptions and returns the original method_output.

As a result:

  • No exception is propagated
  • No warning or error is logged (in this except block)
  • The flow continues normally
  • Callers cannot distinguish between:
    • pre-reviewed output
    • raw model output

This creates a silent fallback: the pre-review step may not have executed, but downstream components are not informed.

Steps to Reproduce

  1. Define a Flow with memory enabled and a method decorated with @human_feedback(learn=True, ...).
  2. Ensure at least one lesson exists in memory (for example run once with learn=True to store feedback).
  3. On a subsequent run, force the pre-review LLM call to fail, for example by patching the LLM call used inside _pre_review_with_lessons to raise: 
    RuntimeError("simulated pre-review failure")
  4. Run kickoff() or kickoff_async().
  5. Observe:
    • No exception is raised
    • The value passed to human feedback is the raw method_output
    • No visible signal that pre-review failed

Expected behavior

If pre-review fails, the system should not silently behave the same as a successful pre-review.

Any of the following would improve correctness and observability:

  • Fail closed: propagate the exception
  • Explicit degradation:
    • log at warning or error with exc_info=True
    • emit a structured signal or event
    • or return metadata such as pre_review_applied=False

Screenshots/Code snippets

Failure path (lib/crewai/src/crewai/flow/human_feedback.py):

except Exception:
    return method_output  # fallback to raw output on any failure

Other paths that return raw output without LLM pre-review:

if mem is None:
    return method_output

if not matches:
    return method_output

Docs reference

  • docs/en/learn/human-feedback-in-flows.mdx — section “Graceful degradation” describes falling back to raw output without blocking the flow.

Operating System

macOS Big Sur

Python Version

3.12

crewAI Version

Latest main (as of May 2026)

crewAI Tools Version

Latest / N/A

Virtual Environment

Venv

Evidence

  • Pre-review failures do not raise exceptions
  • No logging at warning or error level in this path
  • Returned output is indistinguishable from raw method_output
  • Downstream systems cannot detect whether pre-review actually ran

Possible Solution

  • Replace broad except Exception: with narrower handling where possible
  • For unexpected failures:
    • log with logger.warning(..., exc_info=True) or logger.error(..., exc_info=True)
    • optionally re-raise after logging
  • Consider returning structured metadata (for example pre_review_applied=False)
  • Optional strict mode (for example learn_strict=True) to fail on pre-review errors instead of silently substituting raw output

Additional context

  • Scoped only to the learn=True pre-review step.
  • The human feedback step still runs afterward in normal use.
  • The concern is that the automated pre-review layer may silently not run, with no signal to the caller.
  • Other behaviors (outcome collapse, resume emit fallback, listener failures) are out of scope for this issue.

Vote matrix · Quick signals

Works
Did the solution work? Tap to confirm.
Easy Fix
Was it a quick fix?
Time Saver
Did it save you time?
Blocking
Was it severely blocking?
Common Issue
Are others likely hitting this too?
Flaky / Intermittent
Is it intermittent?
Verified / Reproducible
Can you reproduce it reliably?
Loading…

FAQ

Expected behavior

If pre-review fails, the system should not silently behave the same as a successful pre-review.

Any of the following would improve correctness and observability:

  • Fail closed: propagate the exception
  • Explicit degradation:
    • log at warning or error with exc_info=True
    • emit a structured signal or event
    • or return metadata such as pre_review_applied=False
#api#ssr#installation#tensor shape#request timeout

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Back to top recommendations

TRENDING