crewai - ✅(Solved) Fix Add deterministic CI check-state classifier for CrewAI PR triage [2 pull requests, 2 comments, 2 participants]

crewai2026-02-24 11:43:44

ON THIS PAGE

Recommended Tools

×6

Utilities matched from this issue’s tags and category — try them while you read without losing context.

GitHub issue graph ai analysis

Paste a GitHub issue URL. We fetch that issue, discover linked issues from bodies/comments/timeline, collect linked pull requests, and produce a structured English report.

The report is written in English Markdown for sharing and archival.

GitHub issue URL

Helpful · Quick feedback

GitHub stats

crewAIInc/crewAI#4576•Fetched 2026-04-08 00:41:10

View on GitHub

Comments

Participants

Timeline

Reactions

Author

davidahmann

Participants

davidahmann

github-actions[bot]

Timeline (top)

referenced ×6commented ×2cross-referenced ×2closed ×1

Fix Action

Fixed

Fixed by PR: feat: add deterministic CI check-state classifier for PR triage (https://github.com/crewAIInc/crewAI/pull/4577)
Fixed by PR: tooling: add deterministic CI state classifier command (https://github.com/crewAIInc/crewAI/pull/4578)

PR fix notes

PR #4577: feat: add deterministic CI check-state classifier for PR triage

Repository: crewAIInc/crewAI
Author: devin-ai-integration[bot]
State: closed | merged: False
Link: https://github.com/crewAIInc/crewAI/pull/4577

Description (problem / solution / changelog)

feat: add deterministic CI check-state classifier for PR triage

Closes #4576

Summary

Adds a standalone script (scripts/classify_ci_checks.py) that normalizes raw GitHub CI check data into a deterministic JSON output with one of five categories: passed, failed, pending, no_checks, or policy_blocked. It handles both GitHub check runs and commit statuses, retains source metadata for audit, and provides a CLI with meaningful exit codes (0/1/2). 45 unit tests cover all categories, priority ordering, edge cases, and the CLI.

Review & Testing Checklist for Human

Verify issue #4576 is a legitimate feature request the team wants — it was filed by an external user minutes before this task was assigned, with no labels or assignees. Confirm this is desired before merging.
Review the priority ordering in classify() — currently: no_checks (empty) → policy_blocked → failed → pending → passed. Confirm this precedence matches the team's expectations for triage.
Evaluate the test import pattern — tests use importlib.util.spec_from_file_location with Path(__file__).resolve().parents[3] to dynamically import from scripts/. This is fragile if the test file is ever relocated. Consider whether the script should live in a proper package instead.
Note: no .github/workflows change was made — the issue lists .github/workflows as a codepath, but no workflow was added to integrate this classifier into CI. Decide if that's needed before or after merge.
Run the tests locally: cd lib/crewai && uv run pytest tests/test_ci_check_classifier.py -v --override-ini="addopts="

Notes

The _build_summary function accepts a state parameter that it never uses — minor dead parameter.
Two # type: ignore comments in classify() handle the edge case of accepting a raw list as input.
Requested by: João
Link to Devin run

Changed files

lib/crewai/tests/test_ci_check_classifier.py (added, +443/-0)
scripts/classify_ci_checks.py (added, +293/-0)

PR #4578: tooling: add deterministic CI state classifier command

Repository: crewAIInc/crewAI
Author: davidahmann
State: open | merged: False
Link: https://github.com/crewAIInc/crewAI/pull/4578

Description (problem / solution / changelog)

Problem

PR check-state interpretation is currently manual and can misclassify sparse/no-check states.

Why Now

Cross-repo coordination requires deterministic CI state categories.

What Changed

Added scripts/ci_state_classifier.py.
Added deterministic categories: passed, failed, pending, no_checks, policy_blocked.
Added fixture/live modes and stable JSON output.
Added built-in self-test.

Validation

python3 scripts/ci_state_classifier.py --self-test ✅

Refs #4576

[!NOTE] Low Risk Adds a standalone tooling script with no runtime impact on production code; main risk is misclassification due to heuristic policy-name matching and status normalization.

Overview Adds a new scripts/ci_state_classifier.py CLI that fetches (via gh pr view) or loads a JSON statusCheckRollup payload and deterministically classifies PR check state as passed, failed, pending, no_checks, or policy_blocked.

The classifier normalizes check fields, treats known failure/pending statuses specially, and separates policy-style checks (e.g., CLA/DCO/Code Owners) so they can block without marking the PR as failed; output is stable JSON with optional pretty-printing plus a built-in --self-test.

<sup>Written by Cursor Bugbot for commit a3bb111637afd9249d8b2052c5791afa67700baa. This will update automatically on new commits. Configure here.</sup>

Changed files

scripts/ci_state_classifier.py (added, +217/-0)

RAW_BUFFERClick to expand / collapse

Problem PR check interpretation is manual and may misclassify no checks states, which slows reliable triage.

Why now Cross-repo planning/execution now depends on a normalized CI-state contract.

Current behavior is insufficient Raw check output is not machine-normalized and varies by workflow configuration.

Expected behavior Add a deterministic CI classifier for PR triage output.

Acceptance criteria

Deterministic JSON state output.
Categories include failed, pending, no checks, policy-blocked.
Source check metadata retained for audit/review.

Validation

Automated tests for all categories.
Real PR sanity-check where possible.

Codepaths

.github/workflows
lib/crewai/tests
scripts

extent analysis

Fix Plan

The fix involves implementing a deterministic CI classifier.

Steps

Create a new module to handle CI state classification:
- ci_classifier.py with functions to categorize CI states into failed, pending, no checks, and policy-blocked.
Update workflow configurations in .github/workflows to use the new classifier.
Modify lib/crewai/tests to include automated tests for all categories.
Add a script in scripts to validate the classifier using real PR data.

Example Code

# ci_classifier.py
import json

def classify_ci_state(checks):
    if not checks:
        return "no checks"
    for check in checks:
        if check["status"] == "failure":
            return "failed"
        elif check["status"] == "pending":
            return "pending"
    # Add policy-blocked logic here
    return "policy-blocked"

def get_ci_state(pr_data):
    checks = pr_data["checks"]
    state = classify_ci_state(checks)
    return json.dumps({"state": state, "checks": checks})

# Example usage
pr_data = {
    "checks": [
        {"status": "success"},
        {"status": "failure"}
    ]
}
print(get_ci_state(pr_data))  # Output: {"state": "failed", "checks": [...]}

# Update workflow configuration to use the new classifier
# .github/workflows/main.yml
steps:
  - name: Classify CI State
    run: python ci_classifier.py

# Add automated tests in lib/crewai/tests
# test_ci_classifier.py
import unittest
from ci_classifier import classify_ci_state

class TestCIClassifier(unittest.TestCase):
    def test_failed(self):
        checks = [{"status": "failure"}]
        self.assertEqual(classify_ci_state(checks), "failed")

    def test_pending(self):
        checks = [{"status": "pending"}]
        self.assertEqual(classify_ci_state(checks), "pending")

    def test_no_checks(self):
        checks = []
        self.assertEqual(classify_ci_state(checks), "no checks")

Verification

Run automated tests to ensure the classifier works correctly for all categories.
Validate the classifier using real PR data to ensure it produces the expected output.

Extra Tips

Ensure the classifier is extensible to handle new CI states or categories.
Consider adding logging to track classification errors or inconsistencies.

Vote matrix · Quick signals

Works

Did the solution work? Tap to confirm.

Easy Fix

Was it a quick fix?

Time Saver

Did it save you time?

Blocking

Was it severely blocking?

Common Issue

Are others likely hitting this too?

Flaky / Intermittent

Is it intermittent?

Verified / Reproducible

Can you reproduce it reliably?

#api #ssr #installation #tensor shape #autograd error #integration issue #index setup #retrieval issue #search optimization #API routing

Still need to ship something?

×6

Another batch ranked right after the header list — different links, same matching logic.

Data

Security

Network

Code

UI/UX

Text

System

Multimedia

Protocol

API

Engineering

crewai - ✅(Solved) Fix Add deterministic CI check-state classifier for CrewAI PR triage [2 pull requests, 2 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Fix Action

Fixed

PR fix notes

PR #4577: feat: add deterministic CI check-state classifier for PR triage

Description (problem / solution / changelog)

feat: add deterministic CI check-state classifier for PR triage

Summary

Review & Testing Checklist for Human

Notes

Changed files

PR #4578: tooling: add deterministic CI state classifier command

Description (problem / solution / changelog)

Problem

Why Now

What Changed

Validation

Changed files

extent analysis

Fix Plan

Steps

Example Code

Verification

Extra Tips

Still need to ship something?

TRENDING

crewai - ✅(Solved) Fix Add deterministic CI check-state classifier for CrewAI PR triage [2 pull requests, 2 comments, 2 participants]

Recommended Tools

GitHub issue graph ai analysis

Fix Action

Fixed

PR fix notes

PR #4577: feat: add deterministic CI check-state classifier for PR triage

Description (problem / solution / changelog)

feat: add deterministic CI check-state classifier for PR triage

Summary

Review & Testing Checklist for Human

Notes

Changed files

PR #4578: tooling: add deterministic CI state classifier command

Description (problem / solution / changelog)

Problem

Why Now

What Changed

Validation

Changed files

extent analysis

Fix Plan

Steps

Example Code

Verification

Extra Tips

Still need to ship something?

RELATED_DISCOVERY

TRENDING