Open source · MIT licensed

Ship agents with
evidence, not hope.

AgentEval is a CI evaluation harness for LLM agents. Test answers, tool choices, trajectories, latency, cost, and flakiness before a regression reaches production.

Start evaluating View on GitHub

pip install nishanttyagi-agenteval

agent-eval.yml · run #184

passed

→

✓

✓ 8.42s

Correctness0%

Tool accuracy0%

Trajectory match0%

✓Regression gate

PASS

Built for modern agent stacks

CrewAILangGraph-readyCustom agentsAny LLM

The problem

Agents fail differently
from regular software.

A prompt, model, or tool change can silently improve one answer while breaking another. Traditional unit tests cannot see quality drift, fabricated claims, or the wrong execution path.

Without AgentEvalUnverified

Spot-check outputs by hand
Miss flaky tool routing
Discover regressions in production
Guess at latency and spend

With AgentEvalCI protected

Versioned YAML golden cases
Deterministic-first scoring
Reviewable regression gates
Reproducible run evidence

01DefineGolden cases

02RunYour agent

03ScoreFive metrics

04GateShip or block

Evaluation primitives

Everything an agent CI gate needs.

Focused evidence for every layer between a user prompt and an agent’s final answer.

Framework adapters

CrewAI built in. LangGraph-ready.

Normalize CrewAI crews today, or connect LangGraph and custom orchestrators through one compact AgentAdapter contract.

CrewAI LangGraph +Your agent

Execution evidence

Trajectory scoring

Compare expected and actual node sequences with order-aware LCS scoring and exact-match evidence.

Repeatability

Flakiness detection

Repeat selected cases, separate consistently wrong behavior from unstable outputs, and retain every observation.

Performance

Cost + latency tracking

See p50 and p95 response time, provider-reported token usage, and estimated cost for every run.

Quality

Deterministic-first metrics

Score exact, contains, numeric, table, hallucination, and tool-call expectations before using an LLM judge.

Regression control

CI gates that fail loudly.

Catch correctness drops, hallucination spikes, missing cases, evaluator failures, and agent errors without hiding outages inside quality rates.

correctness_drop−0.01pass tool_accuracy1.00pass agent_errors0pass

Small interface, real evidence

From crew to CI report
in a few lines.

Wrap your crew, point AgentEval at a YAML suite, and get a normalized report that can be stored, compared, and gated in CI.

1 Install from PyPI
2 Connect an adapter
3 Run your golden suite

Read the quickstart

evaluate.py

from my_crew.crew import MyCrew
from agenteval.adapters import CrewAIAdapter
from agenteval.core.runner import run_golden_suite

adapter = CrewAIAdapter(
    crew_factory=lambda: MyCrew().crew(),
    input_key="topic",
)

report = run_golden_suite(
    adapter,
    cases_path="tests/golden/cases.yaml",
    adapter_name="research_crew",
)

print(report.correctness_rate)  # 0.952

✓ 21 cases evaluated · report saved

162tests passingDeterministic suite

5quality metricsOne CI decision

PyPIpublished packagenishanttyagi-agenteval ↗ MITopen source

Make regressions reviewable

Give your agents a quality gate.

Start with a golden case. Add the rest when the evidence earns it.