AI Eval Frameworks Compared: Catching LLM Regressions

Shipping an LLM feature without evals is like deploying code with no tests and no CI. You only discover the breakage when a user files a ticket. The problem isn't unique to toy prototypes either. Production systems that used to work correctly start returning wrong answers after a model provider updates a version, a prompt gets tweaked, or a retrieval index is rebuilt. Without LLM eval frameworks in your pipeline, those regressions are invisible until they're embarrassing.

The tooling landscape has matured quickly. Three distinct categories now exist: code-based assertion suites, LLM-as-judge systems, and human-in-the-loop review platforms. They handle non-determinism differently, they integrate with CI at different friction levels, and each is genuinely better than the others in specific situations. Choosing wrong gives you false confidence or so much noise the evals get disabled.

We've run all three categories in production at Laxaar across different client engagements. This comparison is grounded in what actually broke, what caught it, and what generated so many false positives it got disabled.

Why LLM evals are harder than unit tests

Unit tests are deterministic. You call a function, check the return value, pass or fail. LLM outputs aren't deterministic. The same prompt at temperature 0.7 can return a dozen valid answers that differ in phrasing, ordering, and emphasis. A test that asserts output === expectedString will fail constantly on correct behavior.

The deeper problem is that "correctness" for language model output is often semantic, not syntactic. A customer support agent that responds "We can't process that request right now, please try again" is semantically equivalent to "That action isn't available at the moment, give it another go." A string comparison marks one as wrong. This forces you to define correctness at a higher level, which is expensive to do at scale.

There's also the distribution problem. LLMs fail in the long tail. A prompt that works on 995 out of 1000 examples looks fine in a spot check. Catching the five failures requires coverage, and coverage requires a maintained golden dataset (which most teams don't build until after something embarrassing happens).

The honest trade-off: evals cost time to build and maintain. Teams that skip them move faster short-term and pay in unpredictable production incidents long-term.

Code-based eval frameworks

Code-based eval frameworks are test suites where a developer writes assertions in Python or JavaScript, runs them against a batch of model outputs, and gets a pass/fail signal. Tools like DeepEval, Promptfoo, and RAGAS fall into this category.

The core primitive is a metric function: something that takes (input, expected, actual) and returns a score or boolean. For RAG systems, you might write metrics for answer relevance, context recall, and hallucination rate. For classifiers, you write precision/recall checks. The framework handles the test runner, reporting, and CI output format.

Here's a minimal Promptfoo eval configuration:

prompts:
  - "Summarize the following support ticket in one sentence: {{ticket}}"

providers:
  - openai:gpt-4o

tests:
  - vars:
      ticket: "My login stopped working after I changed my email address yesterday."
    assert:
      - type: contains
        value: "login"
      - type: llm-rubric
        value: "The summary is one sentence and mentions the login problem."

The strengths are real: deterministic metric functions run fast, cost nothing per call (no second LLM invocation), and produce binary output that gates CI cleanly. The weakness is that writing good metrics for open-ended outputs is hard. You end up with shallow contains assertions that miss quality regressions, or you bolt on an LLM judge inside the framework anyway.

Code-based evals are the right starting point for any project. They're cheap, fast, and force you to articulate what "correct" means before you have failures to analyze.

LLM-as-judge evaluation

LLM-as-judge is a pattern where a second language model evaluates the output of your primary model. Tools like Braintrust, LangSmith, and Confident AI implement this as a first-class feature. You give the judge a rubric ("rate factual accuracy on a scale of 1 to 5 and explain why") and it returns a structured score.

This approach handles semantic correctness well. The judge can recognize that "I'm unable to help with that" and "That falls outside what we can assist with" are functionally equivalent. It also scales to tasks where writing an algorithmic metric would take days.

from braintrust import Eval
from autoevals import Factuality

Eval(
    "customer-support-qa",
    data=lambda: load_golden_dataset("support_qa_v3.json"),
    task=lambda input: run_support_agent(input["question"]),
    scores=[Factuality],
)

The real cost is the judge call itself. Every evaluation invokes the judge model, which means eval runs are slower and carry per-token charges. On a golden dataset of 500 examples, a GPT-4o judge eval might cost $3-8 per run. That's fine for nightly runs; it adds up for every PR.

Judge models also have their own biases. They tend to favor verbose answers, penalize responses that disagree with their priors, and score inconsistently on domain-specific knowledge outside their training. You need to calibrate the judge against human labels before trusting its scores.

Our opinionated take: LLM-judge evals are better at catching quality regressions than code-based metrics, but you shouldn't gate PR merges on them without calibration. Use them for nightly regression runs and human-review escalation, not as the primary CI gate.

Human-in-the-loop review platforms

Human-in-the-loop (HITL) review platforms like Scale AI Eval, Labelbox, and Argilla route model outputs to human reviewers who score them against a rubric. These platforms are the ground truth for eval quality. Human judgment is what every other approach is trying to approximate.

HITL review catches failure modes that code metrics and LLM judges both miss: off-brand tone, regionally offensive phrasing, legally risky statements, or subtle logical errors that require domain expertise. For high-stakes applications (medical triage, legal document review, financial advice), HITL isn't optional.

The trade-offs are cost and latency. A human review cycle takes hours to days, not seconds. You can't gate a CI pipeline on it. The practical integration pattern is a sampling loop: run automated evals on every build, route a random 5-10% sample (or flagged low-confidence outputs) to human review weekly, and feed annotations back as labeled data to improve your automated metrics.

HITL also requires you to build and maintain a reviewer workforce with clear rubrics, calibration sets, and inter-rater reliability checks. That operational overhead is real.

Comparing the three approaches

Dimension	Code-Based	LLM-as-Judge	Human-in-the-Loop
Cost per run	Near zero	$0.50 - $10+	$50 - $500+
Latency	Seconds	Minutes	Hours to days
CI gate viable	Yes	With caveats	No
Semantic quality	Weak	Strong	Ground truth
Non-determinism handling	Poor (needs statistical sampling)	Good (rubric scoring)	Excellent
Calibration required	No	Yes	Yes (rubric + IAA)
Best for	Regression diffs, structural checks	Quality regressions, open-ended output	High-stakes, ground truth labeling

No single approach covers the full picture. The practical answer is a layered stack: code-based assertions for CI gates, LLM-judge for nightly quality checks, and HITL sampling for ground truth calibration. Teams that pick only one tool end up with either false confidence or an unmaintainable eval system.

CI integration patterns

Getting evals into CI is where most teams stumble. The goal is a signal that's fast enough to not block PRs, specific enough to catch real regressions, and stable enough that flaky evals don't train engineers to ignore failures.

For code-based evals, the pattern is straightforward. Add an eval job to your CI pipeline that runs on the golden dataset, asserts that metric scores don't drop more than a threshold from the baseline, and fails the build if they do.

# In your CI pipeline (GitHub Actions example)
- name: Run LLM evals
  run: |
    promptfoo eval --config evals/ci.yaml \
      --output results.json \
      --max-failures 5

For LLM-judge evals in CI, the key is to keep the dataset small (50-100 examples) and set loose thresholds. The goal isn't a perfect score. You want to detect when mean quality drops by more than 15-20% from baseline. Run the full 500-example suite overnight.

One pattern that works well: maintain two datasets. A "smoke" dataset of 30 golden examples runs on every PR using code-based assertions. The full golden dataset runs nightly with LLM-judge scoring. HITL review happens weekly on a sampled subset. This keeps CI fast while maintaining quality coverage.

The Laxaar team uses this layered approach across AI agent development projects. It's not glamorous, but it's the pattern that keeps regression rates low without making CI unbearable.

How to handle non-deterministic outputs

Non-determinism is the central engineering challenge in LLM evals. A model at temperature 0.7 doesn't return the same output twice. Naive test approaches fail because the output keeps changing even when nothing else has.

The right mental model is statistical: instead of asserting that output X equals expected Y, you assert that the distribution of outputs over N samples has certain properties. This means running the same input multiple times and aggregating scores.

import statistics

def eval_with_sampling(prompt, input_data, n_samples=5):
    scores = []
    for _ in range(n_samples):
        output = run_model(prompt, input_data)
        scores.append(factuality_score(output, input_data["expected"]))
    return {
        "mean": statistics.mean(scores),
        "min": min(scores),
        "stdev": statistics.stdev(scores),
    }

# Gate on mean score, flag on high stdev
result = eval_with_sampling(prompt, test_case)
assert result["mean"] >= 0.8, f"Mean factuality score {result['mean']} below threshold"
assert result["stdev"] <= 0.15, f"High output variance detected: {result['stdev']}"

For deterministic evals, use temperature 0 and a fixed seed where the provider supports it. Many providers expose a seed parameter that increases output consistency without guaranteeing it. This is good for catching clear regressions without sampling overhead.

Baseline diffing is another practical technique: instead of asserting absolute scores, you compare each build's scores against the previous build's scores. A 10% drop in mean factuality on the same dataset is a regression signal regardless of the absolute value. Braintrust and LangSmith both support this natively.

Frequently Asked Questions

What's the difference between LLM evals and traditional software testing?

Traditional tests check deterministic functions against exact expected outputs. LLM evals check probabilistic systems against quality rubrics. The key difference is that an LLM can be "correct" in many different ways, so eval metrics have to be semantic rather than syntactic. You're measuring distributions of quality over many runs rather than pass/fail on individual cases.

Can we run evals on every pull request without making CI too slow?

Yes, with the right dataset size. Keep your CI eval dataset at 30-100 examples maximum and use code-based or temperature-0 assertions for speed. A well-scoped eval suite runs in under two minutes on most CI environments. Save the full dataset and LLM-judge scoring for scheduled nightly runs where latency isn't a constraint.

How do we build a golden dataset when we're just starting out?

Start with your failure cases, not your successes. The most valuable golden dataset examples are inputs where the model got it wrong in production, along with manually verified correct outputs for those inputs. A dataset of 50 real failure cases beats 500 synthetic examples because it covers the distribution your system actually struggles with. Grow it incrementally as new failures surface.

Is an LLM-judge eval circular — using AI to test AI?

It's a legitimate concern. A judge model can have the same blind spots as your primary model, especially for domain-specific knowledge. The answer is calibration: score a random sample of judge outputs against human labels and measure judge accuracy. If the judge agrees with human reviewers 85% of the time on your task, it's a useful signal even if it's imperfect. Never rely on judge scores without calibration data.

Which eval framework should we start with for a new project?

Promptfoo is the easiest entry point for most teams. It's open source, runs from a YAML config, and supports both code-based and LLM-judge metrics out of the box. For teams already on LangChain or LangGraph, LangSmith fits naturally into the existing stack. Braintrust is worth considering once you need detailed baseline diffing and experiment tracking across prompt versions.

Building reliable LLM applications means accepting that eval infrastructure is a product in itself, one you have to maintain alongside your features. The teams that treat evals as an afterthought are the ones debugging surprise regressions at 2am after a model provider silently updates a version.

The Laxaar team builds eval pipelines as a standard deliverable for production AI agent development and generative AI development projects. If you're building an LLM-powered product and want help designing an eval strategy that fits your team's cadence, talk to us about your project. We're also happy to review your existing eval setup through our custom software development and AI automation services engagements.