On this article, you’ll learn to consider LLM functions utilizing the three dominant open-source frameworks — RAGAS, DeepEval, and Promptfoo — and why the LLM-as-a-judge mechanism all of them depend on has measurable biases it is advisable to actively design round.
Matters we are going to cowl embody:
- How RAGAS, DeepEval, and Promptfoo differ in goal and when to make use of each, together with which pairings skilled groups converge on.
- Learn how to implement a faithfulness verify and a CI-gated high quality analysis with working code you possibly can run instantly.
- What place bias, self-preference bias, and verbosity bias are, how you can detect them with an audit harness, and how you can mitigate them in manufacturing.
There’s so much to get by way of, so let’s get proper into it.
Introduction
You ship an LLM characteristic after seeing a few outputs and determine it seems good. Three weeks later, a immediate tweak silently breaks one thing no one was testing for, and no one notices till a person complains. That is the default failure mode for LLM functions, and it’s totally different from a typical software program bug. Conventional code fails with a stack hint. LLM outputs fail by being confidently, plausibly unsuitable, which is precisely the sort of failure a fast handbook look gained’t catch.
Three open-source instruments dominate the sensible facet of this downside in 2026: Promptfoo, DeepEval, and RAGAS. Every is constructed for a special form of downside, not competing for a similar job. Layered above them are production-monitoring platforms like LangSmith and Braintrust, which decide up the place offline analysis leaves off. None of those instruments wins outright; most mature GenAI QA packages run two of them in parallel: a light-weight framework for blocking dangerous deploys plus a platform for ongoing monitoring and human overview.
This text compares the frameworks that really matter, walks by way of examined code for the 2 commonest analysis jobs, and covers the half most comparability items skip fully: the truth that “LLM-as-a-judge“, the mechanism almost each framework right here depends on, has measurable, revealed biases it is advisable to design round, not simply belief.
What “Evaluating an LLM” Really Means
Earlier than evaluating instruments, it helps to separate three issues folks conflate once they say “LLM analysis.” Selecting the unsuitable class right here is the only commonest mistake groups make.
- Mannequin benchmarking compares uncooked mannequin capabilities on standardized educational duties, reminiscent of MMLU, GSM8K, and HumanEval. lm-evaluation-harness is the usual right here, with no actual substitute when the requirement is a standardized educational benchmark. For those who’re selecting between GPT-5 and Claude for a brand new venture, that is the class you need, but it surely tells you virtually nothing about whether or not your particular utility works.
- Utility analysis asks a narrower, extra helpful query: does your RAG pipeline, chatbot, or agent produce right, grounded, secure outputs in your information and your prompts? That is the place RAGAS, DeepEval, and Promptfoo reside, and it’s the place this text spends most of its time.
- Manufacturing monitoring tracks reside visitors after deployment, catching regressions and drift that offline take a look at units by no means anticipated. That is LangSmith, Braintrust, and Arize Phoenix territory.
Most individuals asking “which eval framework ought to I exploit” really want the second class, typically paired with the third. The remainder of this text focuses on that.
The Metrics Beneath the Frameworks
Earlier than the framework comparability is sensible, it’s price realizing what’s truly being scored, as a result of each instrument under implements some model of the identical handful of metrics.
- Faithfulness (or groundedness) checks whether or not a solution incorporates solely claims supported by the retrieved context — the core mechanism for catching RAG hallucinations.
- Context precision and recall verify whether or not retrieval pulled the best paperwork, and solely the best ones, earlier than era even occurs.
- Reply relevancy checks whether or not the response truly addresses the query requested, unbiased of whether or not it’s factually grounded.
- G-Eval, launched by Liu et al., makes use of chain-of-thought prompting mixed with form-filling to information an LLM choose by way of an express rubric, and has been proven to align with human choice extra carefully than naive “price this 1-10” prompting. Past these, most frameworks add task-specific checks for toxicity, bias, and PII leakage.
The actual differentiator between frameworks isn’t metric novelty; they largely implement the identical handful of concepts. It’s workflow match: how the metric will get triggered, the place the end result goes, and whether or not it blocks a deploy or simply generates a report.
RAGAS vs. DeepEval vs. Promptfoo, Head to Head
RAGAS is research-backed, with academic-grade methodology behind metrics like faithfulness, context precision, and context recall, but it surely’s scoped to retrieval and era scoring, with no manufacturing monitoring or collaboration layer in-built. Choose it when your structure is retrieval-heavy and also you need metrics with a broadcast paper behind their definition, not only a vendor’s inside heuristic.
- DeepEval is Python-native and pytest-based, with 14-plus metrics spanning hallucination, bias, toxicity, and RAG-specific checks — constructed explicitly to perform as a CI/CD high quality gate that may block a deploy. Choose it when analysis must reside inside your current take a look at suite quite than as a separate offline report somebody has to recollect to run.
- Promptfoo is CLI-first and YAML-config-driven, strongest at multi-model immediate comparability and adversarial red-teaming, with 500-plus built-in assault vectors in its security-testing suite. Choose it for immediate engineering iteration throughout a number of fashions, or when red-teaming and safety testing are the precise requirement.
The framing that issues most: DeepEval and RAGAS aren’t actually opponents. DeepEval covers broad LLM utility testing, RAGAS specializes particularly in RAG, and a significant share of manufacturing groups run each collectively — with RAGAS scoring the retrieval-specific dimensions and DeepEval dealing with every part else inside the identical CI pipeline.
| Class | RAGAS | DeepEval | Promptfoo |
|---|---|---|---|
| Finest for | RAG-specific scoring | CI/CD high quality gates | Multi-model comparability, red-teaming |
| Integration type | Python library | pytest-native | YAML + CLI |
| Strongest metric set | Faithfulness, context precision/recall | 14+ metrics incl. bias, toxicity | Safety/assault vectors (500+) |
| Manufacturing monitoring | No | No | No |
| Pairs effectively with | DeepEval (broader protection) | RAGAS (RAG-specific depth) | Both for prompt-side testing |
Code Walkthrough: Catching Hallucination with a Faithfulness Verify
Right here’s the mechanism behind RAGAS’s faithfulness metric, demonstrated instantly: decompose a solution into atomic claims, then verify every declare towards the retrieved context. A declare with no assist within the context is a hallucination — precisely the failure mode {that a} fast handbook learn tends to overlook, as a result of the unsupported element typically sounds utterly believable.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
# faithfulness_check.py # Conditions: none past Python’s normal library (re) # Run: python faithfulness_check.py # # Word: this demonstrates the faithfulness-checking MECHANISM that RAGAS’s # actual Faithfulness metric implements with an LLM choose. The keyword-overlap # verify under is a simplified, totally offline-testable stand-in for that # LLM-based declare verification — swap in RAGAS’s precise metric for manufacturing use.
import re
def decompose_claims(reply: str) -> listing[str]: “”“Cut up a solution into atomic, independently-checkable statements.”“” sentences = re.break up(r‘(?<=[.!?])s+’, reply.strip()) return [s.strip() for s in sentences if s.strip()]
def claim_supported_by_context(declare: str, context: str) -> bool: “”“ Verify whether or not a declare has lexical assist within the retrieved context. RAGAS does this with an LLM choose; this overlap verify demonstrates the identical supported/unsupported choice in a deterministic means. ““” claim_words = set(re.findall(r‘b[a-zA-Z]{4,}b’, declare.decrease())) context_words = set(re.findall(r‘b[a-zA-Z]{4,}b’, context.decrease())) if not claim_words: return True overlap = len(claim_words & context_words) / len(claim_words) return overlap >= 0.5
def compute_faithfulness(reply: str, context: str) -> dict: “”“ Faithfulness rating = fraction of claims within the reply supported by context. This mirrors RAGAS’s precise metric definition: supported claims / whole claims. ““” claims = decompose_claims(reply) supported = [c for c in claims if claim_supported_by_context(c, context)] unsupported = [c for c in claims if c not in supported] rating = len(supported) / len(claims) if claims else 1.0 return { “rating”: spherical(rating, 3), “total_claims”: len(claims), “unsupported_claims”: unsupported, }
if __name__ == “__main__”: context = “Abuja grew to become the capital of Nigeria in 1991, changing Lagos because the seat of presidency.”
# Case 1: totally grounded reply — each declare traces again to the context grounded_answer = “The capital of Nigeria is Abuja. It grew to become the capital in 1991.” result_1 = compute_faithfulness(grounded_answer, context) print(“Grounded reply:”) print(f” Faithfulness rating: {result_1[‘score’]}”) print(f” Unsupported claims: {result_1[‘unsupported_claims’]}n”)
# Case 2: the mannequin provides a plausible-sounding element the context by no means talked about hallucinated_answer = ( “The capital of Nigeria is Abuja. It grew to become the capital in 1991. “ “The town has a inhabitants of over 3 million folks.” ) result_2 = compute_faithfulness(hallucinated_answer, context) print(“Reply with a hallucinated element:”) print(f” Faithfulness rating: {result_2[‘score’]}”) print(f” Unsupported claims: {result_2[‘unsupported_claims’]}”) |
Learn how to run (no dependencies required):
|
python faithfulness_check.py |
Output:
|
Grounded reply: Faithfulness rating: 1.0 Unsupported claims: []
Reply with a hallucinated element: Faithfulness rating: 0.667 Unsupported claims: [‘The city has a population of over 3 million people.’] |
That inhabitants determine sounds fully cheap, which is precisely why a handbook overview would doubtless let it by way of. The faithfulness verify catches it as a result of it’s checking towards the precise retrieved context, not towards normal plausibility. That is the mechanism operating beneath RAGAS’s actual Faithfulness metric, which makes use of an LLM to do the declare decomposition and support-checking as a substitute of key phrase overlap — extra correct, identical underlying logic.
To run this with the precise RAGAS library towards a reside mannequin:
|
# Manufacturing sample utilizing the actual RAGAS library # pip set up ragas
from ragas import SingleTurnSample, EvaluationDataset from ragas.metrics import Faithfulness from ragas import consider
pattern = SingleTurnSample( user_input=“What’s the capital of Nigeria?”, response=“The capital of Nigeria is Abuja. It grew to become the capital in 1991. The town has a inhabitants of over 3 million folks.”, retrieved_contexts=[“Abuja became the capital of Nigeria in 1991, replacing Lagos as the seat of government.”], ) dataset = EvaluationDataset(samples=[sample]) outcomes = consider(dataset, metrics=[Faithfulness()]) print(outcomes) |
Code Walkthrough: CI-Gated Analysis with DeepEval
The sample that makes DeepEval distinct from a standalone analysis script is that it runs as an actual pytest take a look at, which means a high quality regression fails the construct the identical means a damaged unit take a look at would — as a substitute of producing a report somebody has to recollect to learn.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
# test_response_quality.py # Conditions: pip set up deepeval pytest # Set your choose mannequin’s API key as an atmosphere variable earlier than operating # Run: deepeval take a look at run test_response_quality.py
import pytest from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import GEval
# G-Eval enables you to outline a customized rubric in plain language — the LLM choose # makes use of chain-of-thought reasoning towards this rubric quite than a generic # “price this 1-10” immediate, which is what provides G-Eval higher alignment # with human judgment than naive scoring prompts. correctness_metric = GEval( title=“Coverage Accuracy”, standards=( “Decide whether or not the precise output precisely displays firm coverage “ “with out including unspoken circumstances or omitting required disclosures.” ), evaluation_params=[“input”, “actual_output”], threshold=0.7, # Minimal rating to go — tune primarily based in your threat tolerance )
def test_refund_policy_response(): “”“ This take a look at fails the construct if the mannequin’s refund coverage rationalization drops under the correctness threshold — the identical means a damaged assertion would fail another pytest take a look at. ““” test_case = LLMTestCase( enter=“What’s the refund coverage?”, actual_output=“You’ll be able to request a refund inside 30 days of buy, no questions requested.”, ) assert_test(test_case, [correctness_metric])
def test_refund_policy_response_with_unstated_condition(): “”“ This case demonstrates what a FAILING take a look at seems like: the response provides a situation (“solely for unopened gadgets“) that wasn’t a part of the precise coverage being examined towards, which ought to drag the rating down. ““” test_case = LLMTestCase( enter=“What’s the refund coverage?”, actual_output=( “You’ll be able to request a refund inside 30 days, however just for unopened gadgets “ “and solely when you have the unique receipt and packaging.” ), ) assert_test(test_case, [correctness_metric]) |
Conditions:
|
pip set up deepeval pytest export OPENAI_API_KEY=your_key # DeepEval makes use of an LLM choose beneath the hood |
Learn how to run:
|
deepeval take a look at run test_response_quality.py |
The primary take a look at ought to go; the response matches a believable, unembellished coverage assertion. The second is written to exhibit a possible failure: it provides circumstances that weren’t a part of the unique enter, which a well-configured G-Eval rubric ought to catch and rating under the 0.7 threshold. In an actual CI pipeline, that failure blocks the merge — precisely the conduct that turns analysis from “one thing we must always verify on” into “one thing the pipeline enforces.”
The Downside No person Mentions: LLM-as-a-Choose Is Biased
Each framework above depends on the identical underlying mechanism for the metrics that matter most: an LLM judging one other LLM’s output. That choose is just not impartial, and the analysis on that is extra developed than most groups understand.
- Place bias is the best-documented of those. A systematic examine at IJCNLP 2025 evaluated 15 LLM judges throughout roughly 150,000 analysis situations and located that judges systematically favor whichever response sits in a selected slot of the immediate, and that this impact is just not attributable to random probability. Swap the order of two responses being in contrast, and the identical choose can flip its verdict purely due to the place every response now sits — not as a result of both response modified.
- Self-preference bias compounds this. Fashions disproportionately favor outputs generated by themselves or by fashions in their very own household, which means that should you use GPT-4o to guage GPT-4o’s personal outputs, the ensuing rating is inflated above what an unbiased choose would assign. Analysis distinguishes this from real high quality variations: not each self-preference sign is biased, however the dangerous model is particularly when a choose fails to penalize its personal mannequin household’s errors.
- Verbosity bias is the third main one: longer responses get rated larger unbiased of whether or not the added size incorporates something helpful. A choose evaluating a concise, right reply towards a padded, partially redundant one will typically favor the longer one.
The usually-cited statistic that LLM judges attain roughly 80% settlement with human evaluators comes from the unique MT-Bench examine and is correct as an mixture determine, but it surely describes common efficiency throughout a broad benchmark — not reliability in your particular process along with your particular choose mannequin. Treating that quantity as a production-readiness assure is the error.
Code: A Place-Bias Detection Harness
The only highest-leverage verify most groups skip: run the identical pairwise comparability twice, with the response order swapped, and see whether or not the decision flips.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
# position_bias_audit.py # Conditions: none past Python’s normal library (random, dataclasses) # Run: python position_bias_audit.py
import random from dataclasses import dataclass
@dataclass class PairwiseResult: question: str verdict_original_order: str verdict_swapped_order: str position_consistent: bool # False means the decision flipped purely on slot place
def run_position_bias_check(question: str, response_x: str, response_y: str, judge_fn) -> PairwiseResult: “”“ Run the identical comparability twice with positions swapped. An unbiased choose ought to decide the identical underlying response each instances no matter which slot it occupies. A flip signifies place bias, not a real high quality sign. ““” # Spherical 1: response_x in slot A, response_y in slot B verdict_1 = judge_fn(response_x, response_y) winner_1 = response_x if verdict_1 == “A” else response_y
# Spherical 2: swap — response_y now in slot A, response_x in slot B verdict_2 = judge_fn(response_y, response_x) winner_2 = response_y if verdict_2 == “A” else response_x
return PairwiseResult( question=question, verdict_original_order=verdict_1, verdict_swapped_order=verdict_2, position_consistent=(winner_1 == winner_2), )
def audit_position_bias(test_pairs: listing[tuple], judge_fn, n_trials: int = 50) -> dict: “”“ Run many position-swapped comparisons and report the speed of inconsistent verdicts. A excessive price means your choose is responding to fit place, not response high quality — and any rating it produces must be handled with actual skepticism till that is addressed. ““” outcomes = [] for question, resp_x, resp_y in test_pairs: for _ in vary(n_trials // len(test_pairs)): outcomes.append(run_position_bias_check(question, resp_x, resp_y, judge_fn))
inconsistent = [r for r in results if not r.position_consistent] return { “total_trials”: len(outcomes), “inconsistent_count”: len(inconsistent), “inconsistency_rate”: spherical(len(inconsistent) / len(outcomes), 3), }
if __name__ == “__main__”: # In manufacturing, change this with an actual name to your choose LLM evaluating # response_1 vs response_2 and returning “A” or “B”. def your_judge_function(response_1: str, response_2: str) -> str: # Placeholder — wire this as much as your precise choose mannequin name. elevate NotImplementedError(“Change along with your actual LLM choose name”)
test_pairs = [ (“Summarize the quarterly report”, “Response variant A”, “Response variant B”), ]
# Demo with a simulated 70%-position-A-biased choose, for illustration random.seed(7) def demo_biased_judge(r1, r2): return “A” if random.random() < 0.7 else “B”
report = audit_position_bias(test_pairs, demo_biased_judge, n_trials=200) print(f“Inconsistency price: {report[‘inconsistency_rate’] * 100:.1f}%”) print(f“({report[‘inconsistent_count’]}/{report[‘total_trials’]} trials flipped purely on place swap)”) print(“nA price meaningfully above 0% signifies place bias in your choose setup.”) print(“Mitigation: common scores throughout each orderings, or use a separate choose”) print(“mannequin from a special household than the mannequin being evaluated.”) |
Learn how to run (no dependencies required):
|
python position_bias_audit.py |
Output (with the simulated biased choose):
|
Inconsistency price: 55.5% (111/200 trials flipped purely on place swap)
A price meaningfully above 0% signifies place bias in your choose setup. Mitigation: common scores throughout each orderings, or use a separate choose mannequin from a totally different household than the mannequin being evaluated |
A 55% flip price is a extreme case used right here for readability; most actual judges aren’t this biased, however even a flip price within the 10–15% vary — which reveals up often in manufacturing choose setups — is sufficient to make a borderline go/fail choice unreliable. The repair prices one further LLM name per analysis: run each orderings and common, or, extra robustly, use a choose mannequin from a special household than whichever mannequin produced the responses being scored, which instantly addresses the self-preference difficulty on the identical time.
Selecting Your Stack
There’s no common reply right here, however the choice tree is pretty clear as soon as you understand what you’re constructing:
- RAG utility: begin with RAGAS for the retrieval-specific metrics (faithfulness, context precision, context recall). Add DeepEval alongside it should you want broader protection of toxicity, bias, and normal correctness in the identical take a look at suite.
- Basic chatbot or content-generation characteristic with CI necessities: DeepEval, run as a part of your current pytest suite, gating merges on a high quality threshold.
- Immediate iteration throughout a number of fashions, or safety/red-teaming: Promptfoo. Its YAML-driven config makes multi-model comparability quick, and its built-in assault suite is essentially the most full open-source possibility for adversarial testing.
- Manufacturing tracing and human overview after deployment: LangSmith in case your stack is already constructed on LangChain or LangGraph; Braintrust in case your stack is extra heterogeneous and also you desire a platform that isn’t coupled to 1 framework.
The sample skilled groups converge on is 2 instruments, not one: a light-weight framework for CI-time gating, paired with a platform for ongoing monitoring, regression monitoring, and the human annotation that no automated metric totally replaces.
Conclusion
No framework right here wins outright, as a result of they’re not fixing the identical downside. RAGAS, DeepEval, and Promptfoo every match a special form of analysis want, and the actual architectural choice is often “which two of those do I run collectively,” not “which one is greatest.”
The larger threat isn’t selecting the unsuitable instrument from this listing; it’s trusting no matter rating an LLM choose produces with out checking it for the biases documented above. Place bias, self-preference, and verbosity bias aren’t edge instances buried in a tutorial paper; they’re measurable results that present up in atypical choose setups, together with those contained in the very frameworks this text simply walked by way of. The frameworks provide the scoring mechanism. The audit behavior within the position-bias part is what makes the ensuing quantity price trusting.
On this article, you’ll learn to consider LLM functions utilizing the three dominant open-source frameworks — RAGAS, DeepEval, and Promptfoo — and why the LLM-as-a-judge mechanism all of them depend on has measurable biases it is advisable to actively design round.
Matters we are going to cowl embody:
- How RAGAS, DeepEval, and Promptfoo differ in goal and when to make use of each, together with which pairings skilled groups converge on.
- Learn how to implement a faithfulness verify and a CI-gated high quality analysis with working code you possibly can run instantly.
- What place bias, self-preference bias, and verbosity bias are, how you can detect them with an audit harness, and how you can mitigate them in manufacturing.
There’s so much to get by way of, so let’s get proper into it.
Introduction
You ship an LLM characteristic after seeing a few outputs and determine it seems good. Three weeks later, a immediate tweak silently breaks one thing no one was testing for, and no one notices till a person complains. That is the default failure mode for LLM functions, and it’s totally different from a typical software program bug. Conventional code fails with a stack hint. LLM outputs fail by being confidently, plausibly unsuitable, which is precisely the sort of failure a fast handbook look gained’t catch.
Three open-source instruments dominate the sensible facet of this downside in 2026: Promptfoo, DeepEval, and RAGAS. Every is constructed for a special form of downside, not competing for a similar job. Layered above them are production-monitoring platforms like LangSmith and Braintrust, which decide up the place offline analysis leaves off. None of those instruments wins outright; most mature GenAI QA packages run two of them in parallel: a light-weight framework for blocking dangerous deploys plus a platform for ongoing monitoring and human overview.
This text compares the frameworks that really matter, walks by way of examined code for the 2 commonest analysis jobs, and covers the half most comparability items skip fully: the truth that “LLM-as-a-judge“, the mechanism almost each framework right here depends on, has measurable, revealed biases it is advisable to design round, not simply belief.
What “Evaluating an LLM” Really Means
Earlier than evaluating instruments, it helps to separate three issues folks conflate once they say “LLM analysis.” Selecting the unsuitable class right here is the only commonest mistake groups make.
- Mannequin benchmarking compares uncooked mannequin capabilities on standardized educational duties, reminiscent of MMLU, GSM8K, and HumanEval. lm-evaluation-harness is the usual right here, with no actual substitute when the requirement is a standardized educational benchmark. For those who’re selecting between GPT-5 and Claude for a brand new venture, that is the class you need, but it surely tells you virtually nothing about whether or not your particular utility works.
- Utility analysis asks a narrower, extra helpful query: does your RAG pipeline, chatbot, or agent produce right, grounded, secure outputs in your information and your prompts? That is the place RAGAS, DeepEval, and Promptfoo reside, and it’s the place this text spends most of its time.
- Manufacturing monitoring tracks reside visitors after deployment, catching regressions and drift that offline take a look at units by no means anticipated. That is LangSmith, Braintrust, and Arize Phoenix territory.
Most individuals asking “which eval framework ought to I exploit” really want the second class, typically paired with the third. The remainder of this text focuses on that.
The Metrics Beneath the Frameworks
Earlier than the framework comparability is sensible, it’s price realizing what’s truly being scored, as a result of each instrument under implements some model of the identical handful of metrics.
- Faithfulness (or groundedness) checks whether or not a solution incorporates solely claims supported by the retrieved context — the core mechanism for catching RAG hallucinations.
- Context precision and recall verify whether or not retrieval pulled the best paperwork, and solely the best ones, earlier than era even occurs.
- Reply relevancy checks whether or not the response truly addresses the query requested, unbiased of whether or not it’s factually grounded.
- G-Eval, launched by Liu et al., makes use of chain-of-thought prompting mixed with form-filling to information an LLM choose by way of an express rubric, and has been proven to align with human choice extra carefully than naive “price this 1-10” prompting. Past these, most frameworks add task-specific checks for toxicity, bias, and PII leakage.
The actual differentiator between frameworks isn’t metric novelty; they largely implement the identical handful of concepts. It’s workflow match: how the metric will get triggered, the place the end result goes, and whether or not it blocks a deploy or simply generates a report.
RAGAS vs. DeepEval vs. Promptfoo, Head to Head
RAGAS is research-backed, with academic-grade methodology behind metrics like faithfulness, context precision, and context recall, but it surely’s scoped to retrieval and era scoring, with no manufacturing monitoring or collaboration layer in-built. Choose it when your structure is retrieval-heavy and also you need metrics with a broadcast paper behind their definition, not only a vendor’s inside heuristic.
- DeepEval is Python-native and pytest-based, with 14-plus metrics spanning hallucination, bias, toxicity, and RAG-specific checks — constructed explicitly to perform as a CI/CD high quality gate that may block a deploy. Choose it when analysis must reside inside your current take a look at suite quite than as a separate offline report somebody has to recollect to run.
- Promptfoo is CLI-first and YAML-config-driven, strongest at multi-model immediate comparability and adversarial red-teaming, with 500-plus built-in assault vectors in its security-testing suite. Choose it for immediate engineering iteration throughout a number of fashions, or when red-teaming and safety testing are the precise requirement.
The framing that issues most: DeepEval and RAGAS aren’t actually opponents. DeepEval covers broad LLM utility testing, RAGAS specializes particularly in RAG, and a significant share of manufacturing groups run each collectively — with RAGAS scoring the retrieval-specific dimensions and DeepEval dealing with every part else inside the identical CI pipeline.
| Class | RAGAS | DeepEval | Promptfoo |
|---|---|---|---|
| Finest for | RAG-specific scoring | CI/CD high quality gates | Multi-model comparability, red-teaming |
| Integration type | Python library | pytest-native | YAML + CLI |
| Strongest metric set | Faithfulness, context precision/recall | 14+ metrics incl. bias, toxicity | Safety/assault vectors (500+) |
| Manufacturing monitoring | No | No | No |
| Pairs effectively with | DeepEval (broader protection) | RAGAS (RAG-specific depth) | Both for prompt-side testing |
Code Walkthrough: Catching Hallucination with a Faithfulness Verify
Right here’s the mechanism behind RAGAS’s faithfulness metric, demonstrated instantly: decompose a solution into atomic claims, then verify every declare towards the retrieved context. A declare with no assist within the context is a hallucination — precisely the failure mode {that a} fast handbook learn tends to overlook, as a result of the unsupported element typically sounds utterly believable.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
# faithfulness_check.py # Conditions: none past Python’s normal library (re) # Run: python faithfulness_check.py # # Word: this demonstrates the faithfulness-checking MECHANISM that RAGAS’s # actual Faithfulness metric implements with an LLM choose. The keyword-overlap # verify under is a simplified, totally offline-testable stand-in for that # LLM-based declare verification — swap in RAGAS’s precise metric for manufacturing use.
import re
def decompose_claims(reply: str) -> listing[str]: “”“Cut up a solution into atomic, independently-checkable statements.”“” sentences = re.break up(r‘(?<=[.!?])s+’, reply.strip()) return [s.strip() for s in sentences if s.strip()]
def claim_supported_by_context(declare: str, context: str) -> bool: “”“ Verify whether or not a declare has lexical assist within the retrieved context. RAGAS does this with an LLM choose; this overlap verify demonstrates the identical supported/unsupported choice in a deterministic means. ““” claim_words = set(re.findall(r‘b[a-zA-Z]{4,}b’, declare.decrease())) context_words = set(re.findall(r‘b[a-zA-Z]{4,}b’, context.decrease())) if not claim_words: return True overlap = len(claim_words & context_words) / len(claim_words) return overlap >= 0.5
def compute_faithfulness(reply: str, context: str) -> dict: “”“ Faithfulness rating = fraction of claims within the reply supported by context. This mirrors RAGAS’s precise metric definition: supported claims / whole claims. ““” claims = decompose_claims(reply) supported = [c for c in claims if claim_supported_by_context(c, context)] unsupported = [c for c in claims if c not in supported] rating = len(supported) / len(claims) if claims else 1.0 return { “rating”: spherical(rating, 3), “total_claims”: len(claims), “unsupported_claims”: unsupported, }
if __name__ == “__main__”: context = “Abuja grew to become the capital of Nigeria in 1991, changing Lagos because the seat of presidency.”
# Case 1: totally grounded reply — each declare traces again to the context grounded_answer = “The capital of Nigeria is Abuja. It grew to become the capital in 1991.” result_1 = compute_faithfulness(grounded_answer, context) print(“Grounded reply:”) print(f” Faithfulness rating: {result_1[‘score’]}”) print(f” Unsupported claims: {result_1[‘unsupported_claims’]}n”)
# Case 2: the mannequin provides a plausible-sounding element the context by no means talked about hallucinated_answer = ( “The capital of Nigeria is Abuja. It grew to become the capital in 1991. “ “The town has a inhabitants of over 3 million folks.” ) result_2 = compute_faithfulness(hallucinated_answer, context) print(“Reply with a hallucinated element:”) print(f” Faithfulness rating: {result_2[‘score’]}”) print(f” Unsupported claims: {result_2[‘unsupported_claims’]}”) |
Learn how to run (no dependencies required):
|
python faithfulness_check.py |
Output:
|
Grounded reply: Faithfulness rating: 1.0 Unsupported claims: []
Reply with a hallucinated element: Faithfulness rating: 0.667 Unsupported claims: [‘The city has a population of over 3 million people.’] |
That inhabitants determine sounds fully cheap, which is precisely why a handbook overview would doubtless let it by way of. The faithfulness verify catches it as a result of it’s checking towards the precise retrieved context, not towards normal plausibility. That is the mechanism operating beneath RAGAS’s actual Faithfulness metric, which makes use of an LLM to do the declare decomposition and support-checking as a substitute of key phrase overlap — extra correct, identical underlying logic.
To run this with the precise RAGAS library towards a reside mannequin:
|
# Manufacturing sample utilizing the actual RAGAS library # pip set up ragas
from ragas import SingleTurnSample, EvaluationDataset from ragas.metrics import Faithfulness from ragas import consider
pattern = SingleTurnSample( user_input=“What’s the capital of Nigeria?”, response=“The capital of Nigeria is Abuja. It grew to become the capital in 1991. The town has a inhabitants of over 3 million folks.”, retrieved_contexts=[“Abuja became the capital of Nigeria in 1991, replacing Lagos as the seat of government.”], ) dataset = EvaluationDataset(samples=[sample]) outcomes = consider(dataset, metrics=[Faithfulness()]) print(outcomes) |
Code Walkthrough: CI-Gated Analysis with DeepEval
The sample that makes DeepEval distinct from a standalone analysis script is that it runs as an actual pytest take a look at, which means a high quality regression fails the construct the identical means a damaged unit take a look at would — as a substitute of producing a report somebody has to recollect to learn.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
# test_response_quality.py # Conditions: pip set up deepeval pytest # Set your choose mannequin’s API key as an atmosphere variable earlier than operating # Run: deepeval take a look at run test_response_quality.py
import pytest from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import GEval
# G-Eval enables you to outline a customized rubric in plain language — the LLM choose # makes use of chain-of-thought reasoning towards this rubric quite than a generic # “price this 1-10” immediate, which is what provides G-Eval higher alignment # with human judgment than naive scoring prompts. correctness_metric = GEval( title=“Coverage Accuracy”, standards=( “Decide whether or not the precise output precisely displays firm coverage “ “with out including unspoken circumstances or omitting required disclosures.” ), evaluation_params=[“input”, “actual_output”], threshold=0.7, # Minimal rating to go — tune primarily based in your threat tolerance )
def test_refund_policy_response(): “”“ This take a look at fails the construct if the mannequin’s refund coverage rationalization drops under the correctness threshold — the identical means a damaged assertion would fail another pytest take a look at. ““” test_case = LLMTestCase( enter=“What’s the refund coverage?”, actual_output=“You’ll be able to request a refund inside 30 days of buy, no questions requested.”, ) assert_test(test_case, [correctness_metric])
def test_refund_policy_response_with_unstated_condition(): “”“ This case demonstrates what a FAILING take a look at seems like: the response provides a situation (“solely for unopened gadgets“) that wasn’t a part of the precise coverage being examined towards, which ought to drag the rating down. ““” test_case = LLMTestCase( enter=“What’s the refund coverage?”, actual_output=( “You’ll be able to request a refund inside 30 days, however just for unopened gadgets “ “and solely when you have the unique receipt and packaging.” ), ) assert_test(test_case, [correctness_metric]) |
Conditions:
|
pip set up deepeval pytest export OPENAI_API_KEY=your_key # DeepEval makes use of an LLM choose beneath the hood |
Learn how to run:
|
deepeval take a look at run test_response_quality.py |
The primary take a look at ought to go; the response matches a believable, unembellished coverage assertion. The second is written to exhibit a possible failure: it provides circumstances that weren’t a part of the unique enter, which a well-configured G-Eval rubric ought to catch and rating under the 0.7 threshold. In an actual CI pipeline, that failure blocks the merge — precisely the conduct that turns analysis from “one thing we must always verify on” into “one thing the pipeline enforces.”
The Downside No person Mentions: LLM-as-a-Choose Is Biased
Each framework above depends on the identical underlying mechanism for the metrics that matter most: an LLM judging one other LLM’s output. That choose is just not impartial, and the analysis on that is extra developed than most groups understand.
- Place bias is the best-documented of those. A systematic examine at IJCNLP 2025 evaluated 15 LLM judges throughout roughly 150,000 analysis situations and located that judges systematically favor whichever response sits in a selected slot of the immediate, and that this impact is just not attributable to random probability. Swap the order of two responses being in contrast, and the identical choose can flip its verdict purely due to the place every response now sits — not as a result of both response modified.
- Self-preference bias compounds this. Fashions disproportionately favor outputs generated by themselves or by fashions in their very own household, which means that should you use GPT-4o to guage GPT-4o’s personal outputs, the ensuing rating is inflated above what an unbiased choose would assign. Analysis distinguishes this from real high quality variations: not each self-preference sign is biased, however the dangerous model is particularly when a choose fails to penalize its personal mannequin household’s errors.
- Verbosity bias is the third main one: longer responses get rated larger unbiased of whether or not the added size incorporates something helpful. A choose evaluating a concise, right reply towards a padded, partially redundant one will typically favor the longer one.
The usually-cited statistic that LLM judges attain roughly 80% settlement with human evaluators comes from the unique MT-Bench examine and is correct as an mixture determine, but it surely describes common efficiency throughout a broad benchmark — not reliability in your particular process along with your particular choose mannequin. Treating that quantity as a production-readiness assure is the error.
Code: A Place-Bias Detection Harness
The only highest-leverage verify most groups skip: run the identical pairwise comparability twice, with the response order swapped, and see whether or not the decision flips.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
# position_bias_audit.py # Conditions: none past Python’s normal library (random, dataclasses) # Run: python position_bias_audit.py
import random from dataclasses import dataclass
@dataclass class PairwiseResult: question: str verdict_original_order: str verdict_swapped_order: str position_consistent: bool # False means the decision flipped purely on slot place
def run_position_bias_check(question: str, response_x: str, response_y: str, judge_fn) -> PairwiseResult: “”“ Run the identical comparability twice with positions swapped. An unbiased choose ought to decide the identical underlying response each instances no matter which slot it occupies. A flip signifies place bias, not a real high quality sign. ““” # Spherical 1: response_x in slot A, response_y in slot B verdict_1 = judge_fn(response_x, response_y) winner_1 = response_x if verdict_1 == “A” else response_y
# Spherical 2: swap — response_y now in slot A, response_x in slot B verdict_2 = judge_fn(response_y, response_x) winner_2 = response_y if verdict_2 == “A” else response_x
return PairwiseResult( question=question, verdict_original_order=verdict_1, verdict_swapped_order=verdict_2, position_consistent=(winner_1 == winner_2), )
def audit_position_bias(test_pairs: listing[tuple], judge_fn, n_trials: int = 50) -> dict: “”“ Run many position-swapped comparisons and report the speed of inconsistent verdicts. A excessive price means your choose is responding to fit place, not response high quality — and any rating it produces must be handled with actual skepticism till that is addressed. ““” outcomes = [] for question, resp_x, resp_y in test_pairs: for _ in vary(n_trials // len(test_pairs)): outcomes.append(run_position_bias_check(question, resp_x, resp_y, judge_fn))
inconsistent = [r for r in results if not r.position_consistent] return { “total_trials”: len(outcomes), “inconsistent_count”: len(inconsistent), “inconsistency_rate”: spherical(len(inconsistent) / len(outcomes), 3), }
if __name__ == “__main__”: # In manufacturing, change this with an actual name to your choose LLM evaluating # response_1 vs response_2 and returning “A” or “B”. def your_judge_function(response_1: str, response_2: str) -> str: # Placeholder — wire this as much as your precise choose mannequin name. elevate NotImplementedError(“Change along with your actual LLM choose name”)
test_pairs = [ (“Summarize the quarterly report”, “Response variant A”, “Response variant B”), ]
# Demo with a simulated 70%-position-A-biased choose, for illustration random.seed(7) def demo_biased_judge(r1, r2): return “A” if random.random() < 0.7 else “B”
report = audit_position_bias(test_pairs, demo_biased_judge, n_trials=200) print(f“Inconsistency price: {report[‘inconsistency_rate’] * 100:.1f}%”) print(f“({report[‘inconsistent_count’]}/{report[‘total_trials’]} trials flipped purely on place swap)”) print(“nA price meaningfully above 0% signifies place bias in your choose setup.”) print(“Mitigation: common scores throughout each orderings, or use a separate choose”) print(“mannequin from a special household than the mannequin being evaluated.”) |
Learn how to run (no dependencies required):
|
python position_bias_audit.py |
Output (with the simulated biased choose):
|
Inconsistency price: 55.5% (111/200 trials flipped purely on place swap)
A price meaningfully above 0% signifies place bias in your choose setup. Mitigation: common scores throughout each orderings, or use a separate choose mannequin from a totally different household than the mannequin being evaluated |
A 55% flip price is a extreme case used right here for readability; most actual judges aren’t this biased, however even a flip price within the 10–15% vary — which reveals up often in manufacturing choose setups — is sufficient to make a borderline go/fail choice unreliable. The repair prices one further LLM name per analysis: run each orderings and common, or, extra robustly, use a choose mannequin from a special household than whichever mannequin produced the responses being scored, which instantly addresses the self-preference difficulty on the identical time.
Selecting Your Stack
There’s no common reply right here, however the choice tree is pretty clear as soon as you understand what you’re constructing:
- RAG utility: begin with RAGAS for the retrieval-specific metrics (faithfulness, context precision, context recall). Add DeepEval alongside it should you want broader protection of toxicity, bias, and normal correctness in the identical take a look at suite.
- Basic chatbot or content-generation characteristic with CI necessities: DeepEval, run as a part of your current pytest suite, gating merges on a high quality threshold.
- Immediate iteration throughout a number of fashions, or safety/red-teaming: Promptfoo. Its YAML-driven config makes multi-model comparability quick, and its built-in assault suite is essentially the most full open-source possibility for adversarial testing.
- Manufacturing tracing and human overview after deployment: LangSmith in case your stack is already constructed on LangChain or LangGraph; Braintrust in case your stack is extra heterogeneous and also you desire a platform that isn’t coupled to 1 framework.
The sample skilled groups converge on is 2 instruments, not one: a light-weight framework for CI-time gating, paired with a platform for ongoing monitoring, regression monitoring, and the human annotation that no automated metric totally replaces.
Conclusion
No framework right here wins outright, as a result of they’re not fixing the identical downside. RAGAS, DeepEval, and Promptfoo every match a special form of analysis want, and the actual architectural choice is often “which two of those do I run collectively,” not “which one is greatest.”
The larger threat isn’t selecting the unsuitable instrument from this listing; it’s trusting no matter rating an LLM choose produces with out checking it for the biases documented above. Place bias, self-preference, and verbosity bias aren’t edge instances buried in a tutorial paper; they’re measurable results that present up in atypical choose setups, together with those contained in the very frameworks this text simply walked by way of. The frameworks provide the scoring mechanism. The audit behavior within the position-bias part is what makes the ensuing quantity price trusting.
