The Precisely as Designed. The Reply Was Nonetheless Flawed.
I need to let you know concerning the second I finished trusting retrieval scores.
I used to be working a question in opposition to a data base I had constructed rigorously. Good chunking. Hybrid search. Reranking. The highest-k paperwork got here again with cosine similarities as excessive as 0.86. Each indicator stated the pipeline was working. I handed these paperwork to a QA mannequin, acquired a assured reply, and moved on.
The reply was fallacious.
Not hallucinated-wrong. Not retrieval-failed-wrong. The proper paperwork had come again. Each of them. A preliminary earnings determine and the audited revision that outmoded it, sitting aspect by aspect in the identical context window. The mannequin learn each, selected one, and reported it with 80% confidence. It had no mechanism to inform me it had been requested to referee a dispute it was by no means designed to evaluate.
That’s the failure mode this text is about. It doesn’t present up in your retrieval metrics. It doesn’t set off your hallucination detectors. It lives within the hole between context meeting and technology — the one step within the RAG pipeline that nearly no person evaluates.
I constructed a reproducible experiment to isolate it. All the things on this article runs on a CPU in about 220 MB. No API key. No cloud. No GPU. The output you see within the terminal screenshots is unmodified.
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
What the Experiment Checks
The setup is intentionally medical. Three questions. One data base containing three conflicting doc pairs that make immediately contradictory claims about the identical reality. Retrieval is tuned to return each conflicting paperwork each time.
The query is just not whether or not retrieval works. It does. The query is: what does the mannequin do whenever you hand it a contradictory temporary and ask it to reply with confidence?
The reply, as you will notice, is that it picks a aspect. Silently. Confidently. With out telling you it had a option to make.
Three Eventualities, Every Drawn from Manufacturing
State of affairs A — The restatement no person informed the mannequin about
An organization’s This autumn earnings launch experiences annual income of $4.2M for fiscal 12 months 2023. Three months later, exterior auditors restate that determine to $6.8M. Each paperwork dwell within the data base. Each are listed. When somebody asks “What was Acme Corp’s income for fiscal 12 months 2023?” — each come again, with similarity scores of 0.863 and 0.820 respectively.
The mannequin solutions $4.2M.
It selected the preliminary determine over the audited revision as a result of the preliminary doc scored marginally larger in retrieval. Nothing concerning the reply indicators {that a} extra authoritative supply disagreed.
State of affairs B — The coverage replace that arrived too late
A June 2023 HR coverage mandates three days per week in-office. A November 2023 revision explicitly reverses it — absolutely distant is now permitted. Each paperwork are retrieved (similarity scores 0.806 and 0.776) when an worker asks concerning the present distant work coverage.
The mannequin solutions with the June coverage. The stricter, older rule. The one which now not applies.
State of affairs C — The API docs that by no means acquired deprecated
Model 1.2 of an API reference states a charge restrict of 100 requests per minute. Model 2.0, printed after an infrastructure improve, raises it to 500. Each are retrieved (scores 0.788 and 0.732).
The mannequin solutions 100. A developer utilizing this reply to configure their charge limiter will throttle themselves to one-fifth of their precise allowance.
None of those are edge instances. Each manufacturing data base accumulates precisely these patterns over time: monetary restatements, coverage revisions, versioned documentation. The pipeline has no layer that detects or handles them.
Working the Experiment
pip set up -r necessities.txt
python rag_conflict_demo.pynecessities.txt
sentence-transformers>=2.7.0 # all-MiniLM-L6-v2 (~90 MB)
transformers>=4.40.0 # deepset/minilm-uncased-squad2 (~130 MB)
torch>=2.0.0 # CPU-only is ok
numpy>=1.24.0
colorama>=0.4.6Two fashions. One for embeddings, one for extractive QA. Each obtain robotically on first run and cache regionally. Complete: ~220 MB. No authentication required.
Section 1: What Naive RAG Does
Right here is the unmodified terminal output from Section 1 — customary RAG with no battle dealing with:
────────────────────────────────────────────────────────────────────
NAIVE | State of affairs A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal 12 months 2023?
Reply : $4.2M
Confidence : 80.3%
Battle : YES — see warning
Sources retrieved
[0.863] This autumn-2023-Earnings-Launch (2024-01-15)
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle pairs
fin-001 ↔ fin-002
numerical contradiction (topic_sim=0.83)
[Q4-2023-Earnings-Release: {'$4.2M'}] vs [2023-Annual-Report-Revised: {'$6.8M'}]
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | State of affairs B — Coverage Battle
────────────────────────────────────────────────────────────────────
Question : What's the present distant work coverage for workers?
Reply : all staff are required to be current within the workplace
a minimal of three days per week
Confidence : 78.3%
Battle : YES — see warning
Sources retrieved
[0.806] HR-Coverage-June-2023 (2023-06-01)
[0.776] HR-Coverage-November-2023 (2023-11-15)
[0.196] HR-Coverage-November-2023 (2023-11-15)
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
NAIVE | State of affairs C — Technical Battle
────────────────────────────────────────────────────────────────────
Question : What's the API charge restrict for the usual tier?
Reply : 100 requests per minute
Confidence : 81.0%
Battle : YES — see warning
Sources retrieved
[0.788] API-Reference-v1.2 (2023-02-10)
[0.732] API-Reference-v2.0 (2023-09-20)
[0.383] API-Reference-v2.0 (2023-09-20)
────────────────────────────────────────────────────────────────────
Three questions. Three fallacious solutions. Confidence between 78% and 81% on each considered one of them.
Discover what is going on within the logs earlier than every response:
09:02:20 | WARNING | Battle detected: {('fin-001', 'fin-002'): "numerical contradiction..."}
09:02:24 | WARNING | Battle detected: {('hr-001', 'hr-002'): "contradiction sign asymmetry..."}
09:02:25 | WARNING | Battle detected: {('api-001', 'api-002'): "contradiction sign asymmetry..."}The conflicts are detected. They’re logged. After which, as a result of resolve_conflicts=False, the pipeline passes the total contradictory context to the mannequin and solutions anyway. That warning goes nowhere. In a manufacturing system with no battle detection layer, you wouldn’t even get the warning.
Why the Mannequin Behaves This Manner
This requires a second of rationalization, as a result of the mannequin is just not damaged. It’s doing precisely what it was educated to do.
deepset/minilm-uncased-squad2 is an extractive QA mannequin. It reads a context string and selects the span with the very best mixed start-logit and end-logit rating. It has no output class for “I see two contradictory claims.” When the context incorporates each $4.2M and $6.8M, the mannequin computes token-level scores throughout your complete string and selects whichever span wins.
That choice is pushed by components that don’t have anything to do with correctness [8]. The 2 main drivers are:
Place bias. Earlier spans within the context obtain marginally larger consideration scores as a result of encoder structure. The preliminary doc ranked larger in retrieval and subsequently appeared first.
Language energy. Direct declarative statements (“income of $4.2M”) outscore hedged or conditional phrasing (“following restatement… is $6.8M”).
A 3rd contributing issue is lexical alignment — spans whose vocabulary overlaps extra intently with the query tokens rating larger no matter whether or not the underlying declare is present or authoritative.
Critically, what the mannequin does not take into account in any respect: supply date, doc authority, audit standing, or whether or not one declare supersedes one other. These indicators are merely invisible to the extractive mannequin.
The identical dynamic performs out in generative LLMs, however much less visibly — the mannequin paraphrases slightly than extracting verbatim spans, so the fallacious reply is wearing fluent prose. The mechanism is similar. Joren et al. (2025) display at ICLR 2025 that frontier fashions together with Gemini 1.5 Professional, GPT-4o, and Claude 3.5 regularly produce incorrect solutions slightly than abstaining when retrieved context is inadequate to reply the question — and that this failure is just not mirrored within the mannequin’s expressed confidence.
The failure is just not a mannequin deficiency. It’s an architectural hole: the pipeline has no stage that detects contradictions earlier than handing context to technology.
Constructing the Battle Detection Layer
The detector sits between retrieval and technology. It examines each pair of retrieved paperwork and flags contradictions earlier than the QA mannequin sees the context. Crucially, embeddings for all retrieved paperwork are computed in a single batched ahead go earlier than pair comparability begins — every doc is encoded precisely as soon as, no matter what number of pairs it participates in.
Two heuristics do the work.
Heuristic 1: Numerical Contradiction
Two topic-similar paperwork that include non-overlapping significant numbers are flagged. The implementation filters out years (1900–2099) and naked small integers (1–9), which seem ubiquitously in enterprise textual content and would generate fixed false positives if handled as declare values.
@classmethod
def _extract_meaningful_numbers(cls, textual content: str) -> set[str]:
outcomes = set()
for m in cls._NUM_RE.finditer(textual content):
uncooked = m.group().strip()
numeric_core = re.sub(r"[$€£MBK%,]", "", uncooked, flags=re.IGNORECASE).strip()
attempt:
val = float(numeric_core)
besides ValueError:
proceed
if 1900 <= val <= 2099 and "." not in numeric_core:
proceed # skip years
if val < 10 and re.fullmatch(r"d+", uncooked):
proceed # skip naked small integers
outcomes.add(uncooked)
return outcomesUtilized to State of affairs A: fin-001 yields {'$4.2M'}, fin-002 yields {'$6.8M'}. Empty intersection — battle detected.
Heuristic 2: Contradiction Sign Asymmetry
Two paperwork discussing the identical subject, the place one incorporates contradiction tokens the opposite doesn’t, are flagged. The token set splits into two teams stored as separate frozenset objects:
_NEGATION_TOKENS: “not”, “by no means”, “no”, “can not”, “doesn’t”, “isn’t”, and associated kinds_DIRECTIONAL_TOKENS: “elevated”, “decreased”, “diminished”, “eradicated”, “eliminated”, “discontinued”
These are unioned into CONTRADICTION_SIGNALS. Protecting them separate makes domain-specific tuning easy — a authorized corpus would possibly want a broader negation set; a changelog corpus would possibly want extra directional tokens.
Utilized to State of affairs B: hr-002 incorporates “no” (from “now not required”); hr-001 doesn’t. Asymmetry detected. Utilized to State of affairs C: api-002 incorporates “elevated”; api-001 doesn’t. Asymmetry detected.
Each heuristics require topic_sim >= 0.68 earlier than firing. This threshold gates out unrelated paperwork that occur to share a quantity or a negation phrase. The 0.68 worth was calibrated for this doc set with all-MiniLM-L6-v2 — deal with it as a place to begin, not a common fixed. Totally different embedding fashions and totally different domains would require recalibration.
The Decision Technique: Cluster-Conscious Recency
When conflicts are detected, the pipeline resolves them by maintaining probably the most lately timestamped doc from every battle cluster. The important thing design determination is cluster-aware.
A top-k outcome might include a number of unbiased battle clusters — two monetary paperwork disagreeing on income and two API paperwork disagreeing on charge limits, all in the identical top-3 outcome. A naive strategy — maintain solely the one most up-to-date doc from the mixed conflicting set — would silently discard the successful doc from each cluster besides probably the most lately printed one general.
As an alternative, the implementation builds a battle graph, finds linked elements by way of iterative DFS, and resolves every part independently:
@staticmethod
def _resolve_by_recency(
contexts: checklist[RetrievedContext],
battle: ConflictReport,
) -> checklist[RetrievedContext]:
# Construct adjacency checklist
adj: dict[str, set[str]] = defaultdict(set)
for a_id, b_id in battle.conflict_pairs:
adj[a_id].add(b_id)
adj[b_id].add(a_id)
# Linked elements by way of iterative DFS
visited: set[str] = set()
clusters: checklist[set[str]] = []
for begin in adj:
if begin not in visited:
cluster: set[str] = set()
stack = [start]
whereas stack:
node = stack.pop()
if node not in visited:
visited.add(node)
cluster.add(node)
stack.lengthen(adj[node] - visited)
clusters.append(cluster)
all_conflicting_ids = set().union(*clusters) if clusters else set()
non_conflicting = [c for c in contexts if c.document.doc_id not in all_conflicting_ids]
resolved_docs = []
for cluster in clusters:
cluster_ctxs = [c for c in contexts if c.document.doc_id in cluster]
# ISO-8601 timestamps type lexicographically — max() offers most up-to-date
greatest = max(cluster_ctxs, key=lambda c: c.doc.timestamp)
resolved_docs.append(greatest)
return non_conflicting + resolved_docsNon-conflicting paperwork go via unchanged. Every battle cluster contributes precisely one winner.
Section 2: What Battle-Conscious RAG Does
────────────────────────────────────────────────────────────────────
RESOLVED | State of affairs A — Numerical Battle
────────────────────────────────────────────────────────────────────
Question : What was Acme Corp's annual income for fiscal 12 months 2023?
Reply : $6.8M
Confidence : 79.6%
Battle : RESOLVED
⚠ Conflicting sources detected — reply derived from most up-to-date
doc per battle cluster.
Sources retrieved
[0.820] 2023-Annual-Report-Revised (2024-04-03)
[0.589] Firm-Overview-2024 (2024-01-01)
Battle cluster resolved: stored '2023-Annual-Report-Revised' (2024-04-03),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | State of affairs B — Coverage Battle
────────────────────────────────────────────────────────────────────
Reply : staff are now not required to keep up
a hard and fast in-office schedule
Confidence : 78.0%
Battle : RESOLVED
Battle cluster resolved: stored 'HR-Coverage-November-2023' (2023-11-15),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────
RESOLVED | State of affairs C — Technical Battle
────────────────────────────────────────────────────────────────────
Reply : 500 requests per minute
Confidence : 80.9%
Battle : RESOLVED
Battle cluster resolved: stored 'API-Reference-v2.0' (2023-09-20),
discarded 1 older doc(s).
────────────────────────────────────────────────────────────────────
Three questions. Three appropriate solutions. The arrogance scores are virtually an identical to Section 1 — 78–81% — which underscores the unique level: confidence was by no means the sign that one thing had gone fallacious. It nonetheless is just not. The one factor that modified is the structure.
What the Heuristics Can not Catch
I need to be exact concerning the failure envelope, as a result of a way that understates its personal limitations is just not helpful.
Paraphrased conflicts. The heuristics catch numerical variations and express contradiction tokens. They won’t catch “the service was retired” versus “the service is presently obtainable.” That may be a actual battle with no numeric distinction and no negation token. For these, a Pure Language Inference mannequin — cross-encoder/nli-deberta-v3-small at ~80 MB — can rating entailment versus contradiction between sentence pairs. That is the extra sturdy path described within the tutorial literature (Asai et al., 2023), and the ConflictDetector class is designed to be prolonged on the _pair_conflict_reason technique for precisely this function.
Non-temporal conflicts. Recency-based decision is acceptable for versioned paperwork and coverage updates. It’s not applicable for skilled opinion disagreements (the minority view could also be appropriate), cross-methodology information conflicts (recency is irrelevant), or multi-perspective queries (the place surfacing each views is the proper response). In these instances, the ConflictReport information construction offers the uncooked materials to construct a distinct response — surfacing each claims, flagging for human overview, or asking the person for clarification.
Scale. Pair comparability is O(k²) in retrieved paperwork. For okay=3 that is trivial; for okay=20 it’s nonetheless high quality. For pipelines retrieving okay=100 or extra, pre-indexing recognized battle pairs or cluster-based detection turns into essential.
The place the Analysis Neighborhood Is Taking This
What you might have seen here’s a sensible heuristic approximation of an issue that lively analysis is attacking at a way more subtle degree.
Cattan et al. (2025) launched the CONFLICTS benchmark — the primary particularly designed to trace how fashions deal with data conflicts in life like RAG settings. Their taxonomy identifies 4 battle classes — freshness, conflicting opinions, complementary info, and misinformation — every requiring distinct mannequin behaviour. Their experiments present that LLMs regularly fail to resolve conflicts appropriately throughout all classes, and that explicitly prompting fashions to motive about potential conflicts considerably improves response high quality, although substantial room for enchancment stays.
Ye et al. (2026) launched TCR (Clear Battle Decision), a plug-and-play framework that disentangles semantic relevance from factual consistency by way of twin contrastive encoders. Self-answerability estimation gauges confidence within the mannequin’s parametric reminiscence, and the ensuing scalar indicators are injected into the generator by way of light-weight soft-prompt tuning. Throughout seven benchmarks, TCR improves battle detection by 5–18 F1 factors whereas including solely 0.3% parameters.
Gao et al. (2025) launched CLEAR (Battle-Localized and Enhanced Consideration for RAG), which probes LLM hidden states on the sentence illustration degree to detect the place conflicting data manifests internally. Their evaluation reveals that data integration happens hierarchically and that conflicting versus aligned data displays distinct distributional patterns inside sentence-level representations. CLEAR makes use of these indicators for conflict-aware fine-tuning that guides the mannequin towards correct proof integration.
The constant discovering throughout all of this work matches what this experiment demonstrates immediately: retrieval high quality and reply high quality are distinct dimensions, and the hole between them is bigger than the neighborhood has traditionally acknowledged.
The distinction between that analysis and this text is 220 MB and no authentication.
What You Ought to Really Do With This
1. Add a battle detection layer earlier than technology. The ConflictDetector class is designed to drop into an present pipeline on the level the place you assemble your context string. Even the 2 easy heuristics right here will catch the patterns that seem most frequently in enterprise corpora: restatements, coverage updates, versioned documentation.
2. Distinguish battle sorts earlier than resolving. A temporal battle (use the newer doc) is a distinct downside from a factual dispute (flag for human overview) or an opinion battle (floor each views). A single decision technique utilized blindly creates new failure modes.
3. Log each ConflictReport. After every week of manufacturing visitors you’ll know the way usually your particular corpus generates conflicting retrieved units, which doc pairs battle most regularly, and what question patterns set off conflicts. That information is extra actionable than any artificial benchmark.
4. Floor uncertainty whenever you can not resolve it. The proper reply to an unresolvable battle is to not choose one and conceal the selection. The warning discipline in RAGResponse is there exactly to assist responses like: “I discovered conflicting info on this subject. The June 2023 coverage states X; the November 2023 replace states Y. The November doc is more moderen.”
Working the Full Demo
# Full output with INFO logs
python rag_conflict_demo.py
# Demo output solely (suppress mannequin loading logs)
python rag_conflict_demo.py --quiet
# Run unit exams with out downloading fashions
python rag_conflict_demo.py --test
# Plain terminal output for log seize / CI
python rag_conflict_demo.py --no-colorAll output proven on this article is unmodified output from an area Home windows machine working Python 3.9+ in a digital surroundings. The code and output are absolutely reproducible by any reader with the listed dependencies put in.
The Takeaway
The retrieval downside is basically solved. Vector search is quick, correct, and well-understood. The neighborhood has spent years optimising it.
The context-assembly downside is just not solved. No one is measuring it. The hole between “appropriate paperwork retrieved” and “appropriate reply produced” is actual, it’s common, and it produces assured fallacious solutions with no sign that something went fallacious.
The repair doesn’t require a bigger mannequin, a brand new structure, or extra coaching. It requires one extra pipeline stage, working on embeddings you have already got, at zero marginal latency.
The experiment above runs in about thirty seconds on a laptop computer. The query is whether or not your manufacturing system has the equal layer — and if not, what it’s silently answering fallacious proper now.
References
[1] Ye, H., Chen, S., Zhong, Z., Xiao, C., Zhang, H., Wu, Y., & Shen, F. (2026). Seeing via the battle: Clear data battle dealing with in retrieval-augmented technology. arXiv:2601.06842. https://doi.org/10.48550/arXiv.2601.06842
[2] Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023). Self-RAG: Studying to retrieve, generate, and critique via self-reflection. arXiv:2310.11511. https://doi.org/10.48550/arXiv.2310.11511
[3] Cattan, A., Jacovi, A., Ram, O., Herzig, J., Aharoni, R., Goldshtein, S., Ofek, E., Szpektor, I., & Caciularu, A. (2025). DRAGged into conflicts: Detecting and addressing conflicting sources in search-augmented LLMs. arXiv:2506.08500. https://doi.org/10.48550/arXiv.2506.08500
[4] Gao, L., Bi, B., Yuan, Z., Wang, L., Chen, Z., Wei, Z., Liu, S., Zhang, Q., & Su, J. (2025). Probing latent data battle for devoted retrieval-augmented technology. arXiv:2510.12460. https://doi.org/10.48550/arXiv.2510.12460
[5] Jin, Z., Cao, P., Chen, Y., Liu, Ok., Jiang, X., Xu, J., Li, Q., & Zhao, J. (2024). Tug-of-war between data: Exploring and resolving data conflicts in retrieval-augmented language fashions. arXiv:2402.14409. https://doi.org/10.48550/arXiv.2402.14409
[6] Joren, H., Zhang, J., Ferng, C.-S., Juan, D.-C., Taly, A., & Rashtchian, C. (2025). Adequate context: A brand new lens on retrieval augmented technology techniques. arXiv:2411.06037. https://doi.org/10.48550/arXiv.2411.06037
[7] Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., … & Kiela, D. (2020). Retrieval-augmented technology for knowledge-intensive NLP duties. arXiv:2005.11401. https://doi.org/10.48550/arXiv.2005.11401
[8] Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023). When to not belief language fashions: Investigating effectiveness of parametric and non-parametric recollections. arXiv:2212.10511. https://doi.org/10.48550/arXiv.2212.10511
[9] Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings utilizing Siamese BERT-networks. arXiv:1908.10084. https://doi.org/10.48550/arXiv.1908.10084
[10] Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Data conflicts for LLMs: A survey. arXiv:2403.08319. https://doi.org/10.48550/arXiv.2403.08319
[11] Xie, J., Zhang, Ok., Chen, J., Lou, R., & Su, Y. (2023). Adaptive chameleon or cussed sloth: Revealing the habits of huge language fashions in data conflicts. arXiv:2305.13300. https://doi.org/10.48550/arXiv.2305.13300
Full Supply Code: https://github.com/Emmimal/rag-conflict-demo
Fashions Used
Each fashions obtain robotically on first run and cache regionally. No API key or HuggingFace authentication is required.
Disclosure
All code was written, debugged, and validated by the writer via a number of iterations of actual execution. All terminal output on this article is unmodified output from an area Home windows machine working Python 3.9+ in a digital surroundings. The code and output are absolutely reproducible by any reader with the listed dependencies put in.
The writer has no monetary relationship with Hugging Face, deepset, or any organisation referenced on this article. Mannequin and library decisions have been made solely on the idea of measurement, licence, and CPU compatibility.
