I had a handful of parallel Claude brokers write me up a method folder for a facet venture. The pull request regarded nice: ten paperwork, all neatly cross-linked, and it learn properly. Then the automated reviewer posted its findings, and Claude, to its credit score, didn’t object:
The complete audit got here again: 30 of the 95 relative hyperlinks pointed at recordsdata that didn’t exist. ./market.md as a substitute of market-analysis.md, ./monetization.md as a substitute of business-model.md, some recordsdata had been even made up fully! Skimming that PR as a human, I’d have accepted it, since each considered one of them regarded believable. Fortunately, my automated reviewer caught it as a result of it resolved each hyperlink as a substitute of judging whether or not the names “vibed” proper.
To be honest, Claude doesn’t have any unhealthy intent behind this. A language mannequin merely generates essentially the most believable continuation, and typically essentially the most believable continuation is fiction delivered with full confidence. From the studying facet, although, that distinction brings little consolation. Assured and incorrect reads precisely like assured and proper.
This submit is concerning the system I’ve in place to assist me overview Claude’s errors and hallucinations: a Codex agent working in GitHub Actions that critiques each pull request. It walks you thru my setup, why cross-provider overview beats each self-review and a human-reads-everything overview, methods to implement reminiscence and accountability within the overview loop, the fee and reliability learnings I gathered alongside the way in which, and the precise code I exploit to make this work.
Agent throughput broke the human overview loop
Hallucinations by themselves are a manageable drawback, since a really cautious overview or practical evaluation would catch them. Nonetheless, what broke over the previous yr is the quantity that wants reviewing. A single engineer now runs a number of agent periods in parallel, and every of them can produce a thousand-line diff within the time it takes to refill your espresso. Producing code has turn out to be the quick and straightforward half; reviewing it actually is what eats the day. That makes the human reviewer the bottleneck of the entire supply pipeline, and the bottleneck is at all times the following factor it’s best to automate.
That bottleneck invitations two acquainted failure modes. Both you approve regardless of the brokers produce with out actually studying it, and the damaged hyperlinks, phantom parameters, and subtly incorrect edge instances ship to manufacturing. Otherwise you nonetheless insist on studying each line your self, and your quick and good AI brokers work even lower than you do. The primary is negligent, the second wasteful, and most groups oscillate between each relying on the deadline.
The conclusion I maintain touchdown on: our overview course of has to scale with the velocity we’re producing code. The human ingredient stays, however the human reviewer’s focus ought to go to the elements that really matter. The grind of checking whether or not hyperlinks resolve, whether or not the brand new parameter exists, whether or not there’s a bug or vulnerability, and whether or not the code adjustments match what was initially requested, all of that has to run robotically on the identical cadence because the coding brokers, even earlier than an individual ever spends a second reviewing the code.
Perception: hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first overview cross needs to be automated.
Decide a reviewer that doesn’t share the writer’s blind spots
The plain first thought to forestall these errors is to have the agent overview its personal work… however additionally it is the weakest one. A hallucination that made it into the diff is, by definition, believable to the mannequin that produced it. Asking the identical weights to search out errors within the code, particularly throughout the identical already-biased coding session, often reproduces the precise blind spots that allow the errors by way of.
This instinct has analysis behind it. Panickssery et al. confirmed at NeurIPS 2024 that LLM evaluators acknowledge and favor their very own generations: the higher a mannequin is at figuring out its personal output, the extra generously it scores it. Observe-up work ties this self-preference bias to familiarity: AI judges rating textual content increased when it reads as predictable to them, and nothing reads as extra predictable to a mannequin than the textual content it simply wrote itself.
Completely different LLM suppliers practice on totally different knowledge, with totally different recipes and totally different suggestions loops. No supplier manages to remove errors, however the type of errors a mannequin makes traces again to the way it was educated, and that differs per supplier. That decorrelation is precisely what you need in your overview agent. Within the yr that I’ve used this setup, OpenAI’s Codex has constantly flagged points in Claude-written PRs that Claude’s personal overview handed with out blinking, from invented filenames to total code blocks that had nothing to do with the preliminary request, or that launched outright safety violations.
One conclusion you can’t take from that is that Codex is the higher engineer and it’s best to hand it the keyboard as a substitute of Claude. Codex misfires pretty frequently, too! Simply in a distinct course than Claude does. The facility sits in working multimodel: combining suppliers throughout the identical workflow, the place the fashions fill one another’s gaps.
One other incorrect conclusion is that Codex’s critiques are excellent and could be accepted blindly. Hallucinations occur in critiques too, so deal with the overview as advisory. Difficult the writer (be it Claude or your self) places the concentrate on potential blind spots, which is the proper sign to zoom in and mirror. It stays the writer’s job to personal the ultimate high quality, and typically meaning writing again to Codex that its overview was incorrect. Extra on this mechanism later within the submit.
Perception: an writer’s bugs are by definition believable to the writer, and analysis confirms LLMs are positively biased towards their very own output. Evaluate worth comes from decorrelation, by utilizing a mannequin from a distinct supplier that fails in a different way, and totally different is what catches what you miss.
The pipeline: one LLM name wrapped in bizarre engineering
Conceptually, an LLM reviewer could be written in a single line: “right here is the code diff, please overview it”. Each OpenAI and Anthropic ship hosted shortcuts for precisely that: OpenAI has a GitHub overview integration that auto-reviews PRs, and Anthropic has claude-code-action. They’re nice beginning factors, however I desire to run my very own motion as a substitute, as a result of I wish to personal the immediate, the overview lifecycle, the merge gating, and the mannequin pin. All these points matter when your day-to-day turns into managing and reviewing brokers.
So what am I utilizing then, precisely? Two recordsdata that reside within the repository they overview:
.github/
├── workflows/
│ └── pr.yml # orchestration: triggers, context, retries, posting
└── actions/
└── codex-pr-review/
└── motion.yml # the reviewer: pinned Codex name + overview immediateThe runtime circulate is brief. Each time a PR opens or receives a push, pr.yml checks out the code, collects the PR’s full dialog historical past, and arms each to the composite motion. The motion runs Codex over the diff and returns its findings as a single message. The workflow then posts that message as a single, repeatedly up to date remark within the PR dialog, and publishes a commit standing that stays purple whereas findings stay unresolved: the go/no-go sign sitting subsequent to the remainder of CI.
On the heart of all of it sits one deceptively easy step, the one line that really touches an LLM:
- makes use of: openai/codex-action@v1
with:
openai-api-key: ${{ inputs.openai-api-key }}
immediate: |
# the overview contract: what to overview, methods to observe findings,
# and methods to report again (unpacked within the subsequent part)Every little thing else exists to run that single step reliably, and to verify its suggestions truly lands on the PR. Right here is the whole workflow:
title: PR Reviewer
on:
pull_request:
branches: [dev] # all work rides function branches: function -> dev -> foremost
sorts: [opened, synchronize, reopened]
# a brand new push cancels the in-flight (costly) overview of the earlier commit
concurrency:
group: pr-reviewer-${{ github.occasion.pull_request.quantity }}
cancel-in-progress: true
jobs:
codex:
if: github.occasion.pull_request.person.login != 'dependabot[bot]'
runs-on: ubuntu-latest
timeout-minutes: 15 # regular overview takes a couple of minutes, so 15m is greater than sufficient
permissions: # the token can learn code and submit suggestions, however can by no means push or merge
contents: learn
points: write
pull-requests: write
statuses: write
steps:
# the /head ref updates synchronously on each push;
# the auto-generated merge ref can race the checkout
- makes use of: actions/checkout@v7
with:
ref: refs/pull/${{ github.occasion.pull_request.quantity }}/head
# be sure that each diff endpoints exist domestically, so Codex can resolve base...head
- title: Pre-fetch base and head refs for the PR
run: |
git fetch --no-tags origin
${{ github.occasion.pull_request.base.ref }}
+refs/pull/${{ github.occasion.pull_request.quantity }}/head
# suggestions lives on three surfaces: concern feedback, critiques, and inline feedback
- title: Fetch PR dialog
id: dialog
makes use of: actions/github-script@v9
with:
result-encoding: string
script: |
const prNumber = context.payload.pull_request.quantity;
const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };
const [comments, reviews, inline] = await Promise.all([
github.paginate(github.rest.issues.listComments,
{ ...opts, issue_number: prNumber }),
github.paginate(github.rest.pulls.listReviews,
{ ...opts, pull_number: prNumber }),
github.paginate(github.rest.pulls.listReviewComments,
{ ...opts, pull_number: prNumber }),
]);
const all = [
...comments.map(c => ({
date: c.created_at,
text: `@${c.user.login}: ${c.body}`,
})),
...reviews.filter(r => r.body).map(r => ({
date: r.submitted_at,
text: `@${r.user.login} (review ${r.state}): ${r.body}`,
})),
...inline.map(c => ({
date: c.created_at,
text: `@${c.user.login} (on ${c.path}): ${c.body}`,
})),
];
all.type((a, b) => new Date(a.date) - new Date(b.date));
return all.map(e => e.textual content).be part of('n---n');
# secrets and techniques don't journey with the yml, talk loudly on lacking keys
- title: Fail quick on lacking OPENAI_API_KEY
env:
OPENAI_API_KEY: ${{ secrets and techniques.OPENAI_API_KEY }}
run: |
if [ -z "${OPENAI_API_KEY}" ]; then
echo "::error::OPENAI_API_KEY secret will not be set for this repository."
exit 1
fi
# retry wrappers can not wrap `makes use of:` steps, so retry manually: transient
# failures (a 401 from a contemporary runner token, a hiccup reaching OpenAI)
# shouldn't flip the entire run purple
- title: Run Codex (try 1)
id: codex_try1
makes use of: ./.github/actions/codex-pr-review
continue-on-error: true
with:
openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}
dialog: ${{ steps.dialog.outputs.outcome }}
# give transient failures a second to clear earlier than retrying
- title: Wait earlier than Codex retry
if: steps.codex_try1.end result == 'failure'
run: sleep 20
- title: Run Codex (try 2)
id: codex_try2
if: steps.codex_try1.end result == 'failure'
makes use of: ./.github/actions/codex-pr-review
with:
openai-api-key: ${{ secrets and techniques.OPENAI_API_KEY }}
dialog: ${{ steps.dialog.outputs.outcome }}
- title: Submit the overview and publish a standing
makes use of: actions/github-script@v9
env:
REVIEW: >-
${}
with:
script: |
const MARKER = '';
const opts = { proprietor: context.repo.proprietor, repo: context.repo.repo };
const prNumber = context.payload.pull_request.quantity;
const uncooked = course of.env.REVIEW;
if (!uncooked) return;
const physique = `${MARKER}n${uncooked}`;
// one canonical remark, edited in place on each push
const feedback = await github.paginate(
github.relaxation.points.listComments, { ...opts, issue_number: prNumber });
const current = feedback.discover(c =>
c.person?.login === 'github-actions[bot]' && c.physique?.contains(MARKER));
const posted = current
? await github.relaxation.points.updateComment(
{ ...opts, comment_id: current.id, physique })
: await github.relaxation.points.createComment(
{ ...opts, issue_number: prNumber, physique });
// go/no-go: a commit standing that stays purple whereas findings are open
// (a lacking trailer reads as "overview posted", by no means as zero findings)
const m = uncooked.match(/`, the place N counts
the unresolved rows and M what number of of these are severity Excessive.
All the time embody it, even when N is 0: CI parses it into the
standing examine.
Pull request title and physique:
----
${{ github.occasion.pull_request.title }}
${{ github.occasion.pull_request.physique }}
Dialog historical past:
----
${{ inputs.dialog }}Three design selections carry the burden right here.
- The overview is stateful. The Suggestions Abstract desk tracks each discovering ever raised on the PR, resolved or not, throughout pushes. Mixed with the edit-in-place remark from the workflow, a PR carries precisely one overview artifact that displays your complete lifecycle, as a substitute of 5 stale feedback that every mirror an replace within the overview course of. Anybody opening the PR, human or agent, sees the total historical past at a look.
- Declines are honored. As a result of Codex reads the entire dialog, the writer can reject a discovering in writing, and the rejection sticks throughout overview cycles. That is the mechanism that makes an imperfect reviewer workable: false positives price one written reply as a substitute of an infinite nag loop, and the objection stays on file for the following reader.
- The decision is machine-readable. The trailing
line will get parsed by the workflow right into aCodex overviewcommit standing. The result’s two impartial indicators on each PR: the workflow examine tells you the overview ran, and the standing tells you whether or not something continues to be open. Crimson standing means unresolved findings; inexperienced means all the things is fastened or explicitly declined. I intentionally maintain the standing non-blocking, because it solely capabilities to tell. The choice to merge or not stays open, and stays human.
Notice how this contract asks for greater than typo looking. Codex critiques the diff in opposition to the PR’s acknowledged intent, ranks each discovering by severity, and attaches a concrete proposed repair, the type of overview that frequently surfaces architecture-level points no linter would ever produce.
Perception: construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.
Shut the loop: the PR turns inexperienced earlier than you even look
The reviewer is just half the story while you take a look at the larger engineering image. The opposite half is educating the writer agent to reply to it, so the overview round-trip doesn’t turn out to be your job (bear in mind: at all times attempt to take away the bottleneck). In my repos I exploit three Claude Code abilities to help me, written as plain markdown instruction recordsdata underneath .claude/abilities/:
/pr-opencreates the PR: analyzes the department’s commits, runs validation domestically, writes an trustworthy title and physique, and targets the correct base. The PR physique issues greater than it appears, because the reviewer reads it because the assertion of intent it critiques in opposition to. I’ve it spell out the total function request and reference the Linear ticket it implements, so each the reviewer and any human arriving later get the whole image with out leaving the PR./pr-iteratehandles one overview spherical: sweeps all three suggestions surfaces, then triages each merchandise right into a repair, a decline with reasoning, or a defer. Fixes are carried out and validated domestically; declines turn out to be PR feedback explaining why. The reviewer is advisory, and “ok to merge” is a legitimate verdict as soon as the PR meets its acknowledged scope./pr-babysitis the watchdog round all of it: watch the CI, repair purple checks, run an iterate spherical when new suggestions lands, and loop till each examine is inexperienced and each discovering is fastened or declined. Its terminal state is “merge-ready”, and it stops to report quite than ping-pong commits when the identical examine fails twice on the identical root trigger.
One ordering rule that you simply and your brokers ought to at all times have in mind: reply first, push after. The overview kicks off on each push, and it really works with the context that exists at that precise second. Something not but written down (a decline, a repair abstract, a remark of your individual) doesn’t exist for that overview spherical, and the reviewer will fortunately re-raise each merchandise it can not see a response to.
The tip state is a tidy division of labor. Claude writes and defends, Codex challenges and tracks, the loop runs till the PR is inexperienced with a clear suggestions desk. My queue comprises solely PRs which might be already merge-ready, every carrying a overview path I can audit in a single scroll: what was discovered, what was fastened, and what was declined and why.
Perception: automate either side of the overview. When the writer agent should repair or contest each discovering in writing earlier than it may well push, the human queue solely ever comprises merge-ready PRs with an auditable overview path.
What a round-trip seems like in apply
To make the mechanism concrete, right here is one change from a backend PR of mine. The PR fastened a capability grid that wrote extra cells than a hardcoded batch cap allowed. The primary overview spherical got here again with two findings:
## Suggestions Abstract
| Situation | Severity | Abstract | Resolved |
| ------------------------------- | -------- | ---------------------------------------- | -------- |
| Truncation past the batch cap | Excessive | Cells previous the cap are silently dropped. | ❌ |
| Cap duplicates a platform restrict | Medium | Transfer the cap into app config. | ❌ |
The Codex overview standing turned purple. Claude carried out the primary discovering for actual, then declined the second in a PR remark (actual remark, anonymized):
Addressed the Excessive discovering (hard-coded row cap within the knowledge layer, inflicting avoidable 502s).
_fetch_capacity_gridnow adapts as a substitute of failing: on a brief web page (which, as a result of the grid is dense at precisely days rows/individual, unambiguously means the deployed cap truncated the batch) it halvesbatch_sizeand retries the identical slice, shrinking till pages match. A brief web page solely turns into a loud 502 as soon asbatch_size == 1 [...]On the secondary suggestion (transfer the cap to config): deliberately not carried out. The actual cap lives within the database’s personal hosted API settings, not an app env var. Mirroring it into app config would add a second supply of fact that would silently drift from the database. Adaptive batching removes the necessity to know the precise worth in any respect, which is the stronger repair. Current capacity-grid DB take a look at nonetheless passes; lint and type-check inexperienced.
Reply first, push after, and have Codex re-review:
## Suggestions Abstract
| Situation | Severity | Abstract | Resolved |
| ------------------------------- | -------- | -------------------------------- | -------- |
| Truncation past the batch cap | Excessive | Fastened through adaptive batching. | ✅ |
| Cap duplicates a platform restrict | Medium | Declined: duplicate would drift. | ✅ |
Each rows resolved, one by a repair and one by a written decline that the reviewer accepted and recorded. That second row is your complete mechanism in a single line: no churn was carried out simply to silence a remark, and the reasoning is now a part of the PR’s historical past for whoever reads it subsequent.
The learnings ledger: what it prices and the place it bites
This setup didn’t come collectively in a single go. Most of its particulars grew out of surprises and lifeless ends alongside the way in which, so I’m writing the learnings down right here, in tough order of cash saved, to spare you from driving into the identical partitions:
- Pin the mannequin. The
mannequinenter ofcodex-actionis non-obligatory, and leaving it empty means “no matter Codex presently defaults to”. I realized this on my invoice when the default silently moved to a more moderen, pricier mannequin (gpt-5.5). Pins clear up this drawback, however want upkeep in return; presently I exploitgpt-5.3-codex, which is already marked deprecated on the Codex fashions web page on the time of writing (whereas remaining accessible on the API). I revisit the pin intentionally as a substitute of letting the default resolve for me. - Cap the hassle at
medium. Greater reasoning effort catches marginally extra points for considerably extra time and cash. Velocity issues greater than it seems: the overview sits contained in the agent’s iterate loop, so its latency multiplies throughout each spherical of each PR, and your individual ready time is a part of the value. Medium has been the issues-per-dollar and issues-per-minute candy spot. Sadly, no effort stage catches all the things; that’s what the merge-owning human is for. - Retry the fast failures. A contemporary runner token often 401s, and the primary connection to the API often hiccups. Two makes an attempt with a gated fallback maintain these from turning the workflow purple “for nothing”, which issues as soon as brokers (and your individual belief) deal with purple as an actual sign.
- Cancel outmoded critiques. The
concurrencyblock means solely the most recent commit of a PR will get reviewed. With out it, an agent pushing three fixes in a row buys you three critiques, two of them for code that now not exists. - Evaluate in opposition to the bottom that may obtain the merge. My PRs goal
dev, so the overview diffs in opposition todev, and I maintain the function department synced with that base. A stale department will get diffed in opposition to a base that has moved on, and the overview then experiences phantom findings for adjustments the PR by no means made. - Function branches are the prerequisite. All of this hangs off
pull_requestoccasions. Direct pushes todevorforemostbypass the reviewer, the standing, and the audit path fully. The one-branch-per-change circulate is what makes each change reviewable within the first place. - Assume PR textual content is untrusted enter. Every little thing the reviewer reads (the PR physique, the dialog, the code itself) is a prompt-injection floor, and the agent executes in your runner along with your API key. The least-privilege token caps the blast radius on the GitHub facet, and
codex-actionships sandboxing and safety-strategy knobs for the runner facet. Additionally know that GitHub doesn’t expose secrets and techniques to workflows triggered from forked PRs, so this sample assumes same-repo branches from trusted contributors. Evaluate the belief mannequin earlier than copying it into an open-source repo.
You’re the editor-in-chief now
The purpose of this pipeline was by no means to take away the human from the loop, its intent is to uplevel you. I ended being the primary reader of each diff and have become the one that reads critiques, adjudicates the occasional standoff between two fashions, and owns the requirements written into the overview immediate. When Claude and Codex agree, the PR might be nice. Once they disagree, that disagreement is a precision-guided pointer at precisely the code that deserves my consideration. The place Claude typically acts sloppy, Codex tends to over-engineer, holding that steadiness in your individual arms is a good place to be.
Reviewing the critiques (so meta) is a greater job than proofreading agent output at agent velocity, and it degrades gracefully. On a lazy day I merge inexperienced PRs on the energy of the path, on a cautious day I learn the suggestions desk and spot-check the declines. Both approach, nothing reaches the merge button on one mannequin’s assured say-so.
Key insights from this submit
- Hallucinations are a relentless, throughput is what modified. Any course of that requires a human to learn each line caps your brokers at human studying velocity, so the very first overview cross needs to be automated.
- An writer’s bugs are by definition believable to the writer, and analysis confirms LLMs are positively biased towards their very own output. Evaluate worth comes from decorrelation, by utilizing a mannequin from a distinct supplier that fails in a different way, and totally different is what catches what you miss.
- The mannequin name is the simple half. Reviewer reliability comes from bizarre CI engineering round it.
- Construction beats uncooked mannequin high quality in a reviewer. A lifecycle desk, a decline rule, and a machine-readable verdict flip free-form LLM opinions right into a merge gate with a mechanism for written pushback that makes false positives low-cost.
- Automate either side of the overview. When the writer agent should repair or contest each discovering in writing earlier than it may well push, the human queue solely ever comprises merge-ready PRs with an auditable overview path.
Remaining perception: work multimodel. Your brokers will typically hand you assured fiction, and fashions from totally different suppliers fill one another’s gaps, so that you don’t must. A second-provider reviewer with reminiscence and a contract, sitting between your brokers and your foremost department is a good place to start out, however it isn’t the one one. At any time when one mannequin’s output is about to matter, a second opinion from a distinct lab is the most affordable insurance coverage you should purchase.
Observe me on LinkedIn for bite-sized AI insights, In direction of Information Science for early entry to new posts, or Medium for the total archive.
