What Is Harness Engineering? A Arms-On Introduction

Let me begin with a powerful opinion.

In the event you’re a software program engineer or a knowledge scientist, nothing you possibly can be taught at the moment will do extra on your profession than studying to design agent harnesses.

It took me a short while to really hit, however once I discovered researchers at prime companies hadn’t written code within the final 8 months, it struck me.

We aren’t going to be writing code anymore.

Let’s clearly outline what a “harness” is within the context of AI

There’s a number of confusion, buzzwords and conceptual conflation within the present state. Thus, we want a collective conceptual settlement on what a harness truly is.

You might have seen this equation earlier than:

The LLM is the bottom. Stateless, unchanging, maybe swappable.

The remaining is the “harness”. I record Reminiscence, Instruments, Planning and Motion, however the universe of “extras” to orchestrate an agent’s conduct might be a lot bigger.

Thus, the world of brokers is solely:

The necessary factor to grasp is, the harness is all the pieces we construct round that mannequin to show its capabilities into an operational agent.

It is necessary to comprehend that Codex, Claude Code, amongst others, are simply harnesses

Right here, the LLM powering these instruments is a second class citizen. Codex and Claude Code are agentic coding brokers constructed round language fashions. They mix mannequin inference with a set of mechanisms (file modifying, repls, git, and so on.) that make autonomous software program growth doable.

The facility lies within the cleverness of the harness.

This distinction modifications the place we must always focus our engineering effort.

Suppose you need an agent to refactor a big codebase. You could possibly spend weeks making an attempt to fine-tune a mannequin that’s marginally higher at producing Python.

Alternatively, you may enhance the harness in order that the agent can map the repository, establish dependencies, assemble a take a look at plan, make incremental modifications, run focused checks, diagnose failures, and confirm that unrelated conduct stays unchanged.

The second strategy sometimes produces a a lot bigger enchancment within the general system with no need to improve the underlying LLM.

After all, mannequin functionality nonetheless issues. A harness can’t give a weak mannequin the reasoning potential it lacks. However the efficiency of your agent is not decided by mannequin functionality alone. It relies on how successfully that functionality is utilized to the duty.

Which brings me to the central thesis: when you’re a software program engineer or knowledge scientist, understanding harness engineering is not non-obligatory. It is important to remaining related.

As your position shifts from writing code to producing brokers who write code, designing the harness turns into the paramount talent. It’s worthwhile to create brokers which might function autonomously on issues which (doubtlessly) span many complicated domains. Stated in any other case, you’re not a person contributor, even when you have no direct studies. You’re a supervisor, a supervisor of brokers.

You may’t depend on Claude Code or Codex to resolve your whole issues

These are good general-purpose harnesses. Very helpful, however will fail spectacularly as quickly as you strategy an issue that may be very complicated or outdoors of the overall objective area. In the event you depend on this strategy, not solely will you be spending your time on drift and slop cleansing, you will even be making your self much less related.

A tragic, however true truth: engineers who can solely construct methods utilizing Claude Code or Codex have gotten commodities. You can’t stay aggressive available in the market in case your solely talent is “I understand how to make use of Claude Code.” Frankly, if that is your technique for SWE within the AI period, you are a goal for layoffs.

Your worth lies in designing the harnesses that flip LLMs into methods that merely work.

So, how does one turn out to be a harness engineer?

This isn’t your “construct an agent in 50 strains” submit. You’ve got in all probability seen the tutorial.

Name the mannequin, test whether or not it requested for a instrument, run the instrument, append the outcome, loop till it says it is carried out. It is an effective way to grasp the mechanics, and it really works in a demo. That is agent 101, the only loop that makes an LLM an agent.

whereas True:    response = mannequin(messages)    if response.is_final:        break    outcome = run_tool(response.tool_call)    messages.append(response.tool_call)    messages.append(outcome)

Nonetheless, this loop won’t clear up complicated issues. Complicated issues want reminiscence, instruments, context administration strategies. The complete harness.

That is about understanding the primitives underlying harnesses that really work, so to design your personal.

You’ve got been promoted. You’re not a senior software program engineer. You’re the core methods architect.

That is the important thing mindshift it is advisable make.

In the event you have been beforehand comfy writing software program within the conventional method: writing a spec, designing and implementing capabilities and unit checks, that job is not accessible. That is now the realm of AI brokers.

That is excellent news. You’ve been robotically promoted to a methods architect.

Your position is now designing sensible frameworks that permit brokers to achieve success in day-to-day operations. You are not writing the logic that solves the issue. You are designing methods that allow AI to resolve issues.

These are essentially completely different actions. It’s worthwhile to take into consideration your staff of brokers like junior software program engineers. The place do you anticipate blockers? What instruments do they want? How can you make sure that your suggestions and design imaginative and prescient will get carried out appropriately and reliably with out you needing to micromanage all the main points?

👉 I’ve damaged this introduction into “primitives”: engineering ideas that mix to supply a dependable agent harness. These primitives ought to be considered like constructing blocks. Each harness ought to embrace them (in various compositions)

Primitive #1. Workflows

Earlier than any precise engineering happens, it is advisable design a workflow. That is in any other case often known as “Loop Engineering.”

A workflow for an agent is almost the identical as it’s for a human. It’s the sequence of phases, choice factors and suggestions loops that an agent follows with a purpose to clear up a activity. Designing these workflows is the place your area experience is available in.

For instance, the quintessential knowledge science workflow, as carried out by a human. A stakeholder asks you to construct a prediction mannequin.

1. Outline the issue: What are we predicting, and what counts as success?

2. Discover the information: Perceive the information, establish points, and discover helpful options.

3. Construct the mannequin: Engineer options and prepare an preliminary mannequin.

4. Consider: Check efficiency on unseen knowledge utilizing related metrics.

5. Iterate: Analyze errors, enhance options, attempt completely different fashions, and repeat till efficiency is passable.

6. Ship: Doc the outcomes and deploy the mannequin.

This is identical precept behind profitable engineering.

Break complicated work into phases with verifiable outputs that may be refined with suggestions.

Now, as a substitute of a human executing every activity, we design workflows in a loop such that they are often accomplished by an agent.

Some profitable workflow design patterns:

1. ReAct

2. Plan-then-Execute

3. Reflexion

4. Sub-agents

ReAct

In ReAct: Synergizing Reasoning and Appearing in Language Fashions, ICLR 2023, Yao et al. launched ReAct (Reasoning + Appearing), which is a conceptually easy sample that interleaves the mannequin’s inside reasoning with concrete actions in an atmosphere.

The construction alternates between three parts:

• Thought : the mannequin causes in regards to the present state, what it is aware of, and what it ought to do subsequent

• Motion: the mannequin invokes a instrument or takes a step within the atmosphere

• Statement: the results of that motion is fed again to the mannequin

For harness engineers, ReAct is the muse. Almost each manufacturing agent harness implements some model of this loop. The ReAct loop is the one referenced above. It is simplistically elegant.

Nonetheless, it’s restricted by its linear nature. The agent follows a single thread of reasoning. When it makes a flawed flip, it typically continues down the flawed path moderately than backtracking. It would not have any reminiscence, and it is context degrades rapidly, utilizing ReAct alone could not result in success on complicated issues.

Plan-then-Execute

In Plan-and-Clear up Prompting (ACL 2023), Wang et al. investigated prompting fashions to formulate a plan earlier than executing it. That is what each profitable coding agent does in follow and can be utilized in a number of contexts outdoors of software program engineering.

The sample:

• Planning section: Collect data, discover the codebase, learn docs, establish all places that want modifications, produce a structured plan

• Assessment checkpoint: The plan is verified (by a human, by a validation step, or by a self-review)

• Execution section: Comply with the plan step-by-step, with the harness offering context in regards to the present step and monitoring progress

Each software program engineer is aware of the distinction between leaping into code versus writing a design doc first. Plan-then-Execute is that instinct, formalized as a harness.

If ReAct is the muse for easy workflows, Plan-then-Execute is the muse for complicated workflows.

Reflexion

Shinn et al.’s Reflexion (Reflexion: Language Brokers with Verbal Reinforcement Studying, NeurIPS 2023) is the sample of iterative self-improvement via reflection on failure. The agent makes an attempt a activity, evaluates the outcome, generates verbal suggestions about what went flawed, shops that suggestions in reminiscence, and tries once more with the advantage of that reflection.

That is immediately what occurs in each coding agent when checks fail. The agent writes code, runs checks, sees failures, displays on what went flawed, and revises. Reflexion formalized this and confirmed an 11-point enchancment on HumanEval (80% → 91%) with no mannequin change.

The mannequin makes use of a metacognitive technique to grasp its personal failure, then makes use of that as context to enhance the ultimate outcome.

Sub-agents

The concept is easy: decompose a big activity into smaller subtasks, and delegate every subtask to a specialised agent with its personal instruments and context window. A supervisor manages the delegation, collects outcomes, and synthesizes them right into a remaining output.

The great factor about this sample is that every sub-agent can use a chic workflow as described above. The core limitations listed here are the power of the delegating agent and the robustness of your workflow.

The important thing design selections when working with sub-agents are:

• Scope boundaries: What does every sub-agent personal? Clear possession prevents brokers from stepping on one another’s work.

• Interface contracts: What data does the coordinator move to every sub-agent, and what does it anticipate again? The extra structured these contracts, the extra dependable the system.

• Failure isolation: If a sub-agent fails or produces rubbish, the coordinator must detect this and both retry, reassign, or escalate, with out contaminating the work of different sub-agents.

• Context independence: Every sub-agent ought to have the ability to function with solely the data it’s given. If a sub-agent must learn the complete 200-file repository to do its job, you’ve got scoped it flawed.

These workflow patterns are important, no matter whether or not you are constructing harnesses for software program engineering, knowledge science or another area.

It is also necessary to notice that, these are simply foundational examples. They’re on no account requisite. For instance, you may design a hybrid: A sub-agent delegate system the place every sub-agent makes use of reflexion + a human-in-the-loop step. These patterns are established sufficient for use like constructing blocks.

⚠️ Do not reinvent the wheel. Enhance or modify these foundations as you see match.

Primitive #2: Structure

Evolving a harness from one thing easy, like autoresearch, to one thing extra complicated, like Codex requires foresight and planning. There must be clear architectural and structural boundaries that manage the harness such that the agent can function cleanly and even modify its personal harness if crucial with out creating huge quantities of technical debt that wants a human to wash.

In the event you’ve constructed manufacturing software program in a staff, you have already got intuitions right here.

First, design the scaffolding of your harness. This organizes the instruments, reminiscence, prompts, and so on. into one thing coherent and legible.

your-harness/├── config/│   ├── fashions.yaml              # Supplier configs, API keys, charge limits├── mannequin/│   ├── interface.py             # Unified mannequin calling API│   ├── context.py               # Context window administration, summarization│   └── suppliers/│       ├── openai.py│       └── anthropic.py├── instruments/│   ├── registry.py              # Instrument registration, lookup, schema validation│   ├── builtin/                 # Your instruments (every file = one instrument + its schema)│   └── customized/                  # Agent-generated instruments land right here├── state/│   ├── reminiscence.py                # Working reminiscence, checkpointing, rollback│   └── plan.py                  # Plan monitoring, step completion standing├── orchestration/│   ├── loop.py                  # Core agent loop│   ├── router.py                # Subsequent-action selections: instrument, sub-agent, escalate│   └── subagents.py             # Sub-agent spawning and coordination├── analysis/│   ├── linters.py               # Customized linters│   └── validators.py            # Output validation (schema, checks, evaluate)├── observability/│   ├── tracer.py                # Structured hint logging│   └── metrics.py               # Price, latency, success charge monitoring├── prompts/│   ├── system.md.jinja          # Base system immediate template│   └── phases/                  # One template per workflow stage├── evals/                       # Benchmark duties for regression testing├── most important.py└── README.md

Each undertaking is completely different, however beginning with a transparent psychological mannequin for a way the agent is structured is critical. Keep in mind, it is more likely to evolve, so maintain issues as clear as doable with out overlaps which might confuse brokers who try to switch the harness.

Evaluate this mannequin to one thing like Hermes agent, which has seemingly suffered from appreciable AI drift. Its root AGENTS.md has 14 completely different sub AGENTS.md. The /brokers listing, which accommodates many of the brokers’ precise logic has 395 separate information and 145k strains of code. This does not even embrace the abilities or prompts. I am not saying it is pointless for Hermes to want this diploma of complexity, but it surely’s most probably protected to say that some AI-generated drift has occurred right here.

Not imposing a transparent boundary on the harness itself and permitting the agent to contribute to its personal harness is a recipe for technical debt. Though the complexity of the undertaking and overlapping modules aren’t clear proof of AI drift, its nonetheless my greatest guess.

A well-scaffolded harness separates no less than the next considerations:

• The mannequin interface layer: The boundary between your harness and the LLM. It handles immediate development, token counting, response parsing, retry logic, and model-specific quirks. In the event you swap from Claude to GPT-4 to Gemini, solely this layer ought to change.

• The instrument/abilities layer: Instrument definitions, enter validation, execution. Every instrument ought to be a self-contained unit with a transparent schema and doc.

• The state layer: The agent’s working reminiscence, activity progress, accrued context, plan state, and any artifacts it has produced.

• The orchestration layer: That is the place your workflow logic lives. It decides what occurs subsequent: which section the agent is in, whether or not to invoke a sub-agent, whether or not to checkpoint, whether or not to escalate to a human.

• The analysis layer: Mechanisms that assess whether or not the agent’s outputs meet high quality thresholds. This may be something from working unit checks to having a second mannequin evaluate the output to checking structural constraints on the outcome. This layer gates transitions between workflow phases.

The success criterion is {that a} human or agent can motive about every one independently, take a look at every one independently, and modify every one with out worrying about cascading unwanted side effects.

This scaffolding serves a twin objective. It makes the system maintainable for you. And it makes the system legible to the agent working inside it. An agent that may learn its personal instrument definitions, perceive its personal state schema, and see the construction of its personal workflow is an agent that may be trusted with extra autonomy.

Self-Modifying Harnesses

Probably the most highly effective harness architectures permit the agent to increase its personal capabilities. In a sensible, engineering sense: the agent encounters a recurring drawback, writes a brand new instrument to resolve it, registers that instrument in its instrument layer, and makes use of it going ahead.

That is already occurring. Coding brokers routinely write helper scripts, create utility capabilities, and construct small instruments mid-task to make subsequent steps simpler. The query is whether or not your structure treats this as an accident or a first-class functionality.

To help protected self-modification, you want:

• A instrument registry that’s modular: If instruments are outlined as configuration (schemas, descriptions, entry factors) moderately than hardcoded into the orchestration logic, the agent can add new instruments by merely writing a brand new configuration entry and the corresponding implementation. It would not want to the touch the core harness.

• Sandboxed execution: Any instrument the agent creates should run in a sandbox. It mustn’t have entry to the harness internals, your credentials, or something outdoors its designated scope. The harness trusts its built-in instruments since you wrote them. Agent-authored instruments earn no such belief by default.

• Validation earlier than registration: Earlier than a brand new instrument turns into accessible, it ought to move primary checks. Does the schema parse? Does a smoke take a look at succeed? Does it battle with current instruments? Use auto linters.

📌The scaffolding is what makes this protected. With out it, self-modification is a recipe for the form of cascading, incomprehensible technical debt that grinds methods to a halt.

Primitive #3. Observability

Agent methods provide you with not one of the deterministic safeguards that conventional software program methods (sometimes) do by default with unit checks. The mannequin is a black field. Its “reasoning” is a stream of tokens. Its selections are probabilistic: the interplay between the mannequin and the harness produces emergent conduct that may be extraordinarily tough to motive about after the very fact.

Observability is the way you make agent methods reliable. With a very good observability stack, you possibly can diagnose failures and iteratively enhance.

Hint Every little thing

Absolutely the minimal: log each mannequin name and each instrument invocation with full inputs and outputs. Each single one. That is your equal of a stack hint.

A single agent activity may contain dozens of mannequin calls and gear invocations. When one thing goes flawed (and it’ll), you want to have the ability to reconstruct precisely what occurred:

• What was within the immediate, reasoning and response at every step?

• What instrument was referred to as, with what arguments and what did it return?

• How did the harness interpret that outcome?

• What choice did your workflow orchestration make subsequent, and why?

Construct it in from the beginning. Each mannequin name ought to move via a logging wrapper. Each instrument invocation ought to be recorded. Each state transition ought to be annotated.

Use structured codecs (JSON, spans with parent-child relationships) that you could question, filter, and visualize. You need to have the ability to reply questions like: “Present me each activity within the final week the place the agent retried a instrument name greater than 3 occasions” or “What was the common context window utilization on the level the place duties began failing?”

An instance schema:

{  "schema_version": "1.0",  "event_id": "evt_000042",  "trace_id": "trace_00121",  "task_id": "task_001",  "session_id": "session_007",  "parent_event_id": "evt_000041",  "sequence": 42,  "timestamp": "2026-10-23T16:00:01.234Z",  "event_type": "mannequin.name",  "part": "agent_runtime",  "standing": "success",  "duration_ms": 842,  "enter": {},  "output": {},  "metadata": {    "retry": 0,    "worker_id": "worker_01",    "tags": ["demo"]  },  "error": null}

Human Assessment Loops

Lastly, observability contains the power for people to examine and intervene in agent conduct. This isn’t the identical as a human-in-the-loop workflow step (although that is helpful too). That is the power for an engineer to:

• Watch an agent work in actual time, seeing its reasoning and actions as they occur

• Pause an agent mid-task to examine state and context

• Replay a accomplished activity step-by-step to grasp conduct

• Flag particular agent behaviors for evaluate and annotation

These capabilities are what rework a harness right into a system you possibly can truly enhance with confidence. Pinpoint the place an agent is failing and design the proper conduct.

Wrapping up

The trajectory is evident. Fashions will proceed to get extra succesful. However the hole between uncooked mannequin functionality and a system that reliably solves actual issues will stay. That hole is the harness. That hole is now your job.

The engineers who thrive within the subsequent decade will not be those who write essentially the most code, and even those who immediate most cleverly. They’re going to be those who can design, construct, and function the methods that flip mannequin intelligence into dependable outcomes. Workflow design. Structure. Observability. These are the primitives. Grasp them, and you will be constructing the infrastructure that everybody else relies on.

Thanks for studying!

References

[1] Lopopolo, R. (2026, February 11). Harness engineering: Leveraging Codex in an agent-first world. OpenAI. https://openai.com/index/harness-engineering/

[2] Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, Ok., & Yao, S. (2023). Reflexion: Language brokers with verbal reinforcement studying. Advances in Neural Data Processing Methods, 36, 8634–8652. https://doi.org/10.48550/arxiv.2303.11366

[3] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. Ok.-W., & Lim, E.-P. (2023). Plan-and-solve prompting: Enhancing zero-shot chain-of-thought reasoning by massive language fashions. Proceedings of the 61st Annual Assembly of the Affiliation for Computational Linguistics (Quantity 1: Lengthy Papers), 2609–2634. https://doi.org/10.18653/v1/2023.acl-long.147

[4] Weng, L. (2026, July 4). Harness engineering for self-improvement. Lil’Log. https://lilianweng.github.io/posts/2026-07-04-harness/

[5] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, Ok., & Cao, Y. (2023). ReAct: Synergizing reasoning and performing in language fashions. Worldwide Convention on Studying Representations (ICLR). https://doi.org/10.48550/arxiv.2210.03629