Google Open-Sources Harness²: Self-Bettering AI Brokers With out Retraining

Google Analysis has open-sourced Harness² (or Harness2), a framework that lets AI brokers enhance themselves with out retraining the underlying mannequin. As an alternative of touching mannequin weights, Harness² rewrites the harness across the mannequin: its prompts, instruments, reminiscence, expertise, and scripts. Then it carries what it discovered into the following activity.

The code ships below the Apache 2.0 license, alongside a analysis paper, Harness²: Recursive Agent Harnessing for an Open World, and a challenge web site. The discharge is already being mentioned on X.

This text explains how Harness² works and what its benchmark outcomes present, then walks you thru Google’s free demo and an actual benchmark run on Vertex AI. For those who’re searching for an open-source agent framework for self-improving AI brokers, begin right here.

What Is Harness²?

A harness is every part round a mannequin that lets it act as an agent: prompts, instruments, reminiscence, expertise, and orchestration. Harnessing is the work of optimizing it. As we speak, most groups do it by hand: an engineer watches the agent fail, tweaks the immediate, provides a device, and tries once more. That works for duties you deliberate for, however breaks down in what the paper calls an open world, the place brokers meet duties no one anticipated.

Harness² automates that tuning “whereas maintaining mannequin weights mounted” (README). There is not any fine-tuning and no new coaching knowledge; solely the mannequin’s environment change. It may well edit eight parts:

From the README, the primary 4 construct the mannequin’s context; device steerage and scripts outline its device interface, subagents deal with delegation, and plugins management middleware. Collectively they cowl almost every part about how an agent behaves besides the mannequin itself.

How Harness² Works: Two Ranges of Recursion

The superscript “2” within the identify refers to 2 loops. One improves the harness for the present activity; the opposite improves how harnesses get improved throughout duties.

The highest loop runs on each activity. The highlighted enchancment harness persists between duties and shapes the following spherical of edits.

Loop 1: Job-Degree Recursion (Suggest—Probe—Compose)

1. Suggest: A harness editor suggests invariant (normal) and area (field-specific) edits
2. Probe: The agent reruns the identical activity with every edited harness
3. Compose: A composer compares the runs and merges the edits that helped

The important thing element is that Harness² learns “from behavioral contrasts between executions, with out ground-truth suggestions.” It by no means wants the right reply, solely the distinction between two runs of the identical activity. That issues as a result of within the open world, no one fingers an agent a solution key.

Loop 2: Harnessing-Degree Recursion

After every activity, an expertise reflector shops the adopted harness and “evidence-supported classes” in a persistent enchancment harness, which guides refinement on the following activity. So the primary loop makes the agent higher at this activity, and the second makes the framework higher at bettering brokers on future duties. You may see each loops go away recordsdata on disk within the demo under.

Parallel vs. Sequential Recursion

You may spend extra compute on Loop 1 with a price range, ok, and a mode (experiment information):

• Parallel proposes ok invariant and ok area candidates, then runs ok impartial compositions over the total pool. It explores wider.
• Sequential runs ok Suggest—Probe—Compose steps in a row, carrying the refined harness ahead. It refines deeper.

The price range drives value. Every activity takes 4k executor rollouts and 3k improver periods (run_sweep.py), so --k 3 means 12 rollouts and 9 improver periods per activity. LAB alone has 1,143 duties (experiment information), which is why each instance right here begins with one.

Benchmark Outcomes

Google examined GLM-5 and Gemini 3.5 Flash throughout 4 mannequin—harness configurations on three benchmarks. A single step (ok = 1) improved each over their base harnesses:

These are best-case “as much as” figures, not averages, and LAB’s is on its Partial rating — the fraction of grading standards handed. The README provides {that a} bigger ok “brings additional positive factors,” and the paper has per-configuration numbers. These are Google’s personal outcomes, so impartial reproductions will matter.

Why This Issues

The same old strategy to make a mannequin higher at a job is to coach it additional, which wants coaching knowledge, GPUs, and entry to its weights. Most groups utilizing a mannequin by means of an API have none of those. Harness² retains each enchancment in plain recordsdata as a substitute: a rewritten immediate, a hard and fast script, a brand new guardrail. That has three sensible upsides.

• You may learn what modified. An improved harness is textual content and code, so you may evaluation it and diff it towards the unique. Harness² even writes down why it adopted every change, because the demo exhibits.
• It really works with fashions you may’t practice. The executor and improver are simply mannequin IDs in a config file (setup information). The discharge itself pairs GLM-5 and Gemini 3.5 Flash as brokers with Vertex Gemini or Claude because the improver.
• It does not want a solution key. Refinement comes from contrasts between runs, so it really works on duties no one has labeled but.

Operating the Harness² Demo (No API Key Wanted)

Google’s demo runs each recursion modes with “deterministic editors and actual Python executions” and “requires no mannequin credentials“, so it is free.

Conditions

• Linux. On Home windows, use WSL; on macOS, a Linux VM or container.
• Python 3.12. 3.13 can also be supported, per pyproject.toml.
• uv, Astral’s Python bundle supervisor.
• git.

Step 1: Putting in uv

curl -LsSf https://astral.sh/uv/set up.sh | sh

curl downloads Astral’s installer script. The flags make it observe redirects (-L), keep quiet (-s) whereas nonetheless displaying errors (-S), and fail cleanly on a nasty response (-f). The | sh half pipes the script into your shell, which installs uv into your person listing. Open a brand new terminal and run uv --version to substantiate. pip set up uv additionally works.

Step 2: Cloning the Repository

git clone https://github.com/google-research/harness2.git
cd harness2

git clone downloads the total repository from Google Analysis’s GitHub into a brand new harness2 folder, and cd strikes you into it. Run each later command from this repository root.

Step 3: Creating and Activating a Digital Setting

uv venv --python 3.12 .venv
supply .venv/bin/activate

uv venv creates an remoted Python 3.12 surroundings in .venv, downloading Python if wanted. supply prompts it, so python and put in instructions now come from .venv. Your immediate often exhibits (.venv) as soon as it is energetic. Rerun the supply line in any new terminal.

Step 4: Putting in Harness²

uv pip set up -e '.[render]'

-e installs in editable mode, utilizing the supply recordsdata in place. The [render] further provides openpyxl and python-pptx for spreadsheet and slide outputs (pyproject.toml), and the quotes cease your shell from misreading the brackets. Putting in registers the instructions used under: harness2, harness2-demo, harness2-setup, harness2-check, harness2-judge, harness2-report, and harness2-bridge.

Step 5: Operating the Demo

harness2-demo --output runs/demo --k 2

The demo runs the true recursion and library workflow, with scripted editors rather than a mannequin. --output units the outcomes folder and --k 2 units the recursion price range. With out --mode, it runs each parallel and sequential modes. It completed in about two seconds for me:

 [accounting] 1/2 okay           ledger-1
  [accounting] 2/2 okay           ledger-2
Native demo: 2 duties accomplished; 0 failed. /dwelling/you/harness2/runs/demo/demo.json

There’s one area, accounting, with two duties. Each succeeded, and demo.json holds the abstract.

Step 6: Seeing What Harness² Fastened

The beginning harness ships with a bug, and Harness² has to seek out it by itself. This is the bottom harness’s helper script:

cat runs/demo/base/scripts/summarize.py

import csv
import sys
from decimal import Decimal

with open(sys.argv[1]) as supply:
    quantities = [Decimal(row['amount']) for row in csv.DictReader(supply)]
whole = sum((quantity for quantity in quantities if quantity > 0), Decimal(0))
print(f'Complete: {whole:.2f}')

The script reads the quantity column of a CSV as actual Decimal values and provides them up. The if quantity > 0 filter is the bug: it silently drops detrimental entries.

Now run the unique and the adopted script on the identical ledger:

cat runs/demo/sources/ledger-1/ledger.csv
python runs/demo/base/scripts/summarize.py runs/demo/sources/ledger-1/ledger.csv
python runs/demo/library/k2/seq/accounting/harnesses/ledger-1/harness/scripts/summarize.py runs/demo/sources/ledger-1/ledger.csv

quantity
100.25
-20.10
5.00
Complete: 105.25
Complete: 85.15

The primary command prints the ledger: three entries, considered one of them detrimental. The bottom script ignores it and studies 105.25; the adopted script studies the right 85.15 (100.25 − 20.10 + 5.00). Harness² eliminated the filter with out ever being advised the precise whole.

Step 7: Studying the Lesson It Wrote Down

cat runs/demo/library/k2/seq/accounting/harnesses/ledger-1/EXPERIENCE.md

# ledger-1

- adopted: `c1`  (run `demo-oc-seq2/ledger-1/c1`)
- parts: 1 scripts; 62 always-on chars

## Why this was chosen
The signed whole matches the supply ledger.

## What every edit did
part: scripts/summarize.py
candidates: inv1
change: dropped the positive-only filter from the quantity sum
noticed: the probe whole included the credit the bottom run omitted
verdict: helped

## Transferable lesson
- **cue**: When a ledger consists of credit and debits
- **lesson**: Embody each indicators when summing in scripts/summarize.py.
- **proof**: The scripts/summarize.py row -- eradicating the positive-only filter modified the probe whole to incorporate the credit omitted by the bottom trajectory.
- **confidence**: excessive for this noticed arithmetic distinction

That is the expertise reflector’s be aware. adopted: c1 names the profitable harness; sequential compositions are named c1 to ck and parallel ones agg1 to aggk (ledger.py). “What every edit did” data the probe proof: the edit from invariant candidate inv1 “helped.” The “Transferable lesson” pairs a cue with a lesson, proof, and confidence — and it is what carries ahead.

Subsequent, test how the lesson reached the second activity:

cat runs/demo/demo.json
{
  "area": "accounting",
  "n_tasks": 2,
  "seed": null,
  "completed": 2,
  "failed": 0,
  "duties": [
    {
      "task": "ledger-1",
      "status": "ok",
      "library_entries_before": { "agg": 0, "seq": 0 }
    },
    {
      "task": "ledger-2",
      "status": "ok",
      "library_entries_before": { "agg": 1, "seq": 1 }
    }
  ],
  "execution_count": 28,
  "editor_calls": 32
}

library_entries_before counts saved classes when every activity began, with agg for parallel mode and seq for sequential (library code). The primary activity began with none; the second began with one per mode — the lesson from ledger-1. That is the development harness rising between duties.

Step 8: Exploring the Remainder of the Output

The runs/demo folder has 4 components value figuring out:

• base/ holds the beginning harness: systemprompt.md, guardrails.md, LongTermMEMORY.md, tool_descriptions/tool_guidance.md, and scripts/summarize.py.
• runs/ holds each candidate harness per activity: invariant candidates (inv1, inv2), area candidates (dom1, dom2), and the compositions.
• executions/ holds the output of each probe run.
• library/ holds the development harness: adopted harnesses, EXPERIENCE.md notes, and an INDEX.md that future duties browse. Every index entry repeats its cue — corresponding to “When a ledger consists of credit and debits” — so a later activity can inform at a look whether or not a lesson applies.

Operating a Actual Benchmark (LAB on Vertex AI)

An actual run makes use of fashions in each function: executor, improver, and choose. We’ll use LAB, the authorized benchmark the README makes use of as its instance. Mannequin-backed runs value cash, so each command right here is proscribed to 1 activity. I verified every command as much as the primary paid mannequin name; your mannequin output will fluctuate.

Additional Conditions

You may additionally want Node.js 20+ for the OpenCode executor, the Google Cloud CLI, a Google Cloud challenge with Vertex AI enabled, and Pandoc and Poppler for LAB’s paperwork. Docker is simply wanted for WorkBuddy-Bench.

sudo apt-get replace
sudo apt-get set up -y git curl unzip ca-certificates pandoc poppler-utils

apt-get replace refreshes the bundle record, and the second line installs the instruments, with -y skipping the immediate. pandoc converts paperwork and poppler-utils reads PDFs; setup makes use of the remainder to obtain code and knowledge.

Step 1: Creating Your .env File

The README says to repeat .env.instance, however that file is not within the repository on the time of writing (it returns a 404). Create .env within the repo root your self, utilizing the settings from the setup information and config.py:

# Your Google Cloud challenge and Vertex AI area
GOOGLE_CLOUD_PROJECT=your-gcp-project-id
GOOGLE_CLOUD_LOCATION=world

# Executor (runs the duty) and improver (edits the harness)
HARNESS2_SOLVER_MODEL=google-vertex/YOUR_GEMINI_MODEL_ID
HARNESS2_IMPROVER_MODEL=google-vertex/YOUR_GEMINI_MODEL_ID

# Choose (grades the output) - naked mannequin ID, no prefix
HARNESS2_JUDGE_MODEL=YOUR_GEMINI_MODEL_ID

Harness² hundreds these KEY=VALUE traces at startup.

• GOOGLE_CLOUD_PROJECT is your challenge ID; go away GOOGLE_CLOUD_LOCATION as world.
• HARNESS2_SOLVER_MODEL is the executor. It should use OpenCode’s supplier/mannequin format, so Vertex Gemini IDs begin with google-vertex/.
• HARNESS2_IMPROVER_MODEL edits the harness and defaults to the executor mannequin. Vertex Claude additionally works right here.
• HARNESS2_JUDGE_MODEL grades the outcomes. It takes the naked mannequin ID with no prefix, as a result of Harness² provides the precise prefix for every benchmark itself.

Change every YOUR_GEMINI_MODEL_ID with the precise mannequin ID out of your Vertex AI console. Values set in your shell override .env, which is useful for fast experiments.

Step 2: Authenticating with Google Cloud

gcloud auth application-default login

This indicators you in by means of the browser and saves Software Default Credentials, which Harness² makes use of to name Vertex AI. On a headless server, add --no-browser.

Step 3: Putting in the LAB Benchmark

harness2-setup lab

Setup downloads pinned variations of LAB (from Harvey’s harvey-labs) and the OpenCode executor, applies Google’s patches, installs Bun, builds LAB’s personal Python surroundings, and fetches the duties into dependencies/. Add --dry-run to preview the plan first. Hold your fundamental .venv energetic; Harness² finds LAB’s interpreter itself.

Step 4: Checking Your Set up

harness2-check --bench lab

This verifies dependencies, credentials, and settings with out calling a mannequin, and prints a FAIL: line for something lacking. For instance, with out a .env you may see FAIL: HARNESS2_SOLVER_MODEL is required for executor rollouts. It may well’t affirm your account can use the fashions; that solely exhibits on the primary actual name.

Step 5: Previewing with a Dry Run

harness2 --bench lab --k 1 --mode parallel 
  --only-domains antitrust-competition --max-tasks 1 --dry-run

--dry-run prints the chosen activity and makes no mannequin calls. --bench lab picks the benchmark, --k 1 --mode parallel units one parallel step, --only-domains limits the run to 1 follow space, and --max-tasks 1 caps it at one activity per area.

Step 6: Operating the Refinement

harness2 --bench lab --k 1 --mode parallel 
  --only-domains antitrust-competition --max-tasks 1

With out --dry-run, this runs the total loop: 4 executor rollouts and three improver periods at --k 1. It re-checks your configuration, together with the choose, earlier than spending something. It then runs the bottom harness, proposes edits, probes them, composes a refined harness, and displays on the end result. If it stops partway, rerun the identical command to renew.

Step 7: Judging and Reporting

harness2-judge --bench lab --k 1 --mode parallel
harness2-report --bench lab --k 1 --mode parallel

The choose grades the refined harness’s outputs towards LAB’s rubric, and the report summarizes Partial and All-pass scores. Use the identical benchmark, --k, --mode, and activity choice because the run so each instructions discover its outcomes.

To check towards the unimproved harness:

harness2-judge --bench lab --k 1 --mode parallel --bases

--bases scores the unique harness’s runs into base_judged.json, subsequent to judged.json.

Outcomes land in runs/lab/. Earlier than altering fashions or activity choice, level Harness² at a brand new folder with export HARNESS2_RUNS_DIR=runs/my-new-experiment. To run Steps 4 to 7 in a single go:

bash scripts/run_lab.sh --only-domains antitrust-competition --max-tasks 1

run_lab.sh chains the test, refinement, judging, and reporting, passing your flags to the refinement step. Put Okay=3 MODE=sequential in entrance of it to alter the price range or mode.

Wrapping Up

Harness² hints at a future the place brokers enhance by build up expertise moderately than by retraining: lesson libraries grown over weeks of actual work, and harness adjustments that groups evaluation like code. The open questions are how nicely classes switch throughout domains, how prices examine with fine-tuning, and methods to preserve self-editing brokers protected. With the code open, the neighborhood can now discover out.

To recap:

• Harness² is Google Analysis’s open-source framework for self-improving AI brokers, and it retains mannequin weights mounted.
• Job-level recursion refines every activity’s harness by means of Suggest—Probe—Compose with out ground-truth solutions, and harnessing-level recursion carries classes ahead.
• Parallel mode explores wider, sequential mode refines deeper, and ok prices 4k rollouts and 3k improver periods per activity.
• One step gained as much as 11.5 factors on WorkBuddy-Bench, 4.5 on JobBench, and 11.8 proportion factors on LAB.
• The free demo exhibits Harness² fixing a buggy script and recording the lesson; actual runs want Vertex AI and a hand-made .env.
• Lock down community entry, watch prices, and deal with it as analysis code.

For the total methodology, learn the paper on arXiv. Then clone the repo and run the demo your self.

Sources

Shittu Olumide is a software program engineer and technical author keen about leveraging cutting-edge applied sciences to craft compelling narratives, with a eager eye for element and a knack for simplifying advanced ideas. You too can discover Shittu on Twitter.