Measuring the Creativity Potential of LLM Brokers

This weblog submit is predicated on our current work, “Can LLM Brokers Uncover? Evaluating Creativity on ML Engineering Duties“, revealed at COLM 2026 and written with Yunxiang Zhang and Professor Lu Wang on the College of Michigan. Do take a look at the paper for a extra detailed studying, whereas this weblog submit acts as a summarized model of our work. The primary query we try to reply right here is that this: whereas there was a large push for AI for Science, with enormous investments in LLM brokers for scientific discovery, these brokers nonetheless fall wanting the top-1 human on actual ML analysis challenges. On the identical time, we see common experiences of LLMs making breakthroughs (AlphaEvolve, the Kosmos AI scientist, and many others.), and OpenAI just lately claimed to have solved Navier-Stokes. So why the disconnect?

A standard reply to that will be the underlying framework and scaffolding the mannequin has entry to, as these higher frameworks would possibly permit for extra environment friendly search of the answer area, however how can we quantify this notion of “higher search”? We argue that creativity provides a helpful lens.

So then what’s Creativity?

A pure trait we wish in these brokers is that they provide you with concepts which can be each novel and ship nice outcomes, and that’s precisely what creativity is.

In response to the “The Commonplace Definition of Creativity” by Mark A. Runco and Garrett J. Jaeger, “Creativity is the manufacturing of concepts or merchandise which can be concurrently authentic and helpful (i.e., efficient or applicable)”

Curiously, there have been many different works in inventive psychology that hyperlink creativity to look in a conceptual area:

Boden, M. A. (1998). “Creativity and Synthetic Intelligence” →

“the technology of novel concepts by the exploration of structured conceptual areas.”

Boden, M. A. (2004). The Artistic Thoughts: Myths and Mechanisms (2nd ed.) →

“Western music springs from a search-space outlined by the principles of concord, and its melodies are pathways by means of a exactly mappable panorama of musical intervals.”

Newell, A., Shaw, J. C., & Simon, H. A. (1962). “The Technique of Artistic Considering” →

“success of an issue solver who’s confronted with a fancy process rests totally on his skill to pick out, appropriately, a really small a part of the full problem-solving maze for exploration.”

This results in the principle query that we try to research on this challenge:

To investigate whether or not the efficiency variations between agent frameworks may be attributed to how they construction and information the inventive search course of,  and to quantify how creativity emerges and evolves inside these frameworks.

Creativity → Originality + Usefulness

For the remainder of this submit, we’re going to break creativity down additional into subparts primarily based on analysis from inventive psychology. Following Boden, originality may be additional damaged down into P-Creativity and H-Creativity. P-Creativity or P-novelty mainly measures how novel one thing is relative to this system’s personal reminiscence and historical past. H-Creativity measures how novel one thing is in comparison with all the physique of human data. Following Chan and Schunn, usefulness may be cut up into affect and feasibility.

Placing these collectively, we get:

Creativity → P-Creativity + H-Creativity + Influence + Feasibility

…which is the definition we’re going to be working with for the remainder of this submit. Curiously, this mix of ‘novelty’ and ‘usefulness’ is what makes creativity fascinating for science brokers.

Drawback Formulation

Okay, now that we’ve got our definition of creativity, the following query is the place it makes essentially the most sense to measure it to be able to take a look at the progressive skill of fashions. In a multi-turn agentic setting, we want three situations to measure creativity meaningfully: quantifiable usefulness metrics, wealthy human baselines for H-creativity comparability, and an answer area the place real novelty is feasible. Based mostly on these standards, machine studying duties are one of the best match, so our drawback turns into:

Given a hard and fast LLM and a collection of ML duties, how do totally different agent frameworks information the technology of inventive options over time, and might we use creativity metrics to elucidate why some frameworks/LLMs outperform others?

As acknowledged earlier, the metrics that we’re desirous about measuring listed here are:

For P and H-creativity, LLM-as-a-judge finally ends up as our most important rating. Influence is a normalized 0–1 rating of how shut the mannequin will get to the top-1 human rating, and feasibility is implicit, in that we measure creativity just for these episodes that are possible. We additionally use this notion of episodes right here the place an episode is a group of steps which led to a profitable submission.

Let’s attempt to perceive our process setup and metric measurement with an instance run:

Cassava Leaf Illness Classification

The duty right here is picture classification. The agent (AIDE with GPT-5) is supplied with a folder containing the practice dataset and the issue assertion. The agent begins off with a ridge classifier method, and since that is the primary episode, with no prior historical past to check in opposition to, it will get a P-creativity of 4 by conference. The agent then strikes on to attempt a pair extra approaches, with H-creativity and affect peaking when it makes use of LightGBM with handcrafted options. Curiously, people in all probability found very early within the competitors that CNNs carried out finest, and so targeted on neural nets, which is why LightGBM stands out as novel in comparison with 3,747 human approaches. After solely episode 4, the agent will get caught making an attempt the identical method many times for the remainder of the run focusing extra on exploitation reasonably than exploration, and finally ends up nicely wanting a medal.

Setup

We take 10 duties from MLE-bench, a benchmark that exams ML engineering skill on Kaggle competitions, spanning picture, NLP, and tabular knowledge. We filtered for competitions with a wealthy corpus of human options, which right here means between 877 and three,747 public notebooks per competitors.

We consider two brokers: AIDE, a grasping tree-search agent, and AIRA-Dojo, which builds on AIDE however provides extra search methods and operators. We use GPT-5 and Qwen3-32B because the spine fashions for these brokers. For every agent, we run 8 runs per model-task mixture, every with an 8-hour finances and a most of 10 episodes.

A bonus of selecting MLE-bench is entry to human trajectories. Curiously, we will additionally construct trajectories of how human affect and P-creativity change over the course of a complete competitors. Think about an individual who labored on a contest for 3 months and posted loads of public work: we will use that trajectory to see how the concepts they used modified because the competitors went on, and what impact these adjustments had on their rating. We are able to use this to straight evaluate in opposition to an agent’s iterative behaviour to see how they match as much as people.

Can we Reliably measure P&H Creativity at scale?

This brings us to our first query: can we even measure P and H-creativity at scale, and the way would we go about it? Ideally, we might use skilled human judges, however that method does not scale in any respect. So can we use automated metrics as a proxy for human judgement? Seems that we will! We had 3 annotators label 300 episodes for P-creativity after which measured automated approaches in opposition to it. A whole lot of earlier works have adopted numerous totally different approaches to measure P-Creativity: some use LLM-as-a-judge, some use conceptual novelty, some use semantic distance, some use surprisal, and so forth. We evaluate all of them in opposition to the human annotations to see which does finest, and use the winner for our P-creativity evaluation.

LLM-as-a-judge carried out one of the best thus driving our determination to make use of it for measuring P-creativity. Semantic distance does decently nicely and the hole between efficiency of various fashions as choose underlines the necessity for higher reasoning capabilities to measure novelty. For H-creativity, given the large human corpus would exceed context size of most LLMs, we went with a two-step technique: first use semantic distance to retrieve the 5 closest neighbors, then run LLM-as-a-judge in opposition to these 5 reference options.

Brokers go from exploration to exploitation

Throughout all our brokers, we see a standard pattern of going from exploration to exploitation. As we spend extra test-time compute, the brokers naturally strikes from exploring new concepts to making an attempt to refine a specific path, however seeing how early an agent begins shifting in direction of exploitation is attention-grabbing. We see the identical pattern in people too however brokers present a a lot steeper decline. This type of means that even when we gave the agent, let’s say, 100 steps, it might solely use the primary few for any exploration. Curiously, P-creativity and affect are primarily uncorrelated: optimizing for one doesn’t assure good outcomes for the opposite!

An extra habits evaluation of agent reasoning traces confirms this exploration-to-exploitation mechanism: strategic exploration accounts for ~75% of reasoning traces early in a run, dropping to ~25% by the tip. Brokers decide to a paradigm shortly and refine inside it.

Search technique alone doesn’t decide creativity or affect

One other attention-grabbing outcome we noticed was that totally different search methods do not actually present very totally different developments! Over a number of iterations, all of them find yourself in about the identical vary, which is form of counterintuitive. We might count on totally different developments and outcomes from totally different search methods, however this outcome underlines the significance of all the things else in a framework: the underlying scaffolding, the prompts, how context is handed, and so forth. We won’t simply change the search technique and count on totally different outcomes. As a substitute, we want all the things across the agent to work in concord.

Brokers attain novel territory, however cannot convert it

One of many key takeaways we had is that after we measure the H-creativity of those brokers in opposition to people who obtain medals submit the competitors finish date, we see that LLMs really present increased novelty than these people but they carry out a lot worse than mentioned people. GPT-5 with AIDE achieves ~2x the historic novelty of gold-medal profitable people but solely 21% of GPT-5 runs achieved any medal. This outcome matches the findings of different works that brokers are in a position to provide you with extra novel options, however these options are not often possible or helpful.

The place are these nearest neighbors?

A possible concern is that agent novelty displays regression to early approaches people later deserted, reasonably than forward-looking exploration.

Temporal evaluation of agent episodes exhibits that agent concepts are unfold all through the competitors timeline. Curiously, GPT-5 exhibits essentially the most uniform unfold, whereas Qwen does present some clustering round concepts the people tried early on. This means that stronger reasoning capabilities in fashions could allow convergence in direction of extra mature human options.

Limitations & Attainable Future Instructions

A key takeaway that I wish to share from this work is the necessity to deal with the twin optimization of novelty and affect if we’re going to have brokers that may do autonomous analysis. With the rising significance of RL, this factors to the viability of utilizing P-creativity and affect as a twin optimization goal.

Scaling to longer trajectories: our analysis caps at 10 episodes resulting from context and compute limits; summarization or agent-as-a-judge approaches might allow P-creativity measurement over longer runs.

Extending to open-ended duties: our framework depends upon a quantitative metric and a bounded human corpus; making use of it to open-ended analysis settings would require surrogate usefulness indicators and richer reference corpora.

Since this work was finished, loads of new outcomes have come out additional exhibiting LLM brokers discovering new algorithms and outcomes. A few of these got here from brokers working with little human involvement, however most got here from people and AI engaged on an issue collectively. Human-AI complementarity appears one of the best path ahead for now. That being mentioned, with every new mannequin launch we’re seeing increasingly work finished autonomously by these brokers, and so they present a lot increased capabilities than what we noticed on this work with GPT-5 and Qwen3-32B. This factors to a future the place AI brokers doing science autonomously can change into a real actuality.

···

Word: All photographs had been created by the writer.