TL;DR
a full working implementation in pure Python, with actual benchmark numbers.
What I did: constructed a managed experiment that isolates one variable, relationship density, from all the things often confounded with it, utilizing a completely deterministic agent coverage as a substitute of stay mannequin calls.
What I discovered: extra communication pathways between brokers didn’t routinely imply higher multi-agent efficiency. Restoration stayed flat throughout the entire density sweep. However the pathways themselves didn’t keep flat — as density rose, the community used a shrinking fraction of the perimeters it had. The extra helpful engineering query isn’t merely what number of connections exist. It’s what number of of them truly carry info.
This isn’t only a conceptual proposal. It’s a working system with measurable, reproducible conduct. The experiment is reproducible; timing numbers are reported solely the place truly measured.
The Assumption I Went In With
Most individuals assume a failing multi-agent system has a immediate downside.
You construct a staff of specialised brokers, hook them up in a unfastened mesh, and run the pipeline. As an alternative of a completed outcome, you get infinite loops, context drift, and a burnt-through token finances. The speedy knee-jerk response is to rewrite the system prompts or swap in a bigger LLM.
I suspected the true perpetrator was structural: the precise ratio of open communication channels between brokers versus the entire channels doable.
In graph concept, that ratio is relationship density. For a directed graph with N nodes and E edges:
D = E / (N * (N - 1))Take an 8-agent setup: you’ve got 56 doable directed communication paths. Density is just the dial that controls what number of of these 56 paths are literally open. I needed to see if adjusting that single structural lever basically modifications how a community performs and whether or not extra connectivity is definitely higher.
A fast notice on the setup: all the info under comes straight from actual benchmark runs executing domestically (Python 3.12, CPU-only, zero exterior API calls), except explicitly famous as a design-phase calculation.
Who This Is For
This experiment design is value adapting in case you are presently choosing multi-agent topologies by intestine feeling: defaulting to a completely linked mesh as a result of it feels safer, or constructing a linear chain as a result of it’s simple to hint. It is usually a strong template if it is advisable to run managed, reproducible experiments on agent architectures with out blowing by way of your API finances on each iteration.
When to skip this:
- For those who simply desire a single magic density quantity to drop into manufacturing: The metrics listed here are tied to 1 particular activity, one topology household, an 8-agent structure, and a deterministic messaging coverage. They won’t copy-paste cleanly into your codebase, and I’m not claiming these actual thresholds maintain for stochastic LLM runs.
- In case your bottleneck is particular person mannequin efficiency: If a single agent is failing at fundamental activity execution, structural routing changes is not going to reserve it.
- In case your analysis requires true mannequin non-determinism: This setup deliberately trades away LLM stochasticity to ensure actual reproducibility throughout runs.
The whole code and the pre-specified take a look at protocol can be found within the repository. https://github.com/Emmimal/graph-density-engine/
Constructing the Experiment
Most comparisons that have a look at community topology make a elementary mistake: they alter two variables directly. They examine a series to a mesh to a completely linked graph, which modifications each the visible form of the community and the precise edge depend on the identical time. When efficiency shifts, there isn’t a technique to know if the motive force was relationship density or the particular structure of the graph.
To isolate the true trigger, this design retains each different issue static and sweeps a single variable.
Right here is the pipeline, finish to finish:
The take a look at plan evaluates 5 distinct density ranges: 20%, 40%, 60%, 80%, and 100%.
The system makes use of a hard and fast depend of eight brokers all through all the benchmark. Every density stage undergoes ten impartial trials, totaling fifty runs. Each run makes use of a novel random seed, with all seeds locked earlier than executing the take a look at suite.
Element 1: The Topology Generator
The community topology household is strictly locked to linked Erdős–Rényi random graphs [1]. Edges are sampled uniformly at random till reaching the goal density stage. Any disconnected graph samples are instantly rejected and resampled till a completely linked path exists throughout all nodes.
This technology course of represents the one graph development pipeline in all the undertaking. There are not any hidden central hubs, star topologies, or hand-tuned structural guidelines that would quietly confuse graph form with pure edge density.
Right here is how a 20% density graph compares to a 100% density graph for the very same 8-agent setup. Every row represents a person agent, and every indicator highlights an lively, outbound communication path to a different node:
def generate_connected_erdos_renyi(num_agents, target_density, rng, max_attempts=20000):
edges = all_possible_directed_edges(num_agents)
target_edge_count = spherical(target_density * len(edges))
for _ in vary(max_attempts):
chosen = rng.pattern(edges, target_edge_count)
adjacency = build_adjacency(chosen, num_agents)
if is_strongly_connected(adjacency):
return adjacency
increase RuntimeError("no linked graph discovered")Element 2: The Agent Coverage
This half units this construct aside from a typical multi-agent demo, and it’s an intentional design alternative moderately than a shortcut.
The brokers are usually not powered by LLM API calls. As an alternative, every of the eight brokers follows a easy, deterministic coverage: contribute whichever of your personal unshared details is least comparable (utilizing TF-IDF [2]) to what has already been said. As soon as an agent has shared all its details, it falls again to repeating whichever of its details is most related to the present subject.
There isn’t any randomness within the resolution logic, no API latency, and no hidden conduct inside a mannequin’s weights.
I selected this strategy for a selected motive. My first draft used a simulator that basically compelled the end result it was attempting to find: redundancy was injected by way of a coin flip linked to message depth, which assured that density correlated with redundancy.
Switching to stay LLM calls would have fastened that synthetic conduct, however it introduces completely different issues. Stay mannequin calls are costly, topic to fee limits, and non-deterministic, making the outcomes tough to audit or replicate cleanly. A transparent, rule-based coverage avoids each points. Each resolution is absolutely inspectable, and all the fifty-run benchmark reproduces bit-for-bit with the identical preliminary seeds.
def _select_message(self, remaining, own_facts, shared_facts):
if remaining:
return self._most_novel(remaining, shared_facts) # novelty-seeking
if not own_facts:
return "NO_KNOWLEDGE_AVAILABLE"
return self._most_on_topic(own_facts, shared_facts) # repeat fallbackElement 3: The Diagnostics
A single top-line metric can not inform you why density did or didn’t have an effect on the end result, so 5 distinct diagnostics run beneath the principle execution:
| Metric | What It Measures |
|---|---|
| Relationship Effectivity | Fraction of messages that added a genuinely new truth to shared state |
| TF-IDF Redundancy | Lexical similarity of every new message to what’s already been stated |
| Data Acquire | Novel details contributed / whole details contributed |
| Edge Utilization | Really-used edges / configured edges at a given density |
| Communication Depth | Complete messages elapsed |
Edge Utilization seems to be the place the flat restoration curve will get attention-grabbing. It’s the single diagnostic that separates what number of pathways exist from what number of pathways truly carry a message.
What I Did
To recap the setup earlier than wanting on the outcomes: eight brokers, every holding a small, non-overlapping slice of a 17-fact incident state of affairs. No single agent begins with the entire image.
We take a look at throughout 5 density ranges, with 10 trials per stage for a complete of fifty runs. Each run will get a tough restrict of a 35-message communication finances. The first activity is to measure how a lot of the ground-truth state of affairs the community manages to consolidate into its last shared state by the tip of the run.
What I Acquired
Going into this, I anticipated to see an inverted U-curve: low density starves the community of connections, excessive density drowns it in redundant chatter, and someplace within the center lies a candy spot.
That’s not what occurred, not less than not for info restoration.
Restoration represents the fraction of the state of affairs’s ground-truth details that made it into the ultimate synthesized output. Listed below are the averages throughout 10 trials per density stage:
| Density | Data Restoration | Relationship Effectivity | Redundancy |
|---|---|---|---|
| 20% | 0.959 ± 0.070 | 0.457 ± 0.034 | 0.240 ± 0.019 |
| 40% | 0.924 ± 0.083 | 0.440 ± 0.039 | 0.243 ± 0.027 |
| 60% | 0.971 ± 0.039 | 0.469 ± 0.019 | 0.240 ± 0.025 |
| 80% | 0.976 ± 0.039 | 0.471 ± 0.023 | 0.249 ± 0.019 |
| 100% | 0.959 ± 0.046 | 0.463 ± 0.021 | 0.250 ± 0.026 |
Restoration sits inside a good band, roughly 92% to 98%, throughout all the sweep. The sparsest community within the setup recovers virtually as a lot floor reality because the absolutely linked graph.
The 40% situation exhibits the bottom common efficiency, touchdown at 0.924 imply restoration in comparison with 0.959–0.976 throughout the opposite 4 settings. That dip is value flagging, however given the 10-trial pattern measurement and within-condition variance, it’s higher handled as a candidate for additional testing moderately than definitive proof of a non-linear impact.
The take-away is particular: inside this topology household, at this agent depend, on this activity, and with this deterministic communication coverage, shifting density from 20% to 100% didn’t materially alter restoration. That could be a much more exact declare than saying “density by no means issues,” however it’s what the benchmark knowledge truly demonstrates.
Trying Beneath the Quantity That Didn’t Transfer
A flat restoration curve shouldn’t be the tip of the evaluation. It’s the place the main focus shifts from “did density change the end result” to “why didn’t it, and what modified as a substitute?”
Relationship Effectivity holds regular between 0.44 and 0.47 throughout each density stage, exhibiting no clear development. Brokers waste roughly the identical proportion of their turns no matter what number of communication paths are open. Redundancy stays flat as properly, remaining between 0.24 and 0.25 throughout all circumstances. Opposite to my preliminary assumption, opening up extra pathways didn’t result in a rise in repeated chatter.
Edge Utilization is the place the underlying mechanics turn out to be clear. Averaged throughout all 50 trials, right here is how configured edges examine in opposition to the perimeters the community truly used:
| Density | Configured edges | Avg. used edges | Avg. utilization |
|---|---|---|---|
| 20% | 11 | 10.7 | 97.3% ± 6.1% |
| 40% | 22 | 15.8 | 71.8% ± 11.1% |
| 60% | 34 | 21.3 | 62.6% ± 5.4% |
| 80% | 45 | 24.8 | 55.1% ± 7.1% |
| 100% | 56 | 26.4 | 47.1% ± 4.6% |
That could be a clear, monotonic drop.
At 20% density, the community makes use of nearly each edge it’s given. It operates near its structural capability, with barely any slack. At 100% density, it makes use of beneath half of what’s configured, on common. In these runs, the absolutely linked community used about 47% of its configured edges. Not zero, and never “by no means carried a single message.” Only a steadily shrinking fraction as extra edges have been added.
Absolutely the variety of edges in lively use nonetheless climbs as density rises (roughly 11 edges at 20% density as much as 26 edges at 100%), so these additional edges are usually not fully inert. Nonetheless, they get used at a sharply diminishing fee relative to what number of you add. Doubling the sting finances from 60% to 100% density practically doubles the configured edges from 34 to 56, however provides solely about 5 extra lively edges in observe (shifting from 21.3 to 26.4).
That’s the actual distinction the flat restoration curve was hiding: configured connectivity and behavioral connectivity are usually not the identical factor, they usually diverge additional because the graph grows denser.
Efficiency Traits
Measured on a fifty-trial full sweep with zero API calls and nil value:
| Operation | Value / Execution Time |
| Section 0: Metric unit checks (16 checks) | beneath 0.25 seconds |
| Section 1: Graph engine validation, 50 runs, DummyAgent | beneath 1 second |
| Section 2: The actual experiment, 50 runs, PureAgent | Not individually benchmarked |
| API value for the complete experiment | $0 |
I’ve not individually benchmarked wall-clock time for Section 2 or verified cross-platform reproducibility throughout completely different working methods. What I can verify is that the suite is absolutely deterministic given a hard and fast seed (Section 0’s take a look at suite explicitly verifies this), and each trial ran to completion with out timeouts.
For those who clone the repository and run the benchmark suite your self, I’d have an interest to listen to what timing and conduct you observe in your machine.
Sincere Design Choices
1. The Deterministic Agent Coverage
The rule-based agent coverage is a deliberate trade-off, not a free win. It provides us whole reproducibility and nil API prices, however it means these outcomes replicate how a hard and fast, rational routing technique behaves beneath various community topologies, moderately than how a stochastic LLM inhabitants would. A mannequin with much less predictable output may work together with density fairly otherwise, and I’d not assume these actual numbers switch on to LLM calls with out express testing.
2. Commonplace Library Key phrase Matching
The Data Restoration metric makes use of keyword-overlap matching as a substitute of semantic embedding similarity, retaining with the standard-library-only design. Early in improvement, this heuristic was miscalibrated: a threshold of 0.6 allowed details from the identical incident to cross-credit one another by way of shared entity tokens like service names or timestamps. Consequently, sharing a single actual truth may spuriously “recuperate” two or three unrelated ones. Elevating the edge to 0.85 after tracing the bug fastened the difficulty, making certain actual restoration matching.
3. Eradicating Round Early Exits
A extra elementary flaw surfaced throughout preliminary protocol design. The unique stopping rule was set to “exit as soon as restoration crosses 70%,” which made restoration each the termination situation and the output metric. That logic was round: each trial outcome was mechanically pinned to whichever truth depend hit the edge first, making it structurally unimaginable to detect a density impact whatever the true underlying dynamics. The repair was easy: take away the early exit totally. Each trial now runs the complete communication finances, and restoration is evaluated as soon as on the very finish.
4. Finances Measurement and Ceiling Results
The 35-message restrict proved fairly beneficiant for a 17-fact state of affairs, making a ceiling impact. Most runs recovered the overwhelming majority of details properly earlier than exhausting their finances, which compressed the room accessible for density to indicate a transparent impression.
To check whether or not this finances buffer was masking an actual impact, I ran a smaller, pre-specified follow-up utilizing a a lot tighter message finances. The outcomes have been suggestive moderately than conclusive: the identical drop at 40% density reappeared, and a paired comparability hinted that mid-density networks may lose extra floor beneath extreme message constraints than both very sparse or very dense ones. That could be a distinct sample value exploring in a devoted experiment, so I’m flagging it right here moderately than claiming it as a confirmed rule.
5. Dependency Footprint
The core simulation runs purely on the Python customary library, requiring no exterior packages for the graph engine, agent logic, or diagnostic metrics. The undertaking repository lists pytest solely to run the 16-test validation suite, which is a testing utility moderately than a runtime requirement for the experiment itself.
Commerce-Offs and What Is Lacking
Actual Mannequin Brokers
The Agent interface was explicitly designed to be modular. The graph engine, message router, and diagnostic metrics are totally agent-agnostic. Dropping an actual LLM into that interface—buying and selling away zero-cost reproducibility for stochastic conduct—would show whether or not these actual structural patterns maintain up when fashions introduce non-determinism and reasoning noise.
Richer Eventualities
The dataset corpus presently rotates three core incident templates throughout ten state of affairs information. A publication-grade iteration wants ten absolutely distinct eventualities, or not less than a transparent disclaimer that template rotation limits semantic selection.
A pre-specified Shortage Research
The tight-budget follow-up pointed to an intriguing sample, however it stays unconfirmed. Doing this justice means pre-registering a devoted take a look at suite with the scarcity-sensitivity speculation locked in earlier than operating the benchmark, moderately than noting it after wanting on the runs.
Weighted and Frequency-Based mostly Density
Proper now, density measures static graph geometry. A future model that weights edges by precise message frequency—moderately than mere existence—would bridge the hole to Edge Utilization, which already proves that configured connectivity and realized site visitors diverge quickly as networks develop denser.
Closing
Inside this experiment—this topology household, this agent depend, this activity, and this deterministic coverage—graphs didn’t enhance simply because that they had extra edges, nor did they degrade. What dictated efficiency was not the sheer quantity of open communication pathways, however what number of of these pathways the community truly required. That operational core remained remarkably steady, even because the graph was given much more structural capability to develop.
I initially anticipated this benchmark to inform a clear, dramatic story about dense networks collapsing beneath their very own communication overhead. As an alternative, it delivered one thing quieter and much more sensible: a transparent reminder {that a} graph’s configured edge depend shouldn’t be the identical factor as its precise conduct, and the one technique to spot the distinction is to instrument the system and measure it immediately. https://github.com/Emmimal/graph-density-engine/
References
[1] Erdos, P., & Renyi, A. (1959). On Random Graphs I. Publicationes Mathematicae Debrecen, 6, 290-297.
[2] Salton, G., & Buckley, C. (1988). Time period-weighting approaches in computerized textual content retrieval. Data Processing & Administration, 24(5), 513-523.
Disclosure
All code on this article was written by me and is unique work, developed and examined on Python 3.12. Benchmark numbers are from precise runs of the system, zero API calls, and are reproducible by cloning the repository and operating the included take a look at suite and experiment scripts, besides the place explicitly famous as protocol-design calculations. The simulation itself makes use of no exterior library past the Python customary library; the take a look at suite makes use of pytest. All photographs and figures on this article, together with the featured picture and each diagram, have been created by me. The featured picture was generated with ChatGPT (DALL·E); the diagrams (system pipeline, adjacency matrices, resolution tree, edge-utilization chart) have been constructed immediately from the experiment’s personal knowledge and design. I’ve no monetary relationship with any software, library, or firm talked about on this article.
