The way to Make the most of OKF Effectively to Allow Data Trade Amongst LLMs

• The sample. That is Google’s Open Data Format skeleton — a Markdown file with a YAML frontmatter block — repurposed for agent hand-off. The repo’s frontmatter carries one further load-bearing subject the overall OKF spec doesn’t outline: token_pointer, an absolute path to the pre-computed .npy array in shared reminiscence. Human-readable physique, machine-readable pointer.
• The mechanism. Three Qwen2.5-Coder fashions of various sizes (7B / 3B / 1.5B) can not share a KV cache — they’ve completely different architectures. However they can share pre-computed token IDs, as a result of the entire Qwen2.5-Coder household ships one an identical BPE vocabulary. This repo tokenizes as soon as, palms off the integer array by /dev/shm/qwen_tokens/, and lets each downstream agent skip its personal tokenizer fully on the enter aspect.
• The numbers. Median of seven trials per immediate, 3 blocks, grasping decoding, 64 new tokens: on the 3B mannequin, imply baseline TTFT drops from 69.3 ms to 49.9 ms — a 28.0% discount. On the 1.5B mannequin, from 49.6 ms to 30.9 ms — a 37.8% discount. Each fashions move the coherence heuristic on each pattern. Full pipeline wall clock is 41.3 s finish to finish (Agent 1: 3.9 s, Agent 2: 18.7 s, Agent 3: 15.7 s).
• The guardrail. Feeding a downstream mannequin an integer array that meant a completely different subword beneath its personal vocabulary doesn’t crash something. It generates a fluent, coherent-looking, utterly incorrect report. So earlier than any agent trusts one other agent’s integers, this pipeline runs a full ~151,936-entry get_vocab() dict equality examine — not a vocab_size comparability, the true factor.
• What this does NOT declare. Quick-block regime (few-hundred-token blocks). No customized CUDA — that is orchestration on high of transformers‘ present mannequin.generate(input_ids=...) API. Tokenizer equivalence is verified for the precise three checkpoints this repo pins, not a family-wide standing assure.

TL;DR up entrance, so you possibly can go away with the purpose: when you’ve got ever wired three or extra LLM-based brokers from the identical mannequin household right into a pipeline that followers out over one shared doc, your CPU is working the very same Byte-Pair Encoding merges over the very same characters two or 3 times in a row, as a result of every agent’s tokenizer is a stateless new child that has no concept the earlier agent already produced the identical integer array. This submit is a couple of small pipeline of three Qwen2.5-Coder fashions (7B, 3B, 1.5B) the place the upstream agent tokenizes as soon as, drops a NumPy array of int64 token IDs into /dev/shm/qwen_tokens/, and each downstream agent calls mannequin.generate(input_ids=...) immediately on that array. It additionally — and that is the place the truly attention-grabbing engineering lives — refuses to let anybody else within the pipeline belief that array till it has confirmed, byte for byte, that each mannequin within the chain agrees on what these integers imply. That is orchestration, not a CUDA kernel. However when you’ve got ever debugged an LLM pipeline that produced fluent, on-topic, wrong-in-a-different-way-every-run output, you already know the form of the issue this piece of infrastructure is designed to stop.

Github repo: https://github.com/AnubhabBanerjee/inter-llm-tokf

---

1. A confession: your second agent is doing all your first agent’s homework, twice

Let me dramatise the second this entire repo is about.

Think about you’ve got three LLM brokers chained collectively. Agent 1 is an enormous mannequin, it reads a design doc. Agent 2 is a mid-sized mannequin, it evaluates a part of it. Agent 3 is a small mannequin, it writes the ultimate report. All three of them come from the identical mannequin household — similar tokenizer, similar vocabulary, similar every part above the hidden layers — simply at three completely different sizes. Since you aren’t fabricated from H100s, and working a 7B mannequin 3 times when a 1.5B mannequin will do for the final step can be, frankly, impolite to your GPU.

Now watch what occurs on a naive setup:

You: “Agent 1, please learn this design doc and move the related sections to Agent 2.”

Agent 1 (7B): “On it. Loading tokenizer. Operating BPE over the entire doc. Sections cut up. Handing off the attention-grabbing sections to Agent 2 as strings. ✅”

You: “Nice. Agent 2?”

Agent 2 (3B): “Hi there, I’m a wonderful, stateless new child. Loading my very own tokenizer. Operating BPE over the identical characters Agent 1 already ran BPE over three seconds in the past. Writing an analysis.”

You: “Wait, you’ve got the very same tokenizer as Agent 1.”

Agent 2 (3B): “I do?”

You: “Sure. You might be actually in the identical mannequin household. Identical vocabulary, similar subword IDs, similar every part.”

Agent 2 (3B): “That’s good. Anyway, I’ve re-tokenized the enter from scratch and I’m able to generate. Please stand by. 🫡”

You: “…and Agent 3?”

Agent 3 (1.5B): “Loading tokenizer. Operating BPE over Agent 2’s output—”

You: “You already know what, overlook I requested.”

That’s the joke, and it’s the soiled secret of each multi-agent LLM pipeline that followers out over one shared piece of textual content utilizing fashions from the identical household. The tokenizer isn’t the bottleneck — a quick Rust-backed BPE tokenizer isn’t gradual, and I cannot mislead you and faux it’s. However the tokenizer is redundant work, and what number of occasions you do redundant work isn’t a operate of how briskly the redundant work is. It’s a operate of what number of downstream customers you fanned out to.

The purpose of this piece of infrastructure, and the entire motive it took greater than a fifteen-line patch, is that the second you resolve to skip the tokenizer on the downstream aspect, you’ve got inherited a correctness downside that the tokenizer was beforehand doing for you. The remainder of this submit is what that appears like whenever you draw it out actually, and the one runtime examine that’s doing all of the load-bearing work.

2. Why three sizes in any respect? (a one-minute crash course on which layer is definitely shared)

Skip this in case you already know. For everybody else, right here is the brief model.

The three fashions on this pipeline are Qwen/Qwen2.5-Coder-7B-Instruct, Qwen/Qwen2.5-Coder-3B-Instruct, and Qwen/Qwen2.5-Coder-1.5B-Instruct. Identical structure household, similar tokenizer, three completely different sizes. The explanation they’re three completely different sizes and never one large one is intentionally telecom-flavored, as a result of that’s the world I truly got here from: the concrete instance this repo is constructed in opposition to is a design doc proposing {that a} chain of LLM brokers assist a cell core community’s operations staff motive a couple of new control-plane characteristic — particularly, bolting MCP (Mannequin Context Protocol) and A2A (Agent-to-Agent protocol) model orchestration onto the prevailing 5G Service-Based mostly Interface. The plan requires a big “Architect” agent that buildings the doc, a mid-sized “Protocol Engineer” that evaluates the attention-grabbing sections, and a small “Edge Analyst” that produces deployment-ready latency steerage — sufficiently small to run at a far-edge website subsequent to a UPF.

Three sizes, three roles, one pipeline.

Now, one structural reality drives the whole design: you can not share a KV cache throughout these three fashions. Completely different sizes imply completely different hidden_size values — 3584 for the 7B, 2048 for the 3B, 1536 for the 1.5B. The form of a KV cache is derived immediately from that quantity, so there is no such thing as a reinterpreting one mannequin’s cache as one other’s. That door is closed, completely, by the mathematics.

What’s not closed is the tokenizer. Qwen2.5-Coder ships one BPE vocabulary throughout its total measurement vary — the entire household is documented to agree on the identical integer-to-subword mapping. So whilst you can’t share activations between differently-sized fashions, you completely can share token IDs, supplied — and this “supplied” is doing loads of work, extra on that in a minute — each mannequin within the chain actually does use that very same vocabulary.

When you’ve got learn sufficient distributed-systems papers to be harmful, this form is acquainted. Two community features on the identical message bus don’t get to imagine they agree on message semantics simply because they’re each plugged into the identical bus. Two fashions in the identical household don’t get to imagine they agree on hidden states simply because they agree on vocabulary. Completely different layer, similar self-discipline: discover the precise layer of the stack the place interoperability is definitely assured, and refuse to imagine it holds one layer increased simply because the layers are adjoining.

The tokenizer is that layer. Every thing above it’s a form mismatch. Every thing at or beneath it, if we’re fortunate and if we examine, is a free integer array.

3. OKF: the “simply hand off the integers” sample

Right here is the pitch in 5 bullets:

1. Agent 1 masses solely the 7B mannequin’s tokenizer — by no means its weights. It splits the doc, tags every part, and tokenizes every part.
2. It saves every part’s token IDs as a NumPy int64 array into /dev/shm/qwen_tokens/. That could be a RAM-backed tmpfs mount, not disk, so studying it again is a memcpy, by no means a search.
3. It additionally writes one Markdown file per part into okf_workspace/. The Markdown physique is the part’s human-readable textual content. The YAML frontmatter carries the metadata — block_id, tags, token_pointer, token_count, tokenizer_model_id, and so forth.
4. Agent 2 (the 3B mannequin) reads the frontmatter, follows token_pointer into shared reminiscence, masses the .npy, and calls mannequin.generate(input_ids=...) immediately on the loaded tensor. No tokenizer name on the enter aspect.
5. Agent 2 tokenizes its personal output (that textual content has, by definition, by no means been tokenized earlier than — nothing to reuse), saves that array to shm, writes one other OKF file, and Agent 3 (1.5B) does the identical trick once more.

A fast introduction on the “OKF” (for many who don’t know but)

OKF stands for Open Data Format, and earlier than you learn the frontmatter block beneath, one factor is price being trustworthy about.

The Open Data Format is a printed spec — Google Cloud shipped v0.1 in June 2026 and v0.2 is now the present model (see GoogleCloudPlatform/knowledge-catalog on GitHub). Its pitch is deliberately minimal: a bundle is a listing of UTF-8 Markdown information, every file is one idea, and every file carries a YAML frontmatter block plus a Markdown physique. The one frontmatter subject the spec requires is kind — a brief human-readable string like BigQuery Desk, Playbook, or Attested Computation. Every thing else is elective metadata. It’s a format, not a platform: no schema registry, no SDK, no central authority. For those who can cat a file, you possibly can learn OKF.

This repo’s okf/ reuses that precise skeleton — one Markdown file per unit of labor, YAML frontmatter plus a human-readable physique — however interprets it for a job the overall spec was not written for: an agent-to-agent hand-off of pre-tokenized integer arrays. So this repo’s required frontmatter fields aren’t Google’s kind; they’re block_id, source_agent, stage, title, tags, token_pointer, token_count, tokenizer_model_id, and created_at (see utils/okf_parser.py‘s REQUIRED_FRONTMATTER_KEYS). The load-bearing one is token_pointer — an absolute path into /dev/shm/qwen_tokens/ — which has no equal within the normal OKF spec as a result of Google’s OKF was designed for sturdy information sharing, not for a shared-memory hand-off between short-lived agent processes on the identical GPU host. Put plainly: this repo’s information are not legitimate Google-OKF bundles as-is (they lack kind, they add token_pointer); the repo is conforming in spirit — similar Markdown+YAML aesthetic, similar “standardise the interoperability floor, not the content material mannequin” intuition — with one domain-specific required subject bolted on. This submit retains the repo’s terminology as a result of that’s what the supply code and the generated information truly use.

With that out of the best way, right here is the schema within the wild — the precise frontmatter block from okf_workspace/block_004_routing_and_signaling_integration_points.md, unedited:

---
block_id: block_004_routing_and_signaling_integration_points
source_agent: agent_1_architect
stage: 1
title: Routing and Signaling Integration Factors
tags:
- routing
- signaling
- safety
- deployment
token_pointer: /dev/shm/qwen_tokens/block_004_routing_and_signaling_integration_points.npy
token_count: 3437
tokenizer_model_id: Qwen/Qwen2.5-Coder-7B-Instruct
created_at: '2026-08-04T12:41:39.249368+00:00'
---

The load-bearing subject is token_pointer. Every thing else — source_agent, stage, tags, token_count, tokenizer_model_id, created_at — exists to help routing and provenance selections round that one array. Agent 2 filters the workspace by tag (routing or signaling, each set off it). Agent 3 filters by supply agent (agent_2_protocol_eval, so it by no means unintentionally picks up its personal output on a re-run). The tokenizer_model_id subject is there so a future audit can cross-check per-file which tokenizer truly produced the bytes at that path, as a substitute of trusting one pipeline-start assertion for all eternity.

Yet another architectural element price calling out: every agent is a separate OS course of. src/run_pipeline.py launches them by way of subprocess.run, one after the other. That’s deliberate, not lazy: a CUDA context solely releases its VRAM again to the motive force when the method holding it exits. So working three multi-GB fashions sequentially inside one course of would leak every prior mannequin’s VRAM into the following agent’s reminiscence finances except each caller remembered to manually del mannequin; torch.cuda.empty_cache() — and even that isn’t at all times ample to totally reclaim CUDA context overhead. Subprocess isolation makes VRAM launch unconditional and computerized. On a single-GPU field, that is what lets the 7B, then the 3B, then the 1.5B every get the entire card to themselves in flip, with out ever needing all three resident in reminiscence concurrently.

4. The precise save/load code, all six significant traces of it

Now the code that does the precise hand-off. From utils/token_manager.py, verbatim:

def save_token_array(token_ids: torch.Tensor, block_name: str) -> Path:
    ...
    token_ids_as_numpy_int64 = token_ids.detach().cpu().numpy().astype(TOKEN_ARRAY_DTYPE)
    destination_path = QWEN_TOKENS_SHM_DIR / f"{block_name}.npy"
    np.save(destination_path, token_ids_as_numpy_int64, allow_pickle=False)
    return destination_path

That’s the write half. Three traces that truly transfer information. QWEN_TOKENS_SHM_DIR is /dev/shm/qwen_tokens, a RAM-backed tmpfs mount. TOKEN_ARRAY_DTYPE is np.int64, matching torch’s default torch.lengthy, particularly so the load aspect by no means wants a casting step. And allow_pickle=False is there as a result of a .npy file with allow_pickle=True will fortunately deserialise and execute pickled Python objects from disk — pointless assault floor for an array that’s, by definition, pure numeric information.

Right here is the learn half:

def load_token_array(pointer_path: Path) -> torch.Tensor:
    ...
    token_ids_as_numpy_int64 = np.load(pointer_path, allow_pickle=False)
    if token_ids_as_numpy_int64.dtype != TOKEN_ARRAY_DTYPE:
        elevate TypeError(...)
    return torch.from_numpy(token_ids_as_numpy_int64)

Additionally three significant traces. np.load reads again the precise .npy header (which embeds dtype, form, and byte-order, all express), the defensive dtype examine refuses to silently .astype() if some future code path ever writes one thing apart from int64 into this namespace, and torch.from_numpy(...) shares reminiscence with the NumPy array — zero-copy, since token IDs from this level ahead are by no means mutated in place by any agent.

That’s the total on-wire format. A NumPy .npy file, int64, on a RAM-backed mount. For those who have been anticipating one thing unique, sorry to disappoint you.

The final piece of the puzzle is what a downstream agent truly does with the loaded tensor. From utils/model_loader.py, the 2 entry factors that Agent 2 and Agent 3 can name — the naive baseline, and the optimized path. Take a look at them aspect by aspect, as a result of the entire optimization is one operate name’s price of distinction:

def generate_from_text(mannequin, tokenizer, prompt_text, max_new_tokens):
    ...
    wall_clock_start = time.perf_counter()

    encoded_prompt = tokenizer(prompt_text, return_tensors="pt")

    input_ids = encoded_prompt["input_ids"].to(mannequin.machine)
    attention_mask = encoded_prompt["attention_mask"].to(mannequin.machine)

    return _generate_and_measure_ttft(
        mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
    )

Baseline. Clock begins earlier than tokenizer(...) runs, so the tokenizer-encode value this pipeline exists to skip is absolutely included within the reported TTFT. That’s not unintended — it’s intentionally trustworthy. If the baseline began its clock after tokenization, the comparability would understate the true financial savings and faux the tokenizer was free. It’s not free. It’s quick, however it’s not free.

Now the optimized aspect:

def generate_from_token_ids(mannequin, tokenizer, token_ids, max_new_tokens):
    ...
    wall_clock_start = time.perf_counter()

    input_ids = token_ids.unsqueeze(0).to(mannequin.machine)
    attention_mask = torch.ones_like(input_ids)

    return _generate_and_measure_ttft(
        mannequin, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
    )

The clock additionally begins right here, with no tokenizer name previous it — the entire level of the comparability. token_ids was already produced by an upstream agent’s tokenizer, already saved into shm, already loaded off shm. All this operate does earlier than beginning the mannequin is unsqueeze a batch dimension and duplicate the array to the GPU. The tokenizer argument remains to be handed in, however solely as a result of _generate_and_measure_ttft wants it to produce pad_token_id and to decode the output tokens again to textual content — the enter aspect genuinely by no means hits the tokenizer.

The only-line distinction between these two features — one line, tokenizer(prompt_text, ...) — is the whole financial savings. It sounds nearly too small to put in writing an article about. Preserve studying, as a result of the failure mode on the opposite aspect of “nearly too small” isn’t small in any respect.

5. The half the place I finished trusting the seller docs

Right here is the sentence from my very own undertaking notes that made me nervous sufficient to put in writing code as a substitute of simply transport the pipeline: “Qwen2.5-Coder is documented to share one tokenizer throughout the entire household.” Documented. By whom? Checked how lately? What occurs to a few brokers’ price of generated textual content if that seems to be true for six of the seven sizes and subtly not true for the one I picked?

A tokenizer mismatch right here doesn’t crash something. That’s the scary half. mannequin.generate(input_ids=[1234, 5678, ...]) doesn’t know or care whether or not 1234 meant the identical subword to whoever produced it because it means to the mannequin about to embed it. It’s going to fortunately run a ahead move on integers that decode to finish nonsense beneath its personal vocabulary, and it’ll fortunately generate a fluent-looking continuation of that nonsense. You get a confidently incorrect report, not an error. Your tokenizer: not the bottleneck. Your assumptions about your tokenizer: fully the bottleneck.

So earlier than any agent is allowed to belief a token array it didn’t produce itself, this runs — from utils/env_checks.py:

def verify_tokenizer_equivalence(
    model_ids: tuple[str, ...] = PIPELINE_MODEL_IDS,
) -> None:
    ...
    loaded_tokenizers = {
        model_id: AutoTokenizer.from_pretrained(model_id) for model_id in model_ids
    }

    reference_model_id = model_ids[0]
    reference_tokenizer = loaded_tokenizers[reference_model_id]
    reference_vocab_size = reference_tokenizer.vocab_size

    reference_vocab = reference_tokenizer.get_vocab()

    for candidate_model_id in model_ids[1:]:
        candidate_tokenizer = loaded_tokenizers[candidate_model_id]

        if candidate_tokenizer.vocab_size != reference_vocab_size:
            elevate RuntimeError(
                f"Tokenizer vocab_size mismatch: {reference_model_id} has "
                f"vocab_size={reference_vocab_size}, however {candidate_model_id} "
                f"has vocab_size={candidate_tokenizer.vocab_size}. Token IDs "
                "produced by one aren't secure to feed into the opposite's "
                "embedding layer."
            )

        if candidate_tokenizer.get_vocab() != reference_vocab:
            elevate RuntimeError(
                f"Tokenizer vocabulary mismatch between {reference_model_id} "
                f"and {candidate_model_id}: at the very least one token string maps "
                "to a special integer id between the 2. Direct token "
                "injection throughout these fashions would silently corrupt "
                "downstream generations."
            )

        if candidate_tokenizer.special_tokens_map != reference_tokenizer.special_tokens_map:
            elevate RuntimeError(
                f"Particular-tokens map mismatch between {reference_model_id} "
                f"({reference_tokenizer.special_tokens_map}) and "
                f"{candidate_model_id} ({candidate_tokenizer.special_tokens_map})."
            )

Three checks, intentionally layered.

The primary examine is vocab_size. It exists purely so a mismatch right here produces a brief, immediately-readable error naming the 2 integers that disagree, as a substitute of forcing whoever is debugging this to diff two ~151,936-entry dicts by hand to seek out that the sizes alone differ.

The second examine — the load-bearing one — is full dictionary equality on get_vocab(). Not a vocab_size comparability. A full dict != dict over the whole ~151,936-entry mapping of each subword string to each integer id. Two tokenizers can have an identical sizes and nonetheless disagree about what integer 42 means. That is the examine that may catch a “shuffled id task for even a single subword” mismatch, which is strictly the form of failure that produces fluent nonsense downstream as a substitute of a loud error.

The third examine is special_tokens_map. A mannequin’s chat template and stopping habits rely upon these precise strings/ids matching too — an accurate essential vocabulary with a divergent EOS id, for instance, would make a downstream agent’s generate() name fail to cease on the boundary Agent 1 meant.

I wished the precise assure, not a budget proxy for it. Ran it in opposition to the true triplet earlier than writing one other line of pipeline code, and it held: Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-1.5B-Instruct all agree, byte for byte. Good. However “it held, this time, for this triplet” is a really completely different sentence from “it’s documented to carry,” and solely a kind of two sentences belongs in a pipeline you’re going to run unattended.

6. The receipts

Identical 3 sections of the design doc (those Agent 1’s key phrase scan tagged routing or signaling — block_002 at 1948 tokens, block_003 at 2292 tokens, block_004 at 3437 tokens). Identical grasping decoding. Max 64 new tokens for the timed comparability. One throwaway warm-up name absorbed earlier than any timed measurement so cuBLAS’s first-call kernel choice doesn’t contaminate the numbers. Median of seven repeated trials per block, to easy out millisecond-scale scheduling and GPU-clock jitter.

Straight from scripts/benchmark.py‘s output:

=== Benchmarking Qwen/Qwen2.5-Coder-3B-Instruct ===
  Metric 1 (TTFT discount): mean_baseline=69.3 ms, mean_injection=49.9 ms, discount=28.0% -- PASS
  Metric 2 (semantic constancy): PASS

=== Benchmarking Qwen/Qwen2.5-Coder-1.5B-Instruct ===
  Metric 1 (TTFT discount): mean_baseline=49.6 ms, mean_injection=30.9 ms, discount=37.8% -- PASS
  Metric 2 (semantic constancy): PASS

[benchmark] ALL ACCEPTANCE METRICS PASSED

In desk type:

The attention-grabbing bit isn’t that each fashions obtained sooner — in fact they did, they stopped doing redundant work. The attention-grabbing bit is why the 1.5B mannequin’s proportion discount is noticeably larger than the 3B mannequin’s, though absolutely the variety of milliseconds saved is roughly comparable. The reason is within the repo’s personal README, and it’s price quoting as a result of it’s the form of factor that journeys folks up in the event that they solely learn the desk:

The tokenizer’s CPU value is identical string, tokenized as soon as, no matter which mannequin reads the outcome — however GPU forward-pass latency scales with mannequin measurement. For the smaller 1.5B mannequin, that GPU-side ground is decrease, so the (roughly fastened) tokenizer value it avoids is a bigger fraction of its whole time-to-first-token.

That can be, by the way, why blindly rising the enter doc additional doesn’t push the discount towards 100%. Previous a sure enter size, GPU compute time itself begins rising too, and the share plateaus reasonably than climbing indefinitely. The financial savings scale with how a lot textual content you’ll in any other case redundantly re-tokenize, occasions what number of downstream brokers share that very same enter, divided by how large every downstream mannequin’s personal ahead move is. On a brief single-hop demo, you get double-digit p.c. On a big supply doc fanned out to many downstream brokers of the identical household, you pay the BPE value as soon as as a substitute of N occasions, which is strictly the regime the plan was constructed for.

The semantic-fidelity aspect of the receipts is a heuristic, on goal. Two ratios: printable-character ratio ≥ 0.98, and unique-word ratio ≥ 0.25 throughout the pattern’s tokens. Low cost sufficient to run on each technology, calibrated to catch the precise “rubbish output” failure mode a tokenizer mismatch or byte-order bug produces — degenerate repetition of 1 token, or a wall of non-printable control-character noise — not a normal high quality judgment. Each pattern from each mannequin handed. Learn extra particulars in regards to the outcomes right here.

7. Wrap: the truly attention-grabbing half was the guardrail

The attention-grabbing a part of this undertaking was by no means “skip the tokenizer, it’s gradual.” Tokenizers, particularly the quick Rust-backed variety, aren’t the bottleneck anybody thinks they’re — the numbers above show that themselves. Saving 20 ms of TTFT is sweet. It’s not the purpose.

The attention-grabbing half was constructing the one piece of infrastructure that makes skipping the tokenizer secure: a runtime examine that refuses to let one agent belief one other agent’s integers till it has truly confirmed they converse the identical language, byte for byte, vocabulary entry for vocabulary entry. That examine is what turns “20 ms sooner” from a footgun right into a dependable engineering transfer. With out it, you’ve got a pipeline that’s quick when it really works and confidently incorrect when it doesn’t, and no clear strategy to inform which one you’re at present dwelling in.

Each multi-agent pipeline that passes state between fashions is making an assumption like this someplace, often silently. Generally it’s about tokenizer vocabularies. Generally it’s about hidden-state dimensions. Generally it’s in regards to the that means of a selected chat-template string. Generally it’s about which aspect of an RPC boundary the retries dwell on. Mine simply occurs to be about BPE integer-to-subword mappings, as a result of that’s what this repo’s optimization technique leans on. Yours is someplace else. Go discover it. It’s in all probability not documented both.

If you wish to reproduce the numbers, python scripts/benchmark.py on a CUDA GPU with sufficient VRAM for a bf16 3B checkpoint will do it. If you wish to reproduce the pipeline itself in opposition to your personal enter, drop your doc into information/raw_input.txt and python src/run_pipeline.py walks by the three phases, cleans up shm on the best way out, and leaves three OKF information behind in okf_workspace/.

Small pipeline. Modest numbers. One load-bearing examine. That’s the entire form of it.

Disclaimer: The illustrations on this article have been generated utilizing AI (Claude Opus 4.8). They’re illustrative, not photographic, and any labels seen inside the photographs are stylized reasonably than authoritative — confer with the article physique and the code itself for exact operate names, metric values, and structure particulars.