In November 2016, three colleagues and I at IBM filed a patent software with a title that sounds nearly quaint at present: “Actual-Time Evaluation of a Musical Efficiency Utilizing Analytics” (printed as US 2018/0137425 A1 [1]).
The state of affairs we described was a rock live performance. A guitarist performs an instrument with an embedded sensor, cameras learn the viewers’s faces and physique language, followers fireplace likes and dislikes from their telephones, and a system fuses all of it, in actual time, into one judgment: is that this efficiency going nicely? If not, it tells the performer, whereas there’s nonetheless time to repair it. In a single embodiment, an LED on the neck of the guitar lights up inexperienced when the half is performed accurately, flashes yellow when the tempo begins to waver, and turns purple on a fallacious be aware.
We known as the core of it an “artificially clever analytics engine,” as a result of in 2016 that was the vocabulary we had.
Ten years later I design information architectures for agentic AI techniques in manufacturing, and not too long ago I re-read our patent with at present’s eyes. The expertise was uncanny. What we specified, element by element and step-by-step, is the anatomy of an AI agent: multimodal notion, context aggregation, inference grounded in episodic and semantic reminiscence, an analysis perform, and actuation by a human within the loop, all beneath a tough real-time constraint. We didn’t have the language for it. We had the loop.
This text walks by that mapping actually: what a 2016 decision-support spec received structurally proper about brokers, and, simply as instructive, what it received fallacious or couldn’t see in any respect.
What we truly specified
Strip away the patent prose and the system has seven components.
Notion, in three modalities. Sensors embedded in devices and microphones report goal efficiency parameters: timing, pitch, intonation, quantity (the spec calls them sensors 113a, and contains MIDI, tactile, vibrational, ultrasonic, strain and infrared variants). Cameras seize the viewers’s involuntary alerts: facial expressions and physique language, explicitly configured for low mild and statistically important protection of the room. And a community path carries deliberate suggestions: rankings, likes, dislikes and feedback submitted from private gadgets to an internet service.
A suggestions aggregator that receives all the pieces, tags every factor by supply and kind, kinds it into per-performer streams, and reformats it for evaluation. Studying it now, that is context engineering: the element that decides what the reasoning engine truly sees, in what form and in what order.
An analytics engine that infers traits of the continuing efficiency. The claims are exact about what grounds the inference, and that is the sentence that stopped me on re-reading: the process is “a perform of a repository of archived details about previous performances by the a number of performers and of a knowledgebase from which the suggestions could also be given semantic which means.” In 2026 phrases: inference as a perform of episodic reminiscence (what occurred in previous performances, together with which tempos traditionally correlated with audiences dropping curiosity) and semantic reminiscence (what the area is aware of, together with the rating being carried out). We even specified the comparability of stay sensor information towards the saved rating to detect {that a} guitarist will not be taking part in a couple of fallacious notes however is within the fallacious key.
A efficiency index: a handmade analysis perform that fuses optimistic tweets, likes, facial sentiment, body-language sentiment and goal accuracy into one quantity, normalized, thresholded, and tracked over time.
A reporting module that closes the loop to the human: a graph, a named sentiment, a suggestion (“finish the bass solo”), or that LED on the guitar neck.
And wrapped round all of it, a real-time funds: the spec insists the loop should full quick sufficient for the performer to behave through the efficiency, ideally beneath a second, at worst inside a couple of.
Sensors in, aggregation, memory-grounded inference, analysis, actuation, iteration. Steps 200 by 280, “carried out repeatedly, as an iterative process.” It’s a sense-think-act loop with reminiscence. In 2016 the taxonomy known as this resolution assist. Right now we’d name the identical diagram an agent, with one deliberate design selection: the actuator is a human.
The mapping, element by element
2016 spec, and its 2026 equal:
Embedded sensors, cameras, social rankings: multimodal device and occasion ingestion. The perception that involuntary alerts (faces, physique language) and deliberate alerts (rankings) are completely different courses of proof, to be fused somewhat than averaged blindly, is context-quality considering that many agent pipelines nonetheless lack.
Suggestions aggregator: the context meeting layer. Tagging by provenance, per-entity streams, format normalization. Right now that is the retrieval and packing stage in entrance of the mannequin, and it’s nonetheless the place most manufacturing failures originate.
Historic database plus knowledgebase: episodic plus semantic reminiscence. The patent treats inference as explicitly a perform of each. That’s the memory-grounded structure that agentic techniques at the moment are converging on, after a detour by stateless prompting.
Efficiency index: the eval. A linear, hand-weighted, interpretable scoring perform with thresholds and persistence situations. Primitive subsequent to a discovered reward mannequin, and in addition auditable in a approach most discovered rewards will not be. Regulated industries are rediscovering the advantage of that trade-off.
Reporting module with suggestions: actuation with a human within the loop, and an autonomy dial. The spec distinguishes between displaying an index (pure info), naming an inferred state (“the viewers is bored”), and recommending a particular motion (“improve your quantity”). Three escalating ranges of company, with the human at all times executing. In 2026, selecting the best level on that dial per motion class is among the central design choices in enterprise brokers.
Actual-time constraint: the latency funds. The patent’s insistence that suggestions should arrive whereas the efficiency can nonetheless be corrected is identical self-discipline that separates brokers that function in workflows from brokers that write studies about them.
What we received structurally proper
4 issues maintain up, they usually maintain up as a result of they’re information engineering, not mannequin science.
First, the loop is the product. The worth was by no means in any single inference however within the closed cycle of notion, analysis and correction. Groups that at present bolt a mannequin onto a workflow with out closing the loop rediscover this the arduous approach.
Second, inference should be grounded in reminiscence. Our engine with out the historic repository and the knowledgebase would have been a sentiment classifier shouting into the void. The identical is true of a contemporary agent with out episodic and semantic shops: fluent, and structurally incapable of studying from Tuesday for Wednesday.
Third, proof has provenance and sophistication. Fusing goal telemetry, inferred sentiment and deliberate rankings as distinct sign households, every with its personal reliability profile, is precisely how a critical agent ought to weigh device outputs towards retrieved paperwork towards person assertions.
Fourth, the human is an architectural element, not an afterthought. The performer is the actuator, and the system’s complete design respects the constraints of that actuator: consideration, timing, cognitive load. An LED that turns purple is a masterpiece of interface financial system beneath these constraints.
What a decade exposes
Now the humbling half, as a result of the gaps train greater than the hits.
The intelligence was the bottleneck, and we couldn’t admit it. Our “artificially clever analytics process” was guidelines, ontologies and classifiers. Fusing frowning faces, sarcastic tweets and MIDI streams right into a judgment like “the viewers is dropping curiosity as a result of the tempo dropped” was, in 2016, largely aspiration. The loop was proper and the reasoner was weak. Basis fashions inverted that: at present the reasoner is robust and it’s all the pieces round it, the loop, the reminiscence, the context, that’s weak in most deployments. The lesson cuts each methods: structure with out intelligence stalls, intelligence with out structure flails.
Reminiscence was append-only, and forgetting didn’t exist. Our historic database solely grew. Nothing decayed, nothing expired, nothing may very well be provably deleted. In 2016 that appeared like a characteristic. Right now, an agent reminiscence with out designed forgetting is a legal responsibility with glorious recall: stale information poisoning choices, and private information accumulating with out lifecycle. Retention, decay and provable erasure are architectural capabilities, not cleanup scripts.
Permissions have been nowhere. Re-read with 2026 eyes, essentially the most hanging absence in all the doc: we cheerfully specified cameras performing sentiment evaluation on the faces of particular person viewers members, with zoom lenses to pick particular folks, and steady per-member sentiment streams, and there’s not one paragraph about consent, entitlement or entry. We filed in November 2016; the GDPR turned relevant eighteen months later [2] and would have had sturdy opinions about each determine in that patent. Right now I spend a big a part of my working life on precisely this layer: what an AI system is allowed to watch, retain and act upon, on whose behalf, enforced within the information layer itself. The 2016 blind spot is the 2026 self-discipline.
Nothing discovered. The system inferred and really useful, however no consequence ever up to date the system itself. The weights, in each sense, have been frozen. The hole between reminiscence as accumulation and studying as enchancment remains to be one of many trustworthy open issues in agentic AI, and pretending retrieval alone closes it repeats our 2016 optimism in new vocabulary.
The lesson for agent builders
Here’s what I take from my very own decade-old doc.
The agentic sample will not be new. Notion, aggregation, memory-grounded inference, analysis, bounded actuation: engineers have been drawing this loop for a very long time, in resolution assist, in management techniques, in monitoring. What modified within the final three years is that the reasoning element lastly works. That could be a real revolution, and it is usually exactly one element of seven.
Which implies the scarce talent in agentic AI will not be novel. It’s the information architect’s self-discipline utilized to a brand new reasoner: design the reminiscence with lifecycle and forgetting, engineer the context with provenance, make the analysis auditable, put entitlements within the information layer, respect the latency funds, and select intentionally the place the human sits within the loop. Each a kind of considerations was seen in 2016, no less than in embryo, to 4 engineers fascinated about rock concert events. Each considered one of them is the place manufacturing brokers succeed or fail in 2026.
We didn’t lack the thought. We lacked the mannequin. Now the business has the mannequin, and what it lacks, deployment after deployment, is the self-discipline round it. The LED on the guitar neck nonetheless appears to me the fitting ambition for the entire discipline: a system that is aware of sufficient, remembers sufficient, and is allowed to know sufficient, to let you know one true and well timed factor, whereas there’s nonetheless time to behave on it.
References
[1] S. D’Alo’, M. Lerro, M. Noioso and N. Piazza, Actual-time evaluation of a musical efficiency utilizing analytics (2018), US Patent Utility US 2018/0137425 A1
[2] European Parliament and Council of the European Union, Regulation (EU) 2016/679 (Common Knowledge Safety Regulation) (2016), Official Journal of the European Union
