I construct agent programs that automate their operational workflows. Throughout my work, I’ve discovered that groups typically put most of their effort and time into engineering the agent harness, whereas giving far much less consideration to the product, workflow and human choices that decide the way it operates contained in the enterprise. When these choices aren’t thought by means of fastidiously, I typically see refined agent programs get deployed but change little or no about how work really will get carried out.
I’ve distilled my expertise into 5 rules for constructing agent programs that succeed inside a enterprise, earn individuals’s belief and enhance over time. This text explains every one by means of an agent system I constructed for a $100M+ managed IT supplier, to handle considered one of its most urgent issues on the time: consumer churn.
- Tailor the product form to the enterprise workflow
- Give every workflow step to the only mechanism that handles it properly
- Redesign the human’s job on the agent handoff
- Design the agent handoff for human verification and suggestions
- Construct a self-improvement loop from human suggestions
The case examine: Detecting churn threat hidden throughout service tickets
The corporate runs IT operations for its purchasers and was dealing with unprecedented consumer churn on the time. One of many clearest warning indicators is a service concern that retains coming again for a similar consumer. The issue is that this warning nearly by no means arrives as a warning.
As an alternative, it arrives as separate incidents unfold over weeks or months. Every incident appears small and self-contained. The industrial threat solely turns into seen when somebody notices that the identical underlying downside retains returning.
Recognizing that sample meant counting on busy individuals to attach the dots. Help technicians noticed particular person service tickets raised by purchasers. Account managers reviewed consumer contact factors and dashboards. For the danger to floor, somebody needed to bear in mind the sooner incidents, decide whether or not they fashioned a sample, collect the proof and alert the one who owned the consumer relationship.
Throughout many consumers and a excessive quantity of service tickets, connecting these dots by hand was prohibitively guide. The corporate couldn’t preserve tabs on each consumer and each threat sign persistently at scale.
Tailor the product form to the enterprise workflow
The frequent intuition I see most groups comply with is to place an agent system behind a chat interface. This firm tried that route, too. The staff first investigated churn by means of its enterprise knowledge agent, a separate system I’d beforehand constructed for them so staff may ask questions on firm knowledge in pure language.
However in my expertise, a chat interface is usually the unsuitable product form for many enterprise workflows, for two causes:
- It waits for somebody to consider the appropriate query on the proper second. Enterprise work often begins as a result of one thing occurred, not as a result of an individual remembered to ask about it. On this case, early churn detection has to start when new proof arrives and preserve working within the background.
- It makes worker adoption a prerequisite for the enterprise final result. For adoption to only work, the agent system has to run with out anybody needing to drive it. Ideally, it also needs to require minimal change to current behaviour.
The corporate’s churn threat workflow additionally relies on patterns that construct up over time. To run effectively, the system ought to retain what it has already realized about every consumer’s ticket historical past as an alternative of rebuilding that historical past in response to each query. For instance, a stateful background course of can retailer every consumer’s concern historical past and proceed from the place it left off, so each new run handles solely the tickets that arrived because the final one.
You might bolt these capabilities onto a chat-based agent with an agent talent that explains easy methods to retailer findings and course of solely new tickets. Nevertheless, as soon as that functionality carries persistent state and fires on a enterprise occasion, it already operates as a separate system. Hiding it behind chat doesn’t simplify the system, it solely disguises a background workflow as a dialog.
As such, I guided the staff away from treating churn detection as one other question-and-answer use case for its chat-based enterprise knowledge agent. As an alternative, I constructed a separate, event-driven agent system. This alternative shifts the system’s job from serving to staff examine churn threat to proudly owning the detection work itself. The system watches for brand spanking new proof, carries every consumer’s ticket historical past ahead and begins the appropriate workflow with out ready for somebody to immediate it.
Give every workflow step to the only mechanism that handles it properly
The error I see groups make most frequently is handing an agent your complete workflow. For instance, a price wants evaluating, so the agent compares it. Or a outcome wants routing, so the agent routes it. Placing all the things in a single immediate can look easier, however in manufacturing it folds precise guidelines, ambiguous investigation and industrial judgement into one opaque LLM run.
The strongest manufacturing programs I’ve seen and constructed don’t use probably the most refined agent at each step. They provide every step to the only mechanism that handles it properly, so the whole workflow stays correct, comprehensible and low cost to run at scale.
The true design work is deciding which steps belong to deterministic code or a specialised methodology, which want an agent and which have to stick with an individual. The agent is one decision-maker contained in the system and goes the place judgement is required, not all over the place by default.
For this case examine, the workflow takes this type: The system watches every consumer’s ticket historical past and teams incidents which may describe the identical underlying concern. When a bunch crosses the corporate’s definition of recurrence, an agent investigates whether or not the tickets characterize one persevering with downside and whether or not the issue deserves an account supervisor’s consideration. Solely a discovering that clears each bars turns into an alert for the account supervisor who owns that consumer.
The agent system I constructed runs this workflow in 2 linked phases:
- A once-off preparation builds a compact image of every consumer’s latest points.
- A scheduled weekly run compares new tickets with that image, investigates recurring patterns and escalates solely those that earn consideration.
A summarizer agent handles language judgement
In the course of the preparation stage, the system pulls every consumer’s final 3 months of service tickets from the ticketing system. A single ticket can maintain an extended e mail thread between the consumer and a assist technician, which is way extra context than the remainder of the workflow wants. To chop context measurement and noise in each downstream step, I compress every ticket to solely its helpful sign.
That is the kind of process an LLM shines at. As such, summarizer agent reads the total ticket and reduces it to at least one concise concern abstract whereas preserving sufficient element to determine and evaluate the issue.
Semantic similarity teams tickets with out utilizing an agent
The following step teams tickets which may describe the identical underlying concern. Similarity is probabilistic, so no precise rule can draw the road. However the system additionally doesn’t want an agent to motive by means of each potential pair, which might be sluggish and costly at quantity.
As such, I exploit semantic similarity right here. The system turns every concern abstract into an embedding, a numerical illustration of its which means, then makes use of clustering to construct the preliminary concern teams individually for every consumer. This manner, the system handles similarity instantly, cheaply and thru a repeatable methodology.
Deterministic code owns the recurrence threshold
The once-off preparation creates the preliminary concern teams. The scheduled weekly run comes subsequent. Each Monday morning, the system pulls the earlier week’s new tickets for every consumer and places every one by means of the identical summarize-and-embed steps. It then compares every new embedding with that consumer’s current concern teams. An in depth sufficient match joins the ticket to that group, whereas a ticket with no shut match begins a brand new group.
Deterministic code checks the corporate’s rule for X comparable tickets inside Y days. A threshold like that’s precise by definition, so it belongs in code quite than an agent which may apply it inconsistently.
If the brink is crossed, the system doesn’t but ship an alert to the account supervisor. Semantic similarity is just an affordable, fast first move at grouping tickets that is likely to be associated. Crossing the brink means the group would possibly include a recurring concern, not that the similarity match proves the consumer has one persevering with downside. An investigation agent has to make that decision.
An investigation agent judges the proof
When a problem group crosses the brink, the system begins an investigation agent for that group. The agent pulls the total historical past of each ticket within the group from the ticketing system. From there, it decides what else it wants and retrieves it from the identical supply, equivalent to particulars in regards to the consumer or the IT system at fault.
No rule or embedding can settle this half. The agent has to acquire proof and make 2 judgements: whether or not the tickets characterize one persevering with concern and whether or not the sample warrants an account supervisor’s consideration.
The two judgements keep separate as a result of a repeated sample and a helpful threat sign aren’t the identical factor. For instance, deliberate upkeep can generate a number of associated tickets with out pointing to a broken relationship, or a recurring technical fault might be real and nonetheless not deserve an account supervisor’s time. Solely a discovering that’s each actual and commercially related turns into an account supervisor alert.
A human handles the choice to act
Deterministic routing in code emails the alert to the account supervisor assigned to that consumer. The agent system doesn’t prescribe what the account supervisor ought to do subsequent. The account supervisor retains the connection judgement as a result of the appropriate response relies on the consumer and relationship context outdoors the technical proof.
Throughout the whole agent system, code handles precise guidelines; embeddings deal with semantic similarity; brokers deal with messy proof and investigations; people preserve the choice with actual industrial penalties. This division offers every step to the only mechanism that handles it properly, so the whole workflow stays correct, comprehensible and low cost to run at scale.
Redesign the human’s job on the agent handoff
Including an agent system to a enterprise workflow adjustments what the individuals in it do. Probably the most vital product choices is the place the system stops, what it palms over and what the people nonetheless personal.
An agent system often ends in considered one of 2 methods:
- It completes the workflow, whereas individuals spot-check its work every so often
- It stops earlier than the ultimate determination and offers an individual the entire context wanted to resolve and act shortly
Within the second case, the agent system is chargeable for filtering what wants human consideration, surfacing solely these circumstances and bringing the related context with them. Finished properly, that turns half-hour of a human reconstructing what occurred right into a 1-minute determination. The system completes the looking, evaluation and proof meeting that come earlier than the judgement name, then stops the place an individual’s authority nonetheless issues.
That is the corporate’s case, the place the account supervisor nonetheless owns the consumer relationship. The system emails a consolidated, absolutely investigated threat alert to whoever is assigned to that consumer. This implies the account supervisor not has to dig by means of tickets or piece collectively what occurred. They will spend their consideration on defending the connection utilizing the proof the system has already assembled.
However this handoff solely works if the account supervisor can belief the system’s findings.
Design the agent handoff for human verification and suggestions
The agent handoff is the ultimate interface an individual sees of the agent’s work.
Verification begins at product design, not on the finish
A background agent system raises the bar for verification. With a chat-based agent, the person sees every response because it arrives. However a background system can repeat a nasty judgement throughout many circumstances earlier than anybody notices.
Groups typically put most of their effort into the AI engineering, and the handoff will get no matter consideration is left. That misses one of many highest-leverage components of the system. A classy agent is ineffective if the individuals receiving its output can’t belief it sufficient to make use of it, and belief comes from verification and traceability. If the individual receiving the agent handoff has to redo the agent’s work to substantiate its conclusion, the system has uncovered the unsuitable proof.
That results in a easy design precept: Make the agent’s work straightforward to confirm earlier than constructing evaluations round its output. Verification can’t be added on the finish as a rating round an in any other case opaque reply. It has to form what the system produces, how the output is damaged down and which proof travels with each conclusion.
On this case, the agent handoff is the account supervisor alert. I designed that alert so reviewable proof is an intentional output, as an alternative of simply stating a threat stage and leaving the account supervisor to reconstruct why.
Begin with how a human does the work
Right here, don’t begin by asking what an alert usually appears like. Begin with how the individual receiving it already checks whether or not a discovering is appropriate. This implies understanding how an account supervisor confirms {that a} recurring consumer concern is actual.
They should see what the system believes is recurring, how typically it occurred, which tickets assist the conclusion and what every ticket says. Additionally they want entry to the supply file. This checking course of determines the handoff interface design.
The proof packet subsequently opens with the consumer, the recurring concern and its frequency. It then reveals the tickets that assist the discovering. Every ticket has a concise abstract and a direct hyperlink to its full file within the ticketing system.
Right here’s what an instance alert appears like:
Reveal the proof with out overwhelming the human
This interface makes use of progressive disclosure. The account supervisor sees the discovering and probably the most helpful ticket proof instantly. It reveals 2 consultant tickets and makes clear that 3 extra belong to the group. The reviewer doesn’t should learn each uncooked ticket earlier than deciding whether or not the sample is smart, however each conclusion stays linked to its supply of fact within the ticketing system.
This design reduces studying with out hiding provenance. When the proof is incomplete or contradictory, the agent states that beside the affected ticket as an alternative of presenting its conclusion as settled. The account supervisor can see what the system is aware of, what stays unsure and the place every declare got here from.
Gather suggestions at every agent determination level
Suggestions helps most when it identifies which a part of the agent system obtained one thing unsuitable. When a system comprises a number of agent determination factors, a generic thumbs-up or thumbs-down on its ultimate output can’t let you know the place the error occurred. That makes the system arduous to enhance.
Merchandise equivalent to ChatGPT and Claude use a generic ranking on the ultimate response as a result of one interface has to assist all the things from monetary evaluation to authorized work and frontend design. However a system constructed for one particular enterprise workflow has a bonus. Its vital choices are identified forward of time, so its interface can ask for suggestions precisely the place every judgement happens.
The design aim is to make such analysis a part of regular use. Account managers don’t have to depart their regular workflow to label summary AI outputs or repeat the looking and evaluation that produced the alert. They make 2 fast calls whereas reviewing a discovering they already care about:
- Is that this ticket a part of this recurring concern? This checks whether or not the system grouped the appropriate consumer incidents collectively.
- Is that this concern price flagging to you? This checks whether or not a confirmed recurring sample deserves the account supervisor’s consideration.
The two questions keep separate as a result of the agent system can group the tickets accurately and nonetheless be unsuitable that the sample issues commercially. Additionally they produce 2 completely different analysis indicators. Ticket-level solutions present whether or not the system teams incidents accurately, and issue-level solutions present whether or not a confirmed sample deserves consideration. As an alternative of 1 ranking on the ultimate alert, the staff learns which determination wants bettering.
Clicking any Sure or No opens an non-obligatory free-text field that captures the explanation behind the human judgement, which is what the development course of wants.
The simpler this suggestions is to offer throughout regular work, the extra probably individuals are to offer it. Breaking one massive discovering into smaller models additionally makes annotation inexpensive and produces extra helpful indicators for future analysis.
Let verification grow to be lighter as belief grows
The correct amount of seen proof can change as customers construct confidence within the agent system. Throughout an early rollout, the handoff interface can present ticket proof and agent reasoning by default so individuals can examine how the system behaves. Because the system earns belief, some element can collapse by default whereas the total proof stays obtainable.
The product ought to scale back the hassle required to evaluate the work as confidence grows, but it surely shouldn’t take away the flexibility to examine the work.
Construct a self-improvement loop from human suggestions
After amassing suggestions, the subsequent worthwhile step is to construct a loop that turns it into proposed enhancements for future runs. In my expertise, groups typically automate this loop too early. They level a coding agent like Codex or Claude Code on the suggestions and the agent system’s codebase, then ask it to make enhancements.
However automating this loop isn’t trivial. It should be taught which suggestions warrants a change, the place that change belongs, and easy methods to make a helpful change with out weakening behaviour that already works. A generic coding agent doesn’t have the corporate’s area data, style or priorities. As such, it may well typically miss the problems that matter.
As an alternative, I like to recommend constructing the primary model of this loop by hand. Begin with a batch of suggestions sufficiently small to evaluate fastidiously. For every merchandise, have a site skilled annotate whether or not it ought to change the system and, if that’s the case, what that ought to appear like. These annotations flip the staff’s implicit judgement into examples an automatic loop can use afterward.
When you’ve outlined what issues, a coding agent can then assist group associated suggestions, discover comparable agent runs, suggest updates to the harness, and step by step automate extra of this loop. I’ve discovered that coding brokers work finest once they scale the judgement you’ve developed by reviewing agent runs and suggestions your self, not once they allow you to keep away from wanting on the knowledge completely.
Closing Ideas
On this case examine, the agent system I helped the corporate construct turns a prohibitively guide consumer churn detection workflow right into a possible ongoing operation at >90% decrease prices.
However I feel price financial savings understate the worth of automating enterprise enterprise workflows. The extra sturdy profit is that agent programs seize and compound an organization’s working judgement. With out a system to seize it, that judgement lives in individuals’s expertise, discussions and one-off choices no one writes down. As soon as it sits inside an agent system, they begin shaping how work is dealt with the subsequent time.
Proper now, plenty of consideration goes to new mannequin releases and the way a lot smarter every one is. However each firm can entry the identical fashions. As an alternative, I feel what actually belongs to an organization is its amassed file of how its individuals resolve what issues, what proof they belief and when to behave. That file is how an organization builds an working functionality that will get extra worthwhile, and extra its personal, the longer it runs.
I had plenty of enjoyable scripting this, and hope you had enjoyable studying too! See you within the subsequent one 🙂
Sheila
