On this article, you’ll find out how an agent’s strategy to managing state — stateless or stateful — shapes each its implementation and the deployment structure constructed round it.
Matters we are going to cowl embody:
- What separates stateless from stateful brokers, and the tradeoffs every design imposes on scaling.
- Methods to implement a stateless agent that relies upon totally on the shopper to produce dialog historical past.
- Methods to implement a stateful agent that manages its personal reminiscence by way of a database layer.
Introduction
A earlier article laid out a complete architectural roadmap for AI agent deployment, analyzing the infrastructure wanted to carry brokers into manufacturing settings.
As a follow-up, we now flip to a basic, sensible query that needs to be answered earlier than any load balancer is configured: the place does the agent’s reminiscence reside? Brokers could deal with their state (the context gained thus far and the dialog historical past) in several methods, and this code-level resolution can considerably affect the whole deployment structure.
This text breaks down the 2 main paradigms for dealing with an agent’s state: stateless and stateful design. A simplified model of a real-world implementation, utilizing open language fashions served by way of the quick Groq API, will illustrate these concepts in follow.
Preliminary Setup
If that is the primary time you’re utilizing language fashions from Groq in a Python program, you’ll want to put in the required library: pip set up groq.
After that, we import it and set our Groq API key within the code under:
|
import os from groq import Groq
# Get an API key in https://console.groq.com/keys and set it right here os.environ[“GROQ_API_KEY”] = “PASTE_YOUR_GROQ_API_KEY_HERE”
# Initializing the shopper shopper = Groq()
# Utilizing an environment friendly mannequin from Groq: Llama 3.1 8B On the spot MODEL_ID = “llama-3.1-8b-instant” |
An essential setup resolution right here is the selection of a particular mannequin. llama-3.1-8b-instant is a extremely cost-efficient mannequin that’s, on the time of writing, generously supported on Groq’s 2026 free tier: it permits as much as 14,400 requests per day. That makes it an excellent alternative for illustrating the stateless and stateful agent paradigms under.
Stateless Brokers: Hearth and Overlook
Stateless brokers deal with every request as utterly remoted and impartial. The agent reads the consumer immediate, invokes the LLM inference engine, and delivers the output. As soon as that execution cycle ends, all the things is forgotten.
The Tradeoff
Architectures primarily based on stateless brokers could be scaled horizontally with outstanding ease. Since no consumer reminiscence is saved on a backend server, incoming requests could be forwarded to any out there occasion. There’s, nonetheless, an essential limitation in multi-turn conversations: the frontend should re-send the entire dialog historical past alongside each new request. Because of this, the context window grows with a snowballing impact, rapidly driving up token utilization.
Illustrative Instance
This runnable code illustrates, by way of a primary situation, how a stateless agent usually interacts with a Groq language mannequin.
First, we outline a stateless_agent operate that emulates an agent’s interplay with our chosen mannequin. Importantly, no state or reminiscence of the dialog is stored internally. As a substitute, the earlier dialog historical past can optionally be handed in as a parameter and appended to the present immediate. The API name to the Groq mannequin takes place in shopper.chat.completions.create().
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
def stateless_agent(immediate: str, provided_history: record = None) -> str: “”“ The agent depends utterly on the shopper to supply context. It retains no info from previous interactions in native reminiscence. ““” # Initializing with a system immediate messages = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# Appending no matter historical past the shopper offered if provided_history: messages.prolong(provided_history)
# Appending the brand new immediate messages.append({“function”: “consumer”, “content material”: immediate})
# The LLM processes the whole chain of messages response = shopper.chat.completions.create( mannequin=MODEL_ID, messages=messages, max_tokens=100 )
return response.selections[0].message.content material.strip() |
To know the constraints of a stateless agent, we simulate a easy user-model dialog by way of it:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
# — Testing the Stateless Agent —
print(“— Flip 1 —“) prompt_1 = “Hello, my identify is Alice and I’m studying about API infrastructure.” response_1 = stateless_agent(prompt_1) print(f“Agent: {response_1}”)
print(“n— Flip 2 (With out Consumer Context) —“) # The agent fails right here as a result of it retained no reminiscence of Flip 1 prompt_2 = “What’s my identify and what am I studying about?” response_2 = stateless_agent(prompt_2) print(f“Agent: {response_2}”)
print(“n— Flip 2 (With Consumer Context) —“) # The frontend MUST inject the historical past into the payload for the agent to succeed frontend_payload = [ {“role”: “user”, “content”: prompt_1}, {“role”: “assistant”, “content”: response_1} ] response_3 = stateless_agent(prompt_2, provided_history=frontend_payload) print(f“Agent: {response_3}”) |
Output:
|
—– Flip 1 —– Agent: Howdy Alice, good to meet you. Studying about API infrastructure can be a fascinating and rewarding matter. What particular facets of API infrastructure would you like to discover or focus on? Are you trying for info on API administration, safety, deployment, or one thing else?
—– Flip 2 (With out Consumer Context) —– Agent: Sadly, I don‘t have any details about you, together with your identify. Our dialog simply began, so I’m right here to assist you with any questions or subjects you‘d wish to study. Please be at liberty to share your identify and a subject you’re in studying about.
—– Flip 2 (With Consumer Context) —– Agent: Your identify is Alice, and you are studying about API infrastructure. |
The implementation is straightforward, however and not using a shopper or frontend that sends the total dialog historical past to the agent on each flip, the agent’s LLM lacks the context it must reply sure questions correctly.
Stateful Brokers: Context-driven Continuity
Below this strategy, the agent takes on the reminiscence burden itself. The shopper, in the meantime, solely must ship the latest consumer immediate along with a singular identifier, usually related to the present session. The agent then retrieves the session historical past or context from a database and appends the brand new message to it. As soon as the LLM inference has been processed, the agent updates the context within the database.
The Tradeoff
It is a a lot neater expertise from the shopper aspect. It additionally facilitates complicated and asynchronous workflows during which brokers could have to pause their execution and look forward to instruments, app responses, or human approval. However it all comes with a price: scaling this answer turns into a lot more durable, beginning with the necessity for a persistent database layer within the structure. In infrastructures that scale horizontally, methods corresponding to centralized reminiscence caching with Redis can also turn out to be essential to keep away from “localized amnesia”, the place a session’s historical past is stranded on the one occasion that occurred to serve the sooner turns.
Illustrative Instance
We illustrate the essential concepts behind a stateful agent by incorporating a “persistent” database layer. For simplicity, we use a tiny SQLite database. The secret’s to have the agent handle its personal dialog reminiscence as an alternative of relying on a frontend to supply it externally:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
import sqlite3 import json
# Initializing an in-memory SQLite database for pocket book testing conn = sqlite3.join(‘:reminiscence:’) cursor = conn.cursor() cursor.execute(”‘CREATE TABLE IF NOT EXISTS agent_memory (session_id TEXT PRIMARY KEY, historical past TEXT)’”) conn.commit()
def stateful_agent(session_id: str, new_prompt: str) -> str: “”“ The agent manages its personal state utilizing a database. The shopper solely sends the brand new immediate and their session ID. ““” # 1. Retrieving present state from the database cursor.execute(“SELECT historical past FROM agent_memory WHERE session_id=?”, (session_id,)) row = cursor.fetchone()
if row: conversation_history = json.hundreds(row[0]) else: # Initializing with system immediate for brand new classes conversation_history = [{“role”: “system”, “content”: “You are a helpful, concise assistant.”}]
# 2. Appending the brand new consumer immediate conversation_history.append({“function”: “consumer”, “content material”: new_prompt})
# 3. Processing the LLM name utilizing the retrieved historical past response = shopper.chat.completions.create( mannequin=MODEL_ID, messages=conversation_history, max_tokens=100 ).selections[0].message.content material.strip()
# 4. Updating the state with the assistant’s reply conversation_history.append({“function”: “assistant”, “content material”: response})
# 5. Saving the brand new state again to the database cursor.execute(”‘ INSERT INTO agent_memory (session_id, historical past) VALUES (?, ?) ON CONFLICT(session_id) DO UPDATE SET historical past=excluded.historical past ‘”, (session_id, json.dumps(conversation_history))) conn.commit()
return response |
Discover how the session identifier is used to question the related info from previous interactions within the dialog at hand.
Now let’s attempt all of it in a dialog much like the earlier one, however this time with the consumer asking the agent to recall the consumer’s personal identify:
|
# — Testing the Stateful Agent —
print(“— Flip 1 —“) print(f“Agent: {stateful_agent(‘user_123’, ‘Hello, I’m Bob and I need to scale my AI app.’)}”)
print(“n— Flip 2 —“) # Discover how the shopper NO LONGER sends the context payload. Simply the session ID. print(f“Agent: {stateful_agent(‘user_123’, ‘What was my identify once more?’)}”) |
Output:
|
—– Flip 1 —– Agent: Howdy Bob, scaling an AI app can be a complicated course of. Could I ask:
1. What kind of AI know-how is your app constructed on? (e.g., machine studying, pure language processing, laptop imaginative and prescient) 2. Are you utilizing any cloud companies like AWS, Google Cloud, or Azure? 3. What are your scalability objectives (e.g., improve consumer rely, scale back latency, enhance response occasions)?
This info will assist me higher perceive your necessities and present extra efficient help.
—– Flip 2 —– Agent: Your identify is Bob. |
This instance is, in fact, a good distance from a scaled-up manufacturing structure, nevertheless it serves to make clear the important thing distinction between how stateful and stateless brokers work.
Wrapping Up: The Tradeoffs
The selection between a stateful and a stateless architectural design boils all the way down to correctly matching the infrastructure to the workflow:
- Stateless brokers are most popular in easy pipelines oriented to very particular duties, like textual content extraction, summarization, or single-turn classification chatbots. They preserve the structure light-weight, which is normally sufficient in such use instances, avoiding database bottlenecks and permitting seamless horizontal scaling.
- Stateful brokers make far more sense if we intend to develop long-running assistants, coding assistants, or multi-turn bots in purposes like customer support. As a result of the agent owns the historical past, the shopper payload stays small on each flip, and the dialog could be trimmed or summarized server-side as an alternative of being resent in full because it grows.
