In Half 1 of this tutorial sequence, we launched AI Brokers, autonomous packages that carry out duties, make selections, and talk with others.
In Half 2 of this tutorial sequence, we understood the right way to make the Agent attempt to retry till the duty is accomplished by Iterations and Chains.
A single Agent can often function successfully utilizing a instrument, however it may be much less efficient when utilizing many instruments concurrently. One method to sort out difficult duties is thru a “divide-and-conquer” method: create a specialised Agent for every process and have them work collectively as a Multi-Agent System (MAS).
In a MAS, a number of brokers collaborate to attain frequent objectives, usually tackling challenges which can be too tough for a single Agent to deal with alone. There are two primary methods they will work together:
- Sequential circulate – The Brokers do their work in a particular order, one after the opposite. For instance, Agent 1 finishes its process, after which Agent 2 makes use of the consequence to do its process. That is helpful when duties rely upon one another and should be achieved step-by-step.
- Hierarchical circulate – Often, one higher-level Agent manages the entire course of and provides directions to decrease stage Brokers which concentrate on particular duties. That is helpful when the ultimate output requires some back-and-forth.
On this tutorial, I’m going to point out the right way to construct from scratch various kinds of Multi-Agent Methods, from easy to extra superior. I’ll current some helpful Python code that may be simply utilized in different comparable instances (simply copy, paste, run) and stroll by each line of code with feedback in an effort to replicate this instance (hyperlink to full code on the finish of the article).
Setup
Please confer with Half 1 for the setup of Ollama and the principle LLM.
import ollama
llm = "qwen2.5" On this instance, I’ll ask the mannequin to course of pictures, due to this fact I’m additionally going to want a Imaginative and prescient LLM. It’s a specialised model of a Giant Language Mannequin that, integrating NLP with CV, is designed to grasp visible inputs, akin to pictures and movies, along with textual content.
Microsoft’s LLaVa is an environment friendly selection as it could additionally run with no GPU.
After the obtain is accomplished, you’ll be able to transfer on to Python and begin writing code. Let’s load a picture in order that we are able to check out the Imaginative and prescient LLM.
from matplotlib import picture as pltimg, pyplot as plt
image_file = "draghi.jpeg"
plt.imshow(pltimg.imread(image_file))
plt.present()With a purpose to take a look at the Imaginative and prescient LLM, you’ll be able to simply move the picture as an enter:
import ollama
ollama.generate(mannequin="llava",
immediate="describe the picture",
pictures=[image_file])["response"]
Sequential Multi-Agent System
I shall construct two Brokers that may work in a sequential circulate, one after the opposite, the place the second takes the output of the primary as an enter, identical to a Chain.
- The primary Agent should course of a picture offered by the consumer and return a verbal description of what it sees.
- The second Agent will search the web and attempt to perceive the place and when the image was taken, primarily based on the outline offered by the primary Agent.
Each Brokers shall use one Software every. The primary Agent could have the Imaginative and prescient LLM as a Software. Please keep in mind that with Ollama, to be able to use a Software, the perform should be described in a dictionary.
def process_image(path: str) -> str:
return ollama.generate(mannequin="llava", immediate="describe the picture", pictures=[path])["response"]
tool_process_image = {'sort':'perform', 'perform':{
'identify': 'process_image',
'description': 'Load a picture for a given path and describe what you see',
'parameters': {'sort': 'object',
'required': ['path'],
'properties': {
'path': {'sort':'str', 'description':'the trail of the picture'},
}}}}The second Agent ought to have a web-searching Software. Within the earlier articles of this tutorial sequence, I confirmed the right way to leverage the DuckDuckGo bundle for looking out the net. So, this time, we are able to use a brand new Software: Wikipedia (pip set up wikipedia==1.4.0). You’ll be able to instantly use the unique library or import the LangChain wrapper.
from langchain_community.instruments import WikipediaQueryRun
from langchain_community.utilities import WikipediaAPIWrapper
def search_wikipedia(question:str) -> str:
return WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper()).run(question)
tool_search_wikipedia = {'sort':'perform', 'perform':{
'identify': 'search_wikipedia',
'description': 'Search on Wikipedia by spending some key phrases',
'parameters': {'sort': 'object',
'required': ['query'],
'properties': {
'question': {'sort':'str', 'description':'The enter should be brief key phrases, not an extended textual content'},
}}}}
## take a look at
search_wikipedia(question="draghi")
First, it is advisable write a immediate to explain the duty of every Agent (the extra detailed, the higher), and that would be the first message within the chat historical past with the LLM.
immediate = '''
You're a photographer that analyzes and describes pictures in particulars.
'''
messages_1 = [{"role":"system", "content":prompt}]One essential choice to make when constructing a MAS is whether or not the Brokers ought to share the chat historical past or not. The administration of chat historical past depends upon the design and targets of the system:
- Shared chat historical past – Brokers have entry to a typical dialog log, permitting them to see what different Brokers have mentioned or achieved in earlier interactions. This could improve the collaboration and the understanding of the general context.
- Separate chat historical past – Brokers solely have entry to their very own interactions, focusing solely on their very own communication. This design is usually used when impartial decision-making is essential.
I like to recommend retaining the chats separate until it’s essential to do in any other case. LLMs may need a restricted context window, so it’s higher to make the historical past as lite as potential.
immediate = '''
You're a detective. You learn the picture description offered by the photographer, and also you search Wikipedia to grasp when and the place the image was taken.
'''
messages_2 = [{"role":"system", "content":prompt}]For comfort, I shall use the perform outlined within the earlier articles to course of the mannequin’s response.
def use_tool(agent_res:dict, dic_tools:dict) -> dict:
## use instrument
if "tool_calls" in agent_res["message"].keys():
for instrument in agent_res["message"]["tool_calls"]:
t_name, t_inputs = instrument["function"]["name"], instrument["function"]["arguments"]
if f := dic_tools.get(t_name):
### calling instrument
print('🔧 >', f"x1b[1;31m{t_name} -> Inputs: {t_inputs}x1b[0m")
### tool output
t_output = f(**tool["function"]["arguments"])
print(t_output)
### remaining res
res = t_output
else:
print('🤬 >', f"x1b[1;31m{t_name} -> NotFoundx1b[0m")
## don't use tool
if agent_res['message']['content'] != '':
res = agent_res["message"]["content"]
t_name, t_inputs = '', ''
return {'res':res, 'tool_used':t_name, 'inputs_used':t_inputs}As we already did in earlier tutorials, the interplay with the Brokers will be began with a whereas loop. The consumer is requested to offer a picture that the primary Agent will course of.
dic_tools = {'process_image':process_image,
'search_wikipedia':search_wikipedia}
whereas True:
## consumer enter
attempt:
q = enter('📷 > give me the picture to investigate:')
besides EOFError:
break
if q == "stop":
break
if q.strip() == "":
proceed
messages_1.append( {"position":"consumer", "content material":q} )
plt.imshow(pltimg.imread(q))
plt.present()
## Agent 1
agent_res = ollama.chat(mannequin=llm,
instruments=[tool_process_image],
messages=messages_1)
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"]
print("👽📷 >", f"x1b[1;30m{res}x1b[0m")
messages_1.append( {"role":"assistant", "content":res} )
The first Agent used the Vision LLM Tool and recognized text within the image. Now, the description will be passed to the second Agent, which shall extract some keywords to search Wikipedia.
## Agent 2
messages_2.append( {"role":"system", "content":"-Picture: "+res} )
agent_res = ollama.chat(model=llm,
tools=[tool_search_wikipedia],
messages=messages_2)
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"] 
The second Agent used the Software and extracted info from the net, primarily based on the outline offered by the primary Agent. Now, it could course of every part and provides a remaining reply.
if tool_used == "search_wikipedia":
messages_2.append( {"position":"system", "content material":"-Wikipedia: "+res} )
agent_res = ollama.chat(mannequin=llm, instruments=[], messages=messages_2)
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"]
else:
messages_2.append( {"position":"assistant", "content material":res} )
print("👽📖 >", f"x1b[1;30m{res}x1b[0m")
This is literally perfect! Let’s move on to the next example.
Hierarchical Multi-Agent System
Imagine having a squad of Agents that operates with a hierarchical flow, just like a human team, with distinct roles to ensure smooth collaboration and efficient problem-solving. At the top, a manager oversees the overall strategy, talking to the customer (the user), making high-level decisions, and guiding the team toward the goal. Meanwhile, other team members handle operative tasks. Just like humans, Agents can work together and delegate tasks appropriately.
I shall build a tech team of 3 Agents with the objective of querying a SQL database per user’s request. They must work in a hierarchical flow:
- The Lead Agent talks to the user and understands the request. Then, it decides which team member is the most appropriate for the task.
- The Junior Agent has the job of exploring the db and building SQL queries.
- The Senior Agent shall review the SQL code, correct it if necessary, and execute it.
LLMs know how to code by being exposed to a large corpus of both code and natural language text, where they learn patterns, syntax, and semantics of programming languages. The model learns the relationships between different parts of the code by predicting the next token in a sequence. In short, LLMs can generate SQL code but can’t execute it, Agents can.
First of all, I am going to create a database and connect to it, then I shall prepare a series of Tools to execute SQL code.
## Read dataset
import pandas as pd
dtf = pd.read_csv('http://bit.ly/kaggletrain')
dtf.head(3)
## Create dbimport sqlite3
dtf.to_sql(index=False, name="titanic",
con=sqlite3.connect("database.db"),
if_exists="replace")
## Connect db
from langchain_community.utilities.sql_database import SQLDatabase
db = SQLDatabase.from_uri("sqlite:///database.db")
Let’s start with the Junior Agent. LLMs don’t need Tools to generate SQL code, but the Agent doesn’t know the table names and structure. Therefore, we need to provide Tools to investigate the database.
from langchain_community.tools.sql_database.tool import ListSQLDatabaseTool
def get_tables() -> str:
return ListSQLDatabaseTool(db=db).invoke("")
tool_get_tables = {'type':'function', 'function':{
'name': 'get_tables',
'description': 'Returns the name of the tables in the database.',
'parameters': {'type': 'object',
'required': [],
'properties': {}
}}}
## take a look at
get_tables()That may present the obtainable tables within the db, and this can print the columns in a desk.
from langchain_community.instruments.sql_database.instrument import InfoSQLDatabaseTool
def get_schema(tables: str) -> str:
instrument = InfoSQLDatabaseTool(db=db)
return instrument.invoke(tables)
tool_get_schema = {'sort':'perform', 'perform':{
'identify': 'get_schema',
'description': 'Returns the identify of the columns within the desk.',
'parameters': {'sort': 'object',
'required': ['tables'],
'properties': {'tables': {'sort':'str', 'description':'desk identify. Instance Enter: table1, table2, table3'}}
}}}
## take a look at
get_schema(tables='titanic')Since this Agent should use multiple Software which could fail, I’ll write a strong immediate, following the construction of the earlier article.
prompt_junior = '''
[GOAL] You're a knowledge engineer who builds environment friendly SQL queries to get knowledge from the database.
[RETURN] It's essential to return a remaining SQL question primarily based on consumer's directions.
[WARNINGS] Use your instruments solely as soon as.
[CONTEXT] With a purpose to generate the right SQL question, it is advisable know the identify of the desk and the schema.
First ALWAYS use the instrument 'get_tables' to seek out the identify of the desk.
Then, you MUST use the instrument 'get_schema' to get the columns within the desk.
Lastly, primarily based on the knowledge you bought, generate an SQL question to reply consumer query.
'''Transferring to the Senior Agent. Code checking doesn’t require any explicit trick, you’ll be able to simply use the LLM.
def sql_check(sql: str) -> str:
p = f'''Double verify if the SQL question is right: {sql}. You MUST simply SQL code with out feedback'''
res = ollama.generate(mannequin=llm, immediate=p)["response"]
return res.change('sql','').change('```','').change('n',' ').strip()
tool_sql_check = {'sort':'perform', 'perform':{
'identify': 'sql_check',
'description': 'Earlier than executing a question, at all times evaluate the SQL question and proper the code if mandatory',
'parameters': {'sort': 'object',
'required': ['sql'],
'properties': {'sql': {'sort':'str', 'description':'SQL code'}}
}}}
## take a look at
sql_check(sql='SELECT * FROM titanic TOP 3')
Executing code on the database is a special story: LLMs can’t try this alone.
from langchain_community.instruments.sql_database.instrument import QuerySQLDataBaseTool
def sql_exec(sql: str) -> str:
return QuerySQLDataBaseTool(db=db).invoke(sql)
tool_sql_exec = {'sort':'perform', 'perform':{
'identify': 'sql_exec',
'description': 'Execute a SQL question',
'parameters': {'sort': 'object',
'required': ['sql'],
'properties': {'sql': {'sort':'str', 'description':'SQL code'}}
}}}
## take a look at
sql_exec(sql='SELECT * FROM titanic LIMIT 3')
And naturally, a great immediate.
prompt_senior = '''[GOAL] You're a senior knowledge engineer who evaluations and execute the SQL queries written by others.
[RETURN] It's essential to return knowledge from the database.
[WARNINGS] Use your instruments solely as soon as.
[CONTEXT] ALWAYS verify the SQL code earlier than executing on the database.First ALWAYS use the instrument 'sql_check' to evaluate the question. The output of this instrument is the proper SQL question.You MUST use ONLY the proper SQL question whenever you use the instrument 'sql_exec'.'''Lastly, we will create the Lead Agent. It has a very powerful job: invoking different Brokers and telling them what to do. There are a lot of methods to attain that, however I discover making a easy Software probably the most correct one.
def invoke_agent(agent:str, directions:str) -> str:
return agent+" - "+directions if agent in ['junior','senior'] else f"Agent '{agent}' Not Discovered"
tool_invoke_agent = {'sort':'perform', 'perform':{
'identify': 'invoke_agent',
'description': 'Invoke one other Agent to give you the results you want.',
'parameters': {'sort': 'object',
'required': ['agent', 'instructions'],
'properties': {
'agent': {'sort':'str', 'description':'the Agent identify, one in all "junior" or "senior".'},
'directions': {'sort':'str', 'description':'detailed directions for the Agent.'}
}
}}}
## take a look at
invoke_agent(agent="intern", directions="construct a question")Describe within the immediate what sort of conduct you’re anticipating. Attempt to be as detailed as potential, for hierarchical Multi-Agent Methods can get very complicated.
prompt_lead = '''
[GOAL] You're a tech lead.
You might have a group with one junior knowledge engineer known as 'junior', and one senior knowledge engineer known as 'senior'.
[RETURN] It's essential to return knowledge from the database primarily based on consumer's requests.
[WARNINGS] You're the just one that talks to the consumer and will get the requests from the consumer.
The 'junior' knowledge engineer solely builds queries.
The 'senior' knowledge engineer checks the queries and execute them.
[CONTEXT] First ALWAYS ask the customers what they need.
Then, you MUST use the instrument 'invoke_agent' to move the directions to the 'junior' for constructing the question.
Lastly, you MUST use the instrument 'invoke_agent' to move the directions to the 'senior' for retrieving the info from the database.
'''I shall maintain chat historical past separate so every Agent will know solely a particular a part of the entire course of.
dic_tools = {'get_tables':get_tables,
'get_schema':get_schema,
'sql_exec':sql_exec,
'sql_check':sql_check,
'Invoke_agent':invoke_agent}
messages_junior = [{"role":"system", "content":prompt_junior}]
messages_senior = [{"role":"system", "content":prompt_senior}]
messages_lead = [{"role":"system", "content":prompt_lead}]All the things is able to begin the workflow. After the consumer begins the chat, the primary to reply is the Chief, which is the one one which instantly interacts with the human.
whereas True:
## consumer enter
q = enter('🙂 >')
if q == "stop":
break
messages_lead.append( {"position":"consumer", "content material":q} )
## Lead Agent
agent_res = ollama.chat(mannequin=llm, messages=messages_lead, instruments=[tool_invoke_agent])
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"]
agent_invoked = res.break up("-")[0].strip() if len(res.break up("-")) > 1 else ''
directions = res.break up("-")[1].strip() if len(res.break up("-")) > 1 else ''
###-->CODE TO INVOKE OTHER AGENTS HERE<--###
## Lead Agent remaining response print("👩💼 >", f"x1b[1;30m{res}x1b[0m") messages_lead.append( {"role":"assistant", "content":res} )
The Lead Agent decided to invoke the Junior Agent giving it some instruction, based on the interaction with the user. Now the Junior Agent shall start working on the query.
## Invoke Junior Agent
if agent_invoked == "junior":
print("😎 >", f"x1b[1;32mReceived instructions: {instructions}x1b[0m")
messages_junior.append( {"role":"user", "content":instructions} )
### use the tools
available_tools = {"get_tables":tool_get_tables, "get_schema":tool_get_schema}
context = ''
while available_tools:
agent_res = ollama.chat(model=llm, messages=messages_junior,
tools=[v for v in available_tools.values()])
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"]
if tool_used:
available_tools.pop(tool_used)
context = context + f"nTool used: {tool_used}. Output: {res}" #->add instrument utilization context
messages_junior.append( {"position":"consumer", "content material":context} )
### response
agent_res = ollama.chat(mannequin=llm, messages=messages_junior)
dic_res = use_tool(agent_res, dic_tools)
res = dic_res["res"]
print("😎 >", f"x1b[1;32m{res}x1b[0m")
messages_junior.append( {"role":"assistant", "content":res} )
The Junior Agent activated all its Tools to explore the database and collected the necessary information to generate some SQL code. Now, it must report back to the Lead.
## update Lead Agent
context = "Junior already wrote this query: "+res+ "nNow invoke the Senior to review and execute the code."
print("👩💼 >", f"x1b[1;30m{context}x1b[0m")
messages_lead.append( {"role":"user", "content":context} )
agent_res = ollama.chat(model=llm, messages=messages_lead, tools=[tool_invoke_agent])
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"]
agent_invoked = res.break up("-")[0].strip() if len(res.break up("-")) > 1 else ''
directions = res.break up("-")[1].strip() if len(res.break up("-")) > 1 else ''
The Lead Agent acquired the output from the Junior and requested the Senior Agent to evaluate and execute the SQL question.
## Invoke Senior Agent
if agent_invoked == "senior":
print("🧓 >", f"x1b[1;34mReceived instructions: {instructions}x1b[0m")
messages_senior.append( {"role":"user", "content":instructions} )
### use the tools
available_tools = {"sql_check":tool_sql_check, "sql_exec":tool_sql_exec}
context = ''
while available_tools:
agent_res = ollama.chat(model=llm, messages=messages_senior,
tools=[v for v in available_tools.values()])
dic_res = use_tool(agent_res, dic_tools)
res, tool_used, inputs_used = dic_res["res"], dic_res["tool_used"], dic_res["inputs_used"]
if tool_used:
available_tools.pop(tool_used)
context = context + f"nTool used: {tool_used}. Output: {res}" #->add instrument utilization context
messages_senior.append( {"position":"consumer", "content material":context} )
### response
print("🧓 >", f"x1b[1;34m{res}x1b[0m")
messages_senior.append( {"role":"assistant", "content":res} )
The Senior Agent executed the query on the db and got an answer. Finally, it can report back to the Lead which will give the final answer to the user.
### update Lead Agent
context = "Senior agent returned this output: "+res
print("👩💼 >", f"x1b[1;30m{context}x1b[0m")
messages_lead.append( {"role":"user", "content":context} )
Conclusion
This article has covered the basic steps of creating Multi-Agent Systems from scratch using only Ollama. With these building blocks in place, you are already equipped to start developing your own MAS for different use cases.
Stay tuned for Part 4, where we will dive deeper into more advanced examples.
Full code for this article: GitHub
I hope you enjoyed it! Feel free to contact me for questions and feedback or just to share your interesting projects.
👉 Let’s Connect 👈
All images, unless otherwise noted, are by the author
