Tables, Photographs, figures or equations will not be drawback anymore! Full Code offered.
Automated doc processing is likely one of the greatest winners of the ChatGPT revolution, as LLMs are capable of deal with a variety of topics and duties in a zero-shot setting, that means with out in-domain labeled coaching knowledge. This has made constructing AI-powered purposes to course of, parse, and mechanically perceive arbitrary paperwork a lot simpler. Although naive approaches utilizing LLMs are nonetheless hindered by non-text context, reminiscent of figures, photos, and tables, that is what we’ll attempt to deal with on this weblog submit, with a particular concentrate on PDFs.
At a fundamental stage, PDFs are only a assortment of characters, photos, and features together with their actual coordinates. They haven’t any inherent “textual content” construction and weren’t constructed to be processed as textual content however solely to be considered as is. That is what makes working with them troublesome, as text-only approaches fail to seize all of the structure and visible parts in most of these paperwork, leading to a big lack of context and knowledge.
One solution to bypass this “text-only” limitation is to do heavy pre-processing of the doc by detecting tables, photos, and structure earlier than feeding them to the LLM. Tables may be parsed to Markdown or JSON, photos and figures may be represented by their captions, and the textual content may be fed as is. Nonetheless, this strategy requires customized fashions and can nonetheless lead to some lack of info, so can we do higher?
Most up-to-date giant fashions at the moment are multi-modal, that means they will course of a number of modalities like textual content, code, and pictures. This opens the way in which to an easier answer to our drawback the place one mannequin does the whole lot directly. So, as an alternative of captioning photos and parsing tables, we are able to simply feed the web page as a picture and course of it as is. Our pipeline will be capable to load the PDF, extract every web page as a picture, break up it into chunks (utilizing the LLM), and index every chunk. If a bit is retrieved, then the total web page is included within the LLM context to carry out the duty. In what follows, we’ll element how this may be applied in apply.
The pipeline we’re implementing is a two-step course of. First, we section every web page into important chunks and summarize every of them. Second, we index chunks as soon as then search the chunks every time we get a request and embrace the total context with every retrieved chunk within the LLM context.
Step 1: Web page Segmentation and Summarization
We extract the pages as photos and move every of them to the multi-modal LLM to section them. Fashions like Gemini can perceive and course of web page structure simply:
- Tables are recognized as one chunk.
- Figures type one other chunk.
- Textual content blocks are segmented into particular person chunks.
- …
For every ingredient, the LLM generates a abstract than may be embedded and listed right into a vector database.
Step 2: Embedding and Contextual Retrieval
On this tutorial we’ll use textual content embedding just for simplicity however one enchancment could be to make use of imaginative and prescient embeddings immediately.
Every entry within the database consists of:
- The abstract of the chunk.
- The web page quantity the place it was discovered.
- A hyperlink to the picture illustration of the total web page for added context.
This schema permits for native stage searches (on the chunk stage) whereas protecting monitor of the context (by linking again to the total web page). For instance, if a search question retrieves an merchandise, the Agent can embrace all the web page picture to supply full structure and further context to the LLM with a purpose to maximize response high quality.
By offering the total picture, all of the visible cues and essential structure info (like photos, titles, bullet factors… ) and neighboring gadgets (tables, paragraph, …) can be found to the LLM on the time of producing a response.
We are going to implement every step as a separate, re-usable agent:
The primary agent is for parsing, chunking, and summarization. This includes the segmentation of the doc into important chunks, adopted by the era of summaries for every of them. This agent solely must be run as soon as per PDF to preprocess the doc.
The second agent manages indexing, search, and retrieval. This consists of inserting the embedding of chunks into the vector database for environment friendly search. Indexing is carried out as soon as per doc, whereas searches may be repeated as many instances as wanted for various queries.
For each brokers, we use Gemini, a multimodal LLM with robust imaginative and prescient understanding skills.
Parsing and Chunking Agent
The primary agent is in command of segmenting every web page into significant chunks and summarizing every of them, following these steps:
Step 1: Extracting PDF Pages as Photographs
We use the pdf2image library. The pictures are then encoded in Base64 format to simplify including them to the LLM request.
Right here’s the implementation:
from document_ai_agents.document_utils import extract_images_from_pdf
from document_ai_agents.image_utils import pil_image_to_base64_jpeg
from pathlib import Pathclass DocumentParsingAgent:
@classmethod
def get_images(cls, state):
"""
Extract pages of a PDF as Base64-encoded JPEG photos.
"""
assert Path(state.document_path).is_file(), "File doesn't exist"
# Extract photos from PDF
photos = extract_images_from_pdf(state.document_path)
assert photos, "No photos extracted"
# Convert photos to Base64-encoded JPEG
pages_as_base64_jpeg_images = [pil_image_to_base64_jpeg(x) for x in images]
return {"pages_as_base64_jpeg_images": pages_as_base64_jpeg_images}
extract_images_from_pdf: Extracts every web page of the PDF as a PIL picture.
pil_image_to_base64_jpeg: Converts the picture right into a Base64-encoded JPEG format.
Step 2: Chunking and Summarization
Every picture is then despatched to the LLM for segmentation and summarization. We use structured outputs to make sure we get the predictions within the format we count on:
from pydantic import BaseModel, Subject
from typing import Literal
import json
import google.generativeai as genai
from langchain_core.paperwork import Docclass DetectedLayoutItem(BaseModel):
"""
Schema for every detected structure ingredient on a web page.
"""
element_type: Literal["Table", "Figure", "Image", "Text-block"] = Subject(
...,
description="Kind of detected merchandise. Examples: Desk, Determine, Picture, Textual content-block."
)
abstract: str = Subject(..., description="An in depth description of the structure merchandise.")
class LayoutElements(BaseModel):
"""
Schema for the checklist of structure parts on a web page.
"""
layout_items: checklist[DetectedLayoutItem] = []
class FindLayoutItemsInput(BaseModel):
"""
Enter schema for processing a single web page.
"""
document_path: str
base64_jpeg: str
page_number: int
class DocumentParsingAgent:
def __init__(self, model_name="gemini-1.5-flash-002"):
"""
Initialize the LLM with the suitable schema.
"""
layout_elements_schema = prepare_schema_for_gemini(LayoutElements)
self.model_name = model_name
self.mannequin = genai.GenerativeModel(
self.model_name,
generation_config={
"response_mime_type": "utility/json",
"response_schema": layout_elements_schema,
},
)
def find_layout_items(self, state: FindLayoutItemsInput):
"""
Ship a web page picture to the LLM for segmentation and summarization.
"""
messages = [
f"Find and summarize all the relevant layout elements in this PDF page in the following format: "
f"{LayoutElements.schema_json()}. "
f"Tables should have at least two columns and at least two rows. "
f"The coordinates should overlap with each layout item.",
{"mime_type": "image/jpeg", "data": state.base64_jpeg},
]
# Ship the immediate to the LLM
end result = self.mannequin.generate_content(messages)
knowledge = json.masses(end result.textual content)
# Convert the JSON output into paperwork
paperwork = [
Document(
page_content=item["summary"],
metadata={
"page_number": state.page_number,
"element_type": merchandise["element_type"],
"document_path": state.document_path,
},
)
for merchandise in knowledge["layout_items"]
]
return {"paperwork": paperwork}
The LayoutElements schema defines the construction of the output, with every structure merchandise sort (Desk, Determine, … ) and its abstract.
Step 3: Parallel Processing of Pages
Pages are processed in parallel for pace. The next technique creates a listing of duties to deal with all of the web page picture directly for the reason that processing is io-bound:
from langgraph.sorts import Shipclass DocumentParsingAgent:
@classmethod
def continue_to_find_layout_items(cls, state):
"""
Generate duties to course of every web page in parallel.
"""
return [
Send(
"find_layout_items",
FindLayoutItemsInput(
base64_jpeg=base64_jpeg,
page_number=i,
document_path=state.document_path,
),
)
for i, base64_jpeg in enumerate(state.pages_as_base64_jpeg_images)
]
Every web page is distributed to the find_layout_items operate as an unbiased job.
Full workflow
The agent’s workflow is constructed utilizing a StateGraph, linking the picture extraction and structure detection steps right into a unified pipeline ->
from langgraph.graph import StateGraph, START, ENDclass DocumentParsingAgent:
def build_agent(self):
"""
Construct the agent workflow utilizing a state graph.
"""
builder = StateGraph(DocumentLayoutParsingState)
# Add nodes for picture extraction and structure merchandise detection
builder.add_node("get_images", self.get_images)
builder.add_node("find_layout_items", self.find_layout_items)
# Outline the movement of the graph
builder.add_edge(START, "get_images")
builder.add_conditional_edges("get_images", self.continue_to_find_layout_items)
builder.add_edge("find_layout_items", END)
self.graph = builder.compile()
To run the agent on a pattern PDF we do:
if __name__ == "__main__":
_state = DocumentLayoutParsingState(
document_path="path/to/doc.pdf"
)
agent = DocumentParsingAgent()# Step 1: Extract photos from PDF
result_images = agent.get_images(_state)
_state.pages_as_base64_jpeg_images = result_images["pages_as_base64_jpeg_images"]
# Step 2: Course of the primary web page (for instance)
result_layout = agent.find_layout_items(
FindLayoutItemsInput(
base64_jpeg=_state.pages_as_base64_jpeg_images[0],
page_number=0,
document_path=_state.document_path,
)
)
# Show the outcomes
for merchandise in result_layout["documents"]:
print(merchandise.page_content)
print(merchandise.metadata["element_type"])
This ends in a parsed, segmented, and summarized illustration of the PDF, which is the enter of the second agent we’ll construct subsequent.
RAG Agent
This second agent handles the indexing and retrieval half. It saves the paperwork of the earlier agent right into a vector database and makes use of the end result for retrieval. This may be break up into two seprate steps, indexing and retrieval.
Step 1: Indexing the Cut up Doc
Utilizing the summaries generated, we vectorize them and save them in a ChromaDB database:
class DocumentRAGAgent:
def index_documents(self, state: DocumentRAGState):
"""
Index the parsed paperwork into the vector retailer.
"""
assert state.paperwork, "Paperwork ought to have a minimum of one ingredient"
# Test if the doc is already listed
if self.vector_store.get(the place={"document_path": state.document_path})["ids"]:
logger.data(
"Paperwork for this file are already listed, exiting this node"
)
return # Skip indexing if already executed
# Add parsed paperwork to the vector retailer
self.vector_store.add_documents(state.paperwork)
logger.data(f"Listed {len(state.paperwork)} paperwork for {state.document_path}")
The index_documents technique embeds the chunk summaries into the vector retailer. We hold metadata such because the doc path and web page quantity for later use.
Step 2: Dealing with Questions
When a person asks a query, the agent searches for essentially the most related chunks within the vector retailer. It retrieves the summaries and corresponding web page photos for contextual understanding.
class DocumentRAGAgent:
def answer_question(self, state: DocumentRAGState):
"""
Retrieve related chunks and generate a response to the person's query.
"""
# Retrieve the top-k related paperwork based mostly on the question
relevant_documents: checklist[Document] = self.retriever.invoke(state.query)# Retrieve corresponding web page photos (keep away from duplicates)
photos = checklist(
set(
[
state.pages_as_base64_jpeg_images[doc.metadata["page_number"]]
for doc in relevant_documents
]
)
)
logger.data(f"Responding to query: {state.query}")
# Assemble the immediate: Mix photos, related summaries, and the query
messages = (
[{"mime_type": "image/jpeg", "data": base64_jpeg} for base64_jpeg in images]
+ [doc.page_content for doc in relevant_documents]
+ [
f"Answer this question using the context images and text elements only: {state.question}",
]
)
# Generate the response utilizing the LLM
response = self.mannequin.generate_content(messages)
return {"response": response.textual content, "relevant_documents": relevant_documents}
The retriever queries the vector retailer to search out the chunks most related to the person’s query. We then construct the context for the LLM (Gemini), which mixes textual content chunks and pictures with a purpose to generate a response.
The complete agent Workflow
The agent workflow has two levels, an indexing stage and a query answering stage:
class DocumentRAGAgent:
def build_agent(self):
"""
Construct the RAG agent workflow.
"""
builder = StateGraph(DocumentRAGState)
# Add nodes for indexing and answering questions
builder.add_node("index_documents", self.index_documents)
builder.add_node("answer_question", self.answer_question)
# Outline the workflow
builder.add_edge(START, "index_documents")
builder.add_edge("index_documents", "answer_question")
builder.add_edge("answer_question", END)
self.graph = builder.compile()
Instance run
if __name__ == "__main__":
from pathlib import Path# Import the primary agent to parse the doc
from document_ai_agents.document_parsing_agent import (
DocumentLayoutParsingState,
DocumentParsingAgent,
)
# Step 1: Parse the doc utilizing the primary agent
state1 = DocumentLayoutParsingState(
document_path=str(Path(__file__).mother and father[1] / "knowledge" / "docs.pdf")
)
agent1 = DocumentParsingAgent()
result1 = agent1.graph.invoke(state1)
# Step 2: Arrange the second agent for retrieval and answering
state2 = DocumentRAGState(
query="Who was acknowledged on this paper?",
document_path=str(Path(__file__).mother and father[1] / "knowledge" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
agent2 = DocumentRAGAgent()
# Index the paperwork
agent2.graph.invoke(state2)
# Reply the primary query
result2 = agent2.graph.invoke(state2)
print(result2["response"])
# Reply a second query
state3 = DocumentRAGState(
query="What's the macro common when fine-tuning on PubLayNet utilizing M-RCNN?",
document_path=str(Path(__file__).mother and father[1] / "knowledge" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
result3 = agent2.graph.invoke(state3)
print(result3["response"])
With this implementation, the pipeline is full for doc processing, retrieval, and query answering.
Let’s stroll by way of a sensible instance utilizing the doc LLM & Adaptation.pdf , a set of 39 slides containing textual content, equations, and figures (CC BY 4.0).
Step 1: Parsing and summarizing the Doc (Agent 1)
- Execution Time: Parsing the 39-page doc took 29 seconds.
- Outcome: Agent 1 produces an listed doc consisting of chunk summaries and base64-encoded JPEG photos of every web page.
Step 2: Questioning the Doc (Agent 2)
We ask the next query:
“Clarify LoRA, give the related equations”
Outcome:
Retrieved pages:
