had been imagined to be learn by a machine. Previous lodge invoices, financial institution statements, payslips, mortgage functions, medical payments, customs types, courtroom filings, work orders.
Most corporations use free instruments alongside paid APIs to attempt to convert these paperwork, and if you’d like structured output, APIs like Textract Structured run you as much as round $65 per 1k pages.
In the previous couple of years, although, a variety of new choices have appeared: smaller open-source imaginative and prescient fashions specialised for OCR, normal vision-language fashions, and doc parsing instruments like LlamaParse — altering what’s potential and the price thereof.
So it felt like a good time to do my very own experiment to check a few of these towards paperwork of various issue.
I scouted 93 docs that might act as a proxy for what corporations use OCR for — handwritten notes, tables, monetary legacy docs, scanned invoices, receipts, charts, previous newspapers, tax types — then ran all of them by 14 totally different engines.
The thought was to see how they dealt with two issues: textual content restoration and the power to protect helpful desk construction.
The principle query I needed answered: do you really want to pay $65 for 1k structured pages, or are you able to slice that all the way down to a fraction? And does the specialised fashions win over the final ones?
When doing experiments like this you at all times discover fairly a number of unusual issues, which I’ll cowl too. However to reply that primary query I’ll take you thru what OCR is (skip if not new), the economics, the take a look at, a few of the outcomes, and what else this confirmed me.
Notice: I didn’t take a look at full area extraction, since that’s tougher to match cleanly throughout fourteen engines.
TL;DR
There isn’t a single greatest OCR engine. OCR is a routing downside.
For clear high-volume paperwork, Tesseract remains to be laborious to beat as a result of it’s free and quick. For combined manufacturing paperwork, Gemini Flash was the very best all-rounder on this take a look at. For tables, Mistral OCR appeared just like the cheaper structured possibility.
The smaller specialist fashions appeared good inside their consolation zone, however failed tougher at paperwork they hadn’t seen. So, for high-stakes or messy paperwork, it is sensible to escalate to a bigger mannequin.
The principle takeaway is financial: don’t pay for costly structured OCR when the doc doesn’t want it. Classify your docs, take a look at engines by yourself information, and route primarily based on price, accuracy, construction, and failure tolerance.
Benchmarks are helpful for discovery, however they won’t inform you what works in your paperwork.
Clarify the OCR area to me
OCR (Optical Character Recognition) is how a machine turns an image into machine readable textual content. Easy in precept, and for simpler docs principally solved, however tougher when issues develop into extra human.
Simply to offer you a fast overview, older OCR discovered textual content on a web page, sliced it into characters, and matched each towards a library of identified shapes. Tesseract has achieved this because the Nineteen Eighties.
Trendy OCR nonetheless (together with newer variations of Tesseract) normally makes use of a neural community that appears on the entire web page without delay and outputs the doc as textual content. So, in case your doc is a clear PDF or a high-quality scan in a normal font, OCR is usually a solved downside.
It stops being solved the second issues get messier: photographed receipts, handwritten notes, bizarre graphs and charts, dense monetary tables, or scanned tax types and mortgage functions.
Firms want this achieved properly for apparent causes, because it’s one thing each downstream system acts on. The higher OCR will get, the extra paperwork turns into one thing a system can motive over as a substitute of one thing a human has to learn by hand.
There may be additionally the truth that if we feed AI methods badly parsed docs, the whole lot after will probably be laborious to belief.
I’m all about economics, so this area caught my eye as soon as I noticed how a lot cash is being poured into it. The Clever Doc Processing (IDP) market is projected to develop to someplace between $20 billion and $90 billion by the early 2030s, relying on which analyst you ask.
Most likely pushed by corporations paying $15–25 per bill in guide dealing with prices.
And since I keep near the tech world, I’ve watched a wave of specialised small OCR fashions ship over the previous yr (principally Chinese language), now being utilized by builders in all places.
Which raises the query I needed to check: can the small open-source fashions truly do the work the costly APIs cost for or ought to we truly look in the direction of the final imaginative and prescient fashions to deal with OCR too?
Skip the subsequent part if you wish to perceive what this experiment confirmed. I’ve to undergo the take a look at setup first.
The docs, the engines, and the metrics
This experiment comes down to 3 questions: what engines we used, what docs we examined with, and the way we determined who gained.
For the engines, I needed a lineup that lined all the alternatives I talked about, this meant: previous and new, open and closed, native and cloud, specialised and normal.
Tesseract grew to become the classical selection. It runs domestically and may be very quick. Then I added two document-parsing pipelines: Docling and Marker. Docling is slower however runs on CPU, Marker is open-weight however needing a GPU to run quick, which exhibits up later within the worth.
Then for the brand new wave of specialised open OCR fashions: GLM-OCR, PaddleOCR-VL, DeepSeek-OCR, and MinerU 2.5 (a borderline case, actually a pipeline with a VLM inside). I picked them off OpenDataLab’s OmniDocBench leaderboard, the place they ranked first, second, fourth, and fifth.
I hosted them on Modal and served the relevant ones with vLLM, batching to hurry issues up. I counted the scale-up time when measuring latency later.
I additionally added one closed purpose-built mannequin, Mistral OCR, which I’d heard good issues about.
On the open aspect, I used Qwen3-VL (8B, from Alibaba), additionally hosted on Modal with the remainder of the smaller fashions. I ought to flag that I gave it a plain transcription immediate slightly than the optimized serving setup it was designed for, so I’ll not have given it a good shot.
On the closed aspect, for the final fashions, I picked Gemini Flash 3.1 Lite (at the moment first on the IDP Leaderboard, the western counterpart constructed on OmniDocBench v1.5) and Claude Sonnet 4.6, at sixth.
For the cloud doc companies: LlamaParse and AWS Textract, in each its textual content and structured types. Structured Textract can do way over I requested of it. I solely examined its textual content accuracy throughout the board and its desk extraction towards eight of the opposite engines.
Let’s flip to the paperwork. I picked seventeen doc sorts that had been both straightforward, medium, or laborious. Ninety-three information in all.
Straightforward was the stuff OCR principally solved years in the past: clear invoices and receipts. Medium got here largely from the OmniAI OCR Benchmark dataset: financial institution statements, medical notes, photographed receipts, delivery paperwork, tax types.
Onerous was chosen when issues turned harder: charts, types, handwritten notes, weirdly scanned monetary tables, authorized papers, newspapers, and previous legacy experiences.
Some docs had been actually fairly tough, such because the legacy scanned docs you see beneath, and this was simply because I used to be curious if some might truly do it properly.
A few of these photos got here with gold floor reality and a few didn’t, and the bottom reality I did have wasn’t at all times constant, some information labeled accurately, some not, which is why we must always briefly cowl the metrics too.
Since each engine emits totally different markup, the standard scoring didn’t fairly match. One may decide Precision and Recall for a case like this.
Precision appears to be like at how most of the OCR output’s phrases truly match within the GT whereas Recall measures what number of occasions every GT phrase was captured.
Precision would punish engines that emit markdown construction the GT doesn’t include, moreover the GT typically skipped labels fully which might punish the engine unfairly. Recall would measure the phrases however punish the frequency.
So, I added on a 3rd metric referred to as Protection. I simply needed to measure how a lot of the bottom reality exhibits up someplace within the engine’s output. It isn’t good, but it surely tells me whether or not an engine caught most of what mattered, with out penalizing it for gaps that had been the bottom reality’s fault slightly than the engine’s.
For the paperwork with no gold floor reality in any respect, I fell again on an LLM decide, with Gemini 3 Professional as the bottom mannequin and anybody who’s used one is aware of that is fickle enterprise.
What this experiment confirmed
We mapped each doc towards the Protection metric to construct a scatter chart, and tracked latency on a separate chart. The factor a generalized chart can’t inform you although is that the engines failed in several methods.
The bubble graph confirmed that almost all engines fall someplace within the center high, with two outliers on either side of it.
Gemini Flash and Textract Textual content did very properly throughout the board with some edge instances. The specialised fashions all fell beneath the final fashions and specialised APIs. Sonnet carried out the best but additionally with a steeper price ticket.
This will likely not have been a shock because the take a look at set was extremely uncommon. A number of the specialised fashions could not have seen lots of them. Moreover, this take a look at was on English paperwork and most of those smaller fashions have Chinese language origin.
Once we additionally mapped latency, a few of the fashions turned out to be very gradual, however once more most wound up someplace within the center.
The outliers right here had been: Tesseract, Claude Sonnet 4.6, and Docling. Tesseract was extremely quick in comparison with all different engines. It needs to be your go-to for simpler paperwork.
These graphs generalise throughout all of the paperwork, however I did separate the outcomes primarily based on the sort and issue stage.
To begin with the simple docs. On invoices, each engine did properly, Tesseract particularly. Receipts knocked everybody down slightly.
The one outlier was Docling, which struggled throughout a variety of the classes, even the simple ones.
After I appeared into the Docling failures I discovered issues like Ifjointreturn as a substitute of “joint return,” and worse, strings like Metropolis,wrostffielfouaveaoreignadresalcomletacesb. DeepSeek additionally missed key particulars right here like bill quantity and date, which is why its quantity sits low.
The identical sample holds within the medium class, although that’s the place PaddleOCR began degrading on particular sorts: financial institution statements, delivery, tax types. Tax types had been laborious for everybody, however PaddleOCR and Docling wound up on the backside.
Textract was the very best engine on a variety of the medium sorts, together with Claude Sonnet 4.6 and Mistral OCR.
On the tougher sorts, Gemini Flash began rising, beating Textract on types and handwritten notes, matching it elsewhere. It did remarkably properly in all places. Tesseract and Docling failed laborious on handwritten, and types had been robust for them too.
Nearly all of the specialised fashions didn’t pull by on these tougher docs besides on monetary tables, the place they held about even.
For the docs with no floor reality (newspapers, authorized, experiences, some scanned legacy paperwork) we used an LLM decide. These are genuinely laborious, so it’s no shock virtually everybody failed on the experiences and newspapers.
Besides Gemini Flash that did moderately properly in all places. Mistral OCR additionally did properly for newspapers. Gemini Flash gained in all places with the decide, although we used Gemini Professional because the decide so take that with a grain of salt (however I did double examine myself).
Earlier than rounding off: I additionally ran 8 engines towards Textract Structured to see how they did on monetary tables, extracting an HTML desk. I used Textract Structured’s output as the bottom reality for TEDS (Tree Edit Distance Similarity) and scored Claude Sonnet 4.6, LlamaParse, Mistral OCR, Gemini Flash, Marker, MinerU, DeepSeek-OCR, and Docling towards it.
Mistral OCR, and LlamaParse, and Sonnet did very properly whereas being less expensive. I additionally ran it by an LLM decide, and the winners had been the identical three (even earlier than Textract Structured) although I’d need to construct that take a look at higher earlier than I totally belief it.
Now, let’s speak about what it prices to scale this up, and what would make sense the place.
When does what make sense
Let’s run by what it prices to scale up with these engines, after which primarily based on these docs what you’d selected the place.
First, the price of utilizing these engines differ wildly, as you noticed earlier than. Generally it helps to see the price not only for one doc, however hundreds as much as one million.
We’re self-hosting on Modal, so these prices come from precise utilization there. You’ll be able to run domestically, however my pc wouldn’t enable it and I didn’t need to attempt it.
If you happen to had been to only use one engine that handles each straightforward and laborious paperwork, I’d suppose you wound up with a much bigger invoice than needed. Utilizing Textract Structured for any paperwork that aren’t wanted would hand you a invoice of $6.5k per 100k docs.
I do surprise what number of corporations go the simple method right here and decide the costly choices for simple in addition to laborious docs and go away some huge cash on the desk.
The important thing thought to take with you right here is that there’s no single greatest engine for each use case, it will depend on doc sort, privateness, desk construction, failure tolerance, price, and so forth.
For the docs we’ve got right here, Gemini Flash 3.1-Lite is a transparent winner. This one was appropriate from trying on the leaderboards. Mistral OCR did properly on structured tables whereas staying low-cost. Claude Sonnet 4.6 did very properly too, but it surely’s very gradual and costly comparatively.
Docling is so very gradual on my laptop computer. I’m certain there are methods to hurry it up, but it surely additionally failed in ways in which make it inherently unstable (nonetheless a small take a look at although).
The specialised OCR fashions had been a little bit of a headache, particularly on English docs; I noticed output errors in Chinese language that I’ll cowl in a bit, so I ponder if that’s a part of it.
Textract is a secure selection, however structured buys you virtually no extra textual content accuracy so in case you’re paying that steep markup for structured output, be sure you truly use it. I’m guessing it’s a reasonably good enterprise mannequin for them.
So, usually for this very small take a look at: for clear, high-volume print, simply use Tesseract. For normal heterogeneous manufacturing, go Gemini Flash. For a cost-floor with desk construction, take a look at Mistral OCR. For prime-stakes docs, path to Sonnet or a bigger mannequin.
Since everybody did properly in several methods you’ll must contact me for specifics but when it is advisable to go personal it could be value it to take a look at fine-tuning a mannequin in your docs. Or, use a small specialised mannequin and escalate on failures.
Let me simply shortly speak about some issues that stood out after doing this experiment.
Different stuff I ought to point out
A handful of issues surfaced from this which are value pulling out on their very own.
First, if you wish to perceive how a mannequin or engine will do in your docs, the one method is to take a look at on these docs, you possibly can’t depend on benchmarks to inform you. This was the primary perception this confirmed. OCR usefulness relies upon by yourself doc combine, layouts, languages, scans, tables, handwriting, and failure tolerance.
Don’t pay for construction in case you don’t want it. I ponder what number of are utilizing sure APIs or fashions for a motive they will’t justify. Map the price to know what you might be dropping by not utilizing the proper engine for the paperwork.
The specialist fashions, as talked about earlier than, have sharp boundaries. That is apparent, they are often wonderful inside their coaching distribution however fail outdoors of it. That is the place the final fashions will win.
If you wish to fine-tune it could assist, however provided that the stream is secure as it can additionally fail whether it is consistently launched new doc courses.
Lastly, the failure modes instructed us greater than the averages.
PaddleOCR had repetition loops, column-merging, fallback into Chinese language textbook template textual content like 书名:___ repeated lots of of occasions. Whereas Docling has character errors, word-merging, and column misalignment all stacking collectively.
DeepSeek OCR has chart blindness and empty outputs on some docs. Tesseract did wonderful on clear docs (as talked about) however failed on picture/handwriting altogether outputting rubbish.
Caveats to think about
Earlier than we spherical up, let me cowl how this take a look at is in the end imperfect by naming the problems within the GT, the metrics used, and the pattern dimension.
I lined this in one of many part above, however the floor reality differs between paperwork relying on the dataset the place they had been discovered. Typically tokenization artifacts could make appropriate OCR look worse than it’s.
Most engines have totally different codecs, some return plain textual content, some markdown, some HTML/wealthy markdown and it’s laborious to generalize throughout all.
We’re utilizing Protection, after which additionally another metrics, however these aren’t good. Protection gained’t cost the engine if it outputs an excessive amount of textual content or the construction of it’s off. Although I did discover that for the engines that failed, they did so at first or mid-way by slightly than on the finish.
This implies it’s helpful for rating however not an ideal strategy to rating.
LLM judges aren’t impartial reality: I’ve lined this up to now, however they’re biased and really immediate delicate.
Then I simply must say that this take a look at is attention-grabbing however not that huge, the pattern dimension is method too small to make use of this as a factual research. However, I don’t totally belief these metrics nor the decide so it was the one method for me to have the ability to double examine the outcomes by myself with out this turning right into a yr lengthy undertaking.
So, this take a look at is helpful for course and getting a way of what works, however for getting a way of your use case, it is advisable to run it by along with your particular docs.
Lastly, latency and reproducibility is unstable. Serverless chilly begins make timing noisy, and API fashions can silently change over time, so actual copy is difficult.
Like at all times with these articles, it takes fairly a bit to do an experiment like this however I don’t simply do it for content material, I do it as a result of I’m genuinely curious.
What it appears to be like like although is that OCR appears to be a routing downside, and maybe an analysis downside. Classify your docs and run them by a number of engines, then attempt to construct a good router and validator in your pipeline to escalate failures after which log the prices.
If it is advisable to get the complete outcomes from this experiment otherwise you need me to run it by your docs, get in contact.
You’ll be able to comply with my writing on Medium, my web site or join with me through LinkedIn.
❤
All datasets used on this benchmark are publicly obtainable and sourced from HuggingFace. Licenses embrace MIT, CC-BY-4.0, and fair-use frameworks (UCSF Trade Paperwork Library) protecting analysis, scholarship, and schooling. No supply paperwork are reproduced — datasets had been used solely as analysis inputs to measure OCR engine efficiency.
