or vision-language fashions is a strong approach that unlocks their potential on specialised duties. Nonetheless, regardless of their effectiveness, these approaches are sometimes out of attain for a lot of customers on account of their excessive computational value and the necessity for GPUs with massive VRAM — sources that solely a small proportion of finish customers can entry.
On this undertaking, I fine-tuned IBM’s Granite-Imaginative and prescient 2B, a comparatively small but highly effective vision-language mannequin, to sort out the problem of changing photographs of tables into clear, structured HTML code.
What makes this undertaking significantly thrilling is that the fine-tuning was carried out on a consumer-grade GPU — the NVIDIA RTX 4070 Ti Tremendous — and but, the ensuing 2-billion-parameter mannequin was capable of outperform a lot bigger fashions, together with meta-llama/Llama-3.2–90B-Imaginative and prescient, on this image-to-text technology job. This success not solely demonstrates the ability of parameter-efficient fine-tuning strategies like LoRA but additionally highlights the sensible worth of constructing specialised small fashions tailor-made to particular issues.
On this put up, I’ll stroll you thru the motivation behind this work, the mannequin and dataset selections, the customized HTML similarity metric I tailored, the experiments and outcomes, and at last, the important thing insights and classes realized all through the method. Whether or not you’re concerned about vision-language fashions, fine-tuning methods, or sensible AI functions, I hope this journey provides helpful takeaways. The fine-tuning code used for this undertaking was tailored from HuggingFace’s Granite Imaginative and prescient fine-tuning cookbook, authored by Eli Schwartz, who in flip tailored the unique code from Sergio Paniego.
Motivation
Whereas engaged on Retrieval-Augmented Era (RAG) initiatives, I encountered a significant problem: precisely extracting massive and sophisticated tables from PDFs, particularly when these tables appeared as photographs. Regardless of attempting completely different approaches — together with instruments like Unstructured and enormous vision-language fashions equivalent to Meta’s Llama 90B — the outcomes usually fell in need of the accuracy wanted.
This led me to think about a special strategy: a small, specialised vision-language mannequin targeted completely on desk understanding and extraction. Such a mannequin might function a devoted preprocessing step to considerably enhance RAG pipelines that depend on correct desk extraction.
Across the similar time, IBM launched Granite-Imaginative and prescient 2B — a vision-language mannequin with simply the best steadiness of measurement and energy. It’s succesful sufficient to deal with complicated tables, but sufficiently small to be fine-tuned on consumer-grade GPUs with 16 GB of VRAM. This made it a really perfect candidate for my undertaking.
The Process: Picture to HTML (Desk Extraction)
One vital design selection was the goal format: HTML. By changing tables into clear HTML code, we acquire a structured and broadly supported illustration that may be simply transformed into different codecs. For instance, HTML tables will be readily imported into knowledge evaluation instruments like Pandas as dataframes, making downstream processing and evaluation far more environment friendly.
The unique plan was to construct a customized dataset by extracting HTML desk tags, rendering them as photographs, and pairing every picture with its corresponding HTML code. Thankfully, I discovered an answer: the PubTabNet-HTML dataset, which incorporates over 568,000 picture–HTML pairs, excess of wanted for this undertaking.
PubTabNet was developed by IBM and relies on scientific articles from the PubMed Central Open Entry Subset (business use assortment). The tables had been extracted by aligning PDF and XML variations of the articles. The annotations (i.e., the HTML labels) are licensed beneath the Group Knowledge License Settlement – Permissive – Model 1.0, and whereas IBM doesn’t personal the pictures, they’re utilized in accordance with the PMC Open Entry Subset Phrases of Use. This makes the dataset appropriate for each analysis and business functions, supplied the license phrases are adopted.
Customized Metric: HTML Similarity
Normal textual content similarity metrics like BLEU or ROUGE are inadequate for evaluating HTML desk technology as a result of they primarily give attention to surface-level textual content matching and ignore vital structural and stylistic facets of HTML code.
To raised seize the standard of generated HTML tables, I tailored a customized HTML Similarity metric that mixes a number of complementary parts, the place crucial ones (model and construction) are imported from niteru:
Model similarity (S): Extracts CSS courses of every html doc and calculates the jaccard similarity of the units of courses.
Structural similarity (T): Makes use of sequence comparability of the html tags to compute the similarity.
Content material similarity (C): Primarily based on normalized edit distance between the extracted plain textual content content material of the tables.
Token overlap similarity (J): The Jaccard similarity between the units of content material tokens.
The ultimate similarity rating M is a weighted sum of those parts:
I manually examined the metric on varied instance outputs, iteratively adjusting the weighting coefficients to raised seize significant similarities. This course of resulted in a balanced analysis that pretty rewards correct desk construction and magnificence, alongside exact textual content material. Python implementation is as follows:
from torchmetrics.textual content import EditDistance
from niteru import style_similarity, structural_similarity
ed_distance = EditDistance()
def extract_table_text(html):
"""Extracts solely the textual content from an HTML desk in row-wise space-separated format."""
soup = BeautifulSoup(html, "html.parser")
desk = soup.discover("desk") # Discover the primary desk
if not desk:
return ""
# Extract rows and be part of cells with areas
return "n".be part of(" ".be part of(cell.get_text(strip=True) for cell in row.find_all(["th", "td"])) for row in desk.find_all("tr"))
def extract_html_table(html):
"""Extracts html desk from textual content"""
match = re.search(r'
The metric additionally features a regex-based perform to extract solely the HTML content material inside
tags. This was obligatory as a result of one of many reference fashions solely generated incomplete or additional HTML exterior of the desk construction. By focusing the comparability strictly on the desk content material, the metric gives a extra truthful and significant analysis throughout fashions.
Creating a customized analysis metric like that is essential for reliably monitoring mannequin enhancements and benchmarking efficiency in opposition to reference fashions.
Coaching Setup
To fine-tune the mannequin effectively on my NVIDIA RTX 4070 Ti Tremendous, which has 16 GB VRAM, I used LoRA (Low-Rank Adaptation). This allowed me to replace solely a small variety of parameters, considerably decreasing GPU reminiscence utilization. In reality, throughout coaching, the mannequin used solely about half of the out there VRAM — with sufficient headroom to mess around with longer sequences, however not sufficient to deal with a couple of batch. Moreover, LoRA is usually quicker to coach than approaches like QLoRA.
LoRA Setup
I used the next LoRA configuration:
# Setup LoRA
target_modules = []
for layer_type in layers_to_tune:
target_modules.lengthen(
identify for identify, _ in mannequin.named_modules()
if (layer_type in identify)
and '_proj' in identify
)
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.1,
target_modules=target_modules,
use_dora=True,
init_lora_weights="gaussian"
)
Key factors:
r=16: This low-rank dimension gives a superb steadiness between mannequin capability and GPU reminiscence utilization.
use_dora=True: DoRA (Weight-Decomposed Low Rank Adaptation) improves the educational capability and stability of LoRA by decomposing the pretrained weights into magnitude and path parts, serving to the mannequin higher resemble the capability of full fine-tuning — all with out including inference overhead. Carried out barely higher than the default setting.
init_lora_weights="gaussian": No specific motive, I didn’t need to experiment with this parameter.
target_modules: This versatile setup permits selectively concentrating on imaginative and prescient layers, language layers, or each, relying on the experiment. In apply, imaginative and prescient layers remained unaffected — even with use_dora=False— since DoRA at the moment helps solely embedding, linear, and Conv2d layers. Consequently, I fine-tuned solely the language layers.
Dataset Setup
Throughout my preliminary experiments, I stored working into out-of-memory (OOM) errors — despite the fact that there was nonetheless loads of out there GPU VRAM after loading mannequin, LoRA layers and optimizer parameters (round 4GB nonetheless free). There have been no reminiscence spikes throughout coaching, however the crashes persistently occurred on the similar coaching step.
After some investigation, I spotted that the issue was brought on by massive tables, which resulted in extraordinarily lengthy token sequences. To deal with this, I adjusted the max_seq_length parameter and filtered out samples that exceeded this restrict. After experimentation, I discovered that utilizing max_seq_length = 1024 allowed me to fine-tune the mannequin reliably with out triggering OOM errors.
To filter out outsized tables, I wrote a easy knowledge processing perform that:
Filters out samples whose HTML token size exceeds max_seq_length
Mechanically balances the variety of coaching and take a look at samples
Makes use of streaming to keep away from loading your complete dataset into reminiscence (PubTabNet-HTML is sort of massive, round 10 GB on disk)
.
def load_process_filter_dataset(dataset, max_seq_length, num_train_images, num_test_images, system_message):
world processor
ds = load_dataset(dataset, break up='practice', streaming=True)
max_html_tokens = max_seq_length - len(processor.tokenizer.tokenize(system_message))
num_total_needed = num_train_images + num_test_images
filtered_samples = []
p_bar = tqdm(whole=num_total_needed, desc="Filtering dataset samples")
for pattern in ds:
processed = process_and_filter_example(pattern, max_html_tokens)
if processed:
filtered_samples.append(processed)
p_bar.replace(1)
if len(filtered_samples) >= num_total_needed:
break
p_bar.shut()
# Convert to in-memory dataset
ds_filtered = Dataset.from_list(filtered_samples)
# Cut up into practice/take a look at
ds_train = ds_filtered.choose(vary(num_train_images))
ds_test = ds_filtered.choose(vary(num_train_images, num_total_needed))
return ds_train, ds_test
def process_and_filter_example(instance, max_html_tokens):
world processor
extracted_table = extract_html_table(instance['html_table'])
token_count = len(processor.tokenizer.tokenize(extracted_table))
if token_count < max_html_tokens:
instance['html_table'] = extracted_table
return instance
return None
The ultimate configuration included num_train_images=10000 and num_test_images=250 to compute the analysis loss.
High quality-Tuning Configuration
For coaching, I used the Transformers SFTTrainer to fine-tune the mannequin:
num_train_epochs=1: The dataset could be very massive, and to run a number of experiments effectively, I selected to coach for just one full epoch whereas maximizing studying per pattern and variety of coaching samples.
per_device_train_batch_size=1: Bigger batch sizes wouldn't slot in GPU reminiscence with out considerably decreasing max_seq_length — which might harm efficiency on massive tables. Retaining longer sequences was extra vital for this job.
gradient_accumulation_steps=8: Used to successfully simulate a bigger batch measurement and assist stabilize the educational course of, compensating for the small bodily batch. That is the ultimate worth, however experimented with gradient_accumulation_steps=4 as properly.
optim="adamw_torch_fused" and bf16=True: These settings leverage fashionable NVIDIA architectures (Ada Lovelace) to speed up coaching and scale back reminiscence utilization — as advisable for this {hardware}.
Analysis Loss Workaround
On the time of creating the undertaking, there's a identified difficulty within the Transformers + LoRA integration that causes an error when working analysis with a validation dataset throughout coaching. Thankfully, a community-tested workaround is obtainable (though not but merged into the primary department), and I efficiently used this repair in my experiments.
Analysis (Inference) Setup
The analysis dataset used for ultimate scoring was utterly unbiased from the eval_dataset used throughout coaching. It consists of 500 randomly chosen photographs, none of which had been included in both the train_dataset or the coaching eval_dataset.
As soon as fine-tuning was full, I used the finest mannequin checkpoint — chosen based mostly on the bottom analysis loss — to run inference on these 500 samples.
Initially, I tried to carry out inference by merely loading the LoRA/DoRA adapter on high of the bottom mannequin. Nonetheless, I discovered that inference with DoRA adapters is extraordinarily sluggish when not merged into the mannequin weights (as defined within the official PEFT docs). In reality, producing one take a look at random pattern took about 90 seconds on this configuration.
To resolve this, I merged the adapter weights into the bottom mannequin — which is the advisable apply — and after merging, inference velocity improved dramatically: right down to ~20 seconds for a similar pattern, making full analysis runs far more sensible.
The reference fashions used for comparability with my fine-tuned fashions are:
meta-llama/Llama-3.2–90B-Imaginative and prescient: Meta’s huge 90-billion parametermannequin — the primary baseline I aimed to surpass via specialization and parameter-efficient fine-tuning of a a lot smaller VLM.
KennethTM/pix2struct-base-table2html: A a lot smaller mannequin fine-tuned from Google’s pix2struct-base, extremely specialised for precisely the identical dataset I used on this undertaking. Due to its smaller measurement, the developer(s) was capable of practice it for a lot of extra samples and over longer coaching runs — demonstrating the important thing benefit of utilizing smaller, focused fashions for particular duties.
These two baselines allowed me to benchmark each scaling-based efficiency (vs the 90B mannequin) and specialization effectivity (vs the smaller, devoted Pix2Struct mannequin).
Experiments & Outcomes
A complete of 9 experiments had been performed, iteratively modifying one or two parts at a time. The objective was to know the impact of every change on mannequin efficiency, step by step refining the setup to realize the absolute best HTML Similarity rating in comparison with reference fashions.
The experimental course of was incremental: every time a change improved the outcomes, it was included into the following spherical of experiments and continued exploring new variations.
The experiments targeted on adjusting the next parts:
Imaginative and prescient vs. Language Layers
1.1 lang_only
1.2 vision_only
1.3 lang_vision
2. Floor Fact Output Format
3. Coaching Framework
3.1 lang_table_unsloth
3.2 vision_table_unsloth
4. Gradient Accumulation
5. Immediate Format
6. Gradient Accumulation & Dataset Measurement
Each the analysis loss and the HTML Similarity metric had been used to evaluate mannequin efficiency, and I discovered them to be properly correlated — confirming that HTML Similarity is an effective proxy for the way properly the mannequin is studying the duty.
Earlier than diving into the outcomes of every experiment, let’s first take a look at GPU reminiscence utilization throughout coaching, which is commonly probably the most essential consider figuring out whether or not a mannequin will be fine-tuned on shopper {hardware}.
GPU Reminiscence Utilization Throughout Coaching | Picture by writer from wandb.ai
As proven within the graph, GPU utilization remained steady all through coaching — averaging round 75% VRAM utilization, or roughly 12 GB on my GPU. Most of VRAM utilization (~5.5 GB) is the frozen mannequin weights. LoRA gradients + optimizer states take little or no (<< 1 GB). Activations + overhead ought to fill the remainder (~5–6 GB), which is determined by batch_size and max_seq_length.
First Run: lang_only
This experiment makes use of the next preliminary parts/parameters:
These had been the beginning values for the primary experiment. In subsequent runs, I modified a lot of them as I refined the strategy. This primary experiment targeted solely on tuning language layers, whereas coaching the mannequin to foretell the total uncooked HTML output — together with every little thing inside and across the
tags.
Since this was the primary run, I’ll embody the coaching loss curve right here for instance the way it behaves. For later experiments, I’ll omit this graph — because the conduct was related throughout runs, with minor variations. In apply, analysis loss is extra helpful for evaluating efficiency throughout experiments.
Coaching Loss | Picture by writer from wandb.ai
One vital notice concerning the logging configuration: logging_steps=25 signifies that the coaching loss is simply logged after each 25 steps, the place every logged worth is the typical over gradient_accumulation_steps=4. Consequently, the biggest drop in loss seems on the second log level — the place many of the preliminary studying occurs. After that, the mannequin continues studying extra step by step, with a sluggish lowering development, relying on the problem of the coaching samples.
Now, let’s check out the analysis loss:
Validation Loss 1 | Picture by writer from wandb.ai
Since we're evaluating on the identical set of 250 validation samples, the analysis loss curve offers us a extra steady and significant view of mannequin studying — and can function a baseline for comparisons throughout future runs.
Right here, we observe a transparent and constant downward development all through coaching. The preliminary loss begins near 0.03, with a gradual enchancment as coaching progresses, finally stabilizing slightly below 0.015.
The graceful nature of this curve — in comparison with the extra variable coaching loss — displays the common construction of the validation set and confirms that the mannequin is generalizing properly to unseen samples, even with a small batch measurement and a single epoch of coaching.
Now, let’s examine the efficiency of this fine-tuned mannequin in opposition to the reference fashions on the HTML Similarity metric:
As we will see, this primary experiment already delivers robust efficiency beneficial properties — enhancing the bottom Granite-Imaginative and prescient 2B mannequin by a big margin (+0.18) and clearly outperforming LLaMA 90B Imaginative and prescient on this specialised job. Solely Pix2Struct retains a slight lead at this stage.
Second Run: vision_only
There isn’t a lot to investigate on this run. I examined a number of variations that might probably unblock studying within the imaginative and prescient layers — together with drastically rising the educational fee — however with out success.
Whereas the bottom code means that fine-tuning imaginative and prescient layers ought to be potential, in apply I discovered it was not working on this setup. The next analysis loss curve confirms that no studying occurred — the loss remained fixed all through coaching. To keep away from losing compute sources, I ended the run early:
Validation Loss 2 | Picture by writer from wandb.ai
Moreover, coaching was noticeably quicker on this run in comparison with the earlier lang_only experiment — suggesting that the language layers (which comprise the majority of the mannequin’s parameters) remained frozen, and solely the small imaginative and prescient layers had been being processed:
Validation Samples per Second 1 | Picture by writer from wandb.ai
Third Run: lang_vision
At this level, it was clear that solely language layers had been being successfully skilled. On this lang_vision run — the place each language and imaginative and prescient layers had been chosen — I anticipated outcomes just like lang_only.
Certainly, the analysis loss curve confirmed this expectation, displaying almost equivalent conduct to lang_only:
Validation Loss 3 | Picture by writer from wandb.ai
As soon as this was clear, I once more stopped coaching early to preserve sources, and proceeded to check new approaches.
Fourth Run: lang_table_only
This experiment modified the next element:
The objective of this run was to coach the mannequin to foretell solely the desk content material, with none surrounding HTML wrapper code. This strategy might assist enhance studying — by eradicating pointless tokens — and in addition align the coaching conduct extra intently with Pix2Struct’s mannequin.
Moreover, by stripping out the wrapper HTML, the goal sequences grew to become shorter — which allowed longer and extra complicated tables to suit throughout the mannequin’s context window. This variation might additionally enhance the mannequin’s capability to generalize to bigger or extra detailed tables.
Let’s take a look at the analysis loss in comparison with the primary run:
Validation Loss 4 | Picture by writer from wandb.ai
At first look, the upper analysis loss might sound counterintuitive. Nonetheless, there’s a transparent clarification: the wrapper HTML code is trivial for the mannequin to be taught — because it tends to be almost equivalent throughout many coaching samples. These repetitive tokens scale back cross-entropy loss, artificially reducing the typical loss in earlier runs. By eradicating them, the mannequin now focuses completely on the tougher and variable desk content material — leading to the next however extra significant loss worth.
Now, let’s see how this variation impacted the HTML Similarity metric:
On this first take a look at, we observe no important achieve or degradation from utilizing this new output format. It's potential that the mannequin would wish extra epochs or bigger coaching samples to totally adapt to this new format. One other concept is to replace the immediate — in order that from the very first step the mannequin understands it ought to focus solely on desk content material, somewhat than having to deduce this conduct via coaching alone. This will probably be explored within the subsequent experiments.
At this level, I found the promising Unsloth framework — which claims to supply 2x quicker coaching with as much as 70% decrease reminiscence utilization. In fact, I needed to check whether or not it might speed up my workflow.
My first concept was to leverage the improved reminiscence dealing with to run longer sequences (max_seq_length=2048), however in my case this shortly led to Out of Reminiscence (OOM) errors — so I reverted to my earlier configuration.
The coaching velocity enhancements, nevertheless, had been simple — virtually 4x quicker than my earlier runs:
Validation Samples per Second 2 | Picture by writer from wandb.ai
Sadly, this got here at a transparent value to loss efficiency:
Validation Loss 5 | Picture by writer from wandb.ai
Given this noticeable drop in high quality, I paused the experiment to analyze additional — significantly to see if Unsloth would permit me to coach imaginative and prescient layers, which is considered one of its marketed benefits. Nonetheless, I encountered precisely the identical conduct as with HuggingFace Transformers — no precise studying in imaginative and prescient layers.
With these leads to thoughts, I made a decision to put aside Unsloth for this undertakingand proceed utilizing HuggingFace Transformers, which had proven extra dependable studying in earlier runs.
Seventh Run: lang_table_only_2
Listed below are the brand new parameters for this run:
Going again to the earlier configuration, I needed to investigate the influence of a bigger digital batch measurement (by way of greater gradient_accumulation_steps).
The outcomes had been promising — the analysis loss grew to become smoother and trended nearer to the unique lang_only run, despite the fact that the mannequin was now predicting solely the desk content material:
Validation Loss 6 | Picture by writer from wandb.ai
Primarily based on this optimistic end result, I made a decision to maintain this gradient_accumulation_steps=8 setting for the ultimate experiment.
Evaluating this mannequin on HTML Similarity resulted in a small however significant enchancment — lastly reaching parity with Pix2Struct:
Naturally, the objective is not only to match Pix2Struct — however to surpass it. Two vital levers remained to discover: dataset measurement and immediate.
Eighth Run: lang_table_only_3
The up to date parameters for this run had been:
I by accident reverted gradient_accumulation_steps again to 4 on this run, solely realizing it as soon as the coaching was almost full — however this truly gave me an extra-chance to watch its impact on studying.
The principle objective right here was to double the coaching measurement (to 10K photographs) and to check the up to date, clearer immediate format. Sadly, a random CUDA error brought on coaching to halt round 80% completion — besides, the advance was clear:
Validation Loss 7 | Picture by writer from wandb.ai
As anticipated, some smoothness was misplaced as a result of smaller digital batch measurement, however the brand new immediate proved very efficient — noticeably boosting mannequin studying.
This set the stage completely for the ultimate experiment, utilizing this improved immediate, 10K coaching samples, and restoring gradient_accumulation_steps to eight.
Ultimate Run: lang_table_only_4
The ultimate set of parameters are:
The analysis loss for this ultimate run:
Validation Loss 7 | Picture by writer from wandb.ai
As anticipated, restoring the gradient_accumulation_steps to eight smoothed the loss curve, decreasing spikes and reaching barely decrease general loss values. With a full epoch of coaching on 10K photographs, this grew to become the best-performing mannequin throughout all experiments.
Now, let’s take a look at the ultimate outcomes on the HTML Similarity metric:
Ultimate HTML Similarity Outcomes | Picture by writer from matplotlib
The objective of this undertaking was achieved — the fine-tuned mannequin now surpasses each reference fashions on this job. Wanting again on the unique Granite-Imaginative and prescient 2B, the LoRA fine-tuning improved efficiency to 0.77, a +21 proportion level achieve — all achieved in beneath 8 hours on a consumer-grade GPU.
Qualitative Outcomes
To raised illustrate how a lot the mannequin improved via fine-tuning, let’s take a look at a particular instance: Picture ID 618932.
PubTabNet Analysis Pattern with ID 618932 | Picture from PMC
This desk is especially tough — beneath the Kappa column there are sub-headers (Current research and King et al. 2001). These complicated layouts usually problem generic VLMs, particularly after they haven’t been uncovered to sufficient related examples throughout coaching. Fashions can often perceive these sub-headers and reply questions about them, however producing the total desk construction in HTML usually requires additional immediate tuning and specialised fine-tuning.
Let’s first see how a base, non-fine-tuned Granite-Imaginative and prescient 2B mannequin performs on this job.
Baseline: Uncooked Granite-Imaginative and prescient 2B
The mannequin can reply questions based mostly on the desk accurately:
immediate='What's the Kappa worth for the query "Do you talk with this energy?" within the current research?'
res = predict(pattern['image'], immediate=immediate)
print(res)
Out[1]:
74
Nonetheless, when requested to generate the total HTML desk, the mannequin struggles:
immediate = "Convert desk to HTML ()"
html = predict(pattern['image'], immediate=immediate)
html = '' if '
' not in html else html
show(HTML(html))
Out[2]:
And the HTML Similarity metrics for this try:
Model similarity: 1.0000
Structural similarity: 0.4091
Lev-Edit Distance: 0.1434
Ultimate HTML Similarity Rating: 0.3619
High quality-Tuned Mannequin: lang_table_only_4
Now, let’s attempt the very same take a look at utilizing the fine-tuned mannequin:
from src.fashions.granite_vision.transformers_library import LLM as granite_vision
mannequin = granite_vision(
model_path,
adapter='lang_table_only_4'
)
Out[4]:
Mannequin loaded
Adapter 'lang_table_only_4' loaded
Adapter 'lang_table_only_4' merged
Utilizing cuda: NVIDIA GeForce RTX 4070 Ti SUPER
And the identical prediction immediate:
immediate = "Convert desk to HTML ()"
html = mannequin.predict(pattern['image'], max_new_tokens=1024, question=immediate)
show(HTML(html))
Out[5]:
The fine-tuned mannequin now produces an output that intently matches the bottom fact, accurately capturing the desk construction and sub-headers — one thing the bottom mannequin struggled with.
Ultimate HTML Similarity metrics:
Model similarity: 1.0000
Structural similarity: 0.9231
Lev-Edit Distance: 1.0000
Ultimate HTML Similarity Rating: 0.9615
This instance exhibits a transparent quantitative enchancment as properly: from a rating of 0.36 to 0.96 on a fancy desk construction — confirming that fine-tuning on this specialised job dramatically boosts the mannequin’s functionality.
Inference Pace
One main benefit of utilizing a smaller mannequin — except for the power to fine-tune on consumer-grade {hardware} — is inference velocity. Even when bigger fashions provide aggressive efficiency, latency and throughput stay key components, particularly in manufacturing settings.
Let’s examine the inference velocity of the completely different fashions:
Inference SpeedM | Picture by writer from matplotlib
As proven within the plot, Pix2Struct is by far the quickest mannequin. For some use circumstances — equivalent to batch-processing 1000's of paperwork for desk extraction — this velocity benefit might translate into important time financial savings and decrease compute prices.
Nonetheless, the fine-tuned Granite-Imaginative and prescient 2B achieves a superb steadiness when the quantity of paperwork to course of isn't huge, having a superior accuracy on this specialised job and fairly quick inference with out the necessity for very massive compute infrastructure.
Conclusions
This undertaking demonstrated that with LoRA-based fine-tuning and a focused job (desk extraction → HTML), a small vision-language mannequin (Granite-Imaginative and prescient 2B) can outperform a lot bigger fashions — even Meta’s 90B LLaMA Imaginative and prescient — whereas requiring solely a shopper GPU and fewer than a day of coaching.
A number of key takeaways:
Small, specialised fashions matter — you don’t all the time want 70B+ fashions to unravel particular issues with excessive accuracy.
Parameter-efficient fine-tuning (LoRA) is a game-changer: adapting massive basis fashions turns into accessible for many practitioners.
Immediate design and coaching targets have an enormous affect — small modifications (like switching to lang_table_only or refining the immediate) immediately impacted efficiency.
Having a customized metric (HTML Similarity) was essential to trace significant progress past generic text-based metrics.
Smaller fashions not solely practice quicker, but additionally infer quicker — ultimate for manufacturing pipelines with excessive quantity.
Lastly — and perhaps most significantly — one of these experimentation exhibits that you possibly can transfer quick and iterate even with restricted {hardware}. High quality-tuning highly effective open fashions and adapting them to real-world duties is not reserved to huge labs anymore.
I hope this encourages different AI engineers to experiment with small VLMs and fine-tuning methods for their very own initiatives and options — and to see that highly effective outcomes are potential even with out huge compute budgets!
What’s Subsequent?
There are undoubtedly some attention-grabbing follow-up concepts that may be explored subsequent:
Immediate engineering refinements: Ultimate assessments (whereas scripting this weblog) confirmed that separating prompts into system message (defining conduct) and consumer message (offering job directions) considerably improved the bottom mannequin’s efficiency. Making use of this technique throughout fine-tuning might additional improve the mannequin’s capability to persistently generate correct HTML. This will probably be examined in upcoming experiments.
Coaching imaginative and prescient layers: At present, solely the language layers are fine-tuned, as coaching the imaginative and prescient layers via text-only loss proved ineffective. A extra superior strategy might contain including an auxiliary imaginative and prescient loss — for instance, contrastive studying between imaginative and prescient outputs and HTML construction — to raised adapt the imaginative and prescient spine for desk extraction duties.
Improved generalization: The present mannequin is fine-tuned on a single dataset. Increasing coaching to incorporate extra various doc layouts, desk types, and noisy OCR eventualities might enhance robustness and transferability to real-world knowledge.
Hyperlinks
In case you preferred this put up, be happy to achieve out or share your individual experiments!
In direction of Knowledge Science is a group publication. Submit your insights to achieve our world viewers and earn via the TDS Writer Cost Program.
