This publish was co-authored with Max Silfverberg (Knowledge Scientist, AI Options Lead), Antti Hallavo (Lead AI Software program Engineer), and Pontus Huotari (Lead Knowledge Scientist). We work at Alma Media, a Finnish digital companies, marketplaces and media firm. One in all our focus areas is growing AI/ML options for actual property itemizing companies, the place understanding picture content material performs an necessary function.
companies deal with a whole bunch of 1000’s of listings a 12 months. Most of these include dozens of photographs with no details about what they present. In the meantime, search, suggestions, and a variety of inside use circumstances all profit from figuring out whether or not a photograph represents a kitchen, flooring plan, or backyard.
Our answer is to mechanically tag photographs with room-type and content material courses. Our room varieties embody LIVING ROOM, KITCHEN, and BEDROOM. We additionally tag schematic content material like flooring plans and website plans. Moreover, we acknowledge realtor advertising supplies, aerial photographs, and backyard photographs. Altogether, there are 23 courses. As Determine 1 reveals, it is a basic multi-label classification activity; the identical house can embody a number of room varieties directly.
On the face of it, this sounds easy, however we have to make some difficult choices. How must you deal with a lounge picture that reveals a bed room by means of a doorway? What if the picture solely reveals 10% lounge and the remaining 90% is eating space? The solutions depend upon the applying.
If we have to discover all photographs exhibiting kitchens, we additionally wish to establish lounge photographs that present a kitchen within the background. Nonetheless, if the consumer particularly asks for kitchen photographs, we solely wish to present those the place the kitchen is in focus. To assist determine what to return, classification confidence is necessary. However relying on the way you implement your classifier, you may not have entry to that data.
Picture classifiers might be inbuilt many methods. The trendy default method is to run photos by means of a third-party API which internally makes use of a vision-language mannequin (VLM) to investigate photos and generate tags in accordance with a immediate.
One other possibility is to coach picture classifiers on prime of open-source ViT basis fashions like Google SigLIP and Meta DINO, both freezing the muse mannequin or fine-tuning it. Every of those designs comes with its personal benefits and trade-offs.
There already exists loads of work evaluating the approaches based mostly on numerical efficiency [1]. This weblog publish goes additional; we ask the generally missed query: How must you construct picture classifiers in a enterprise context?
Three questions earlier than you prepare something
We constructed our proprietary classifiers by fine-tuning google/siglip-base-patch16-224. The query is: why do this? Verify Determine 2 for the TL;DR. Learn on for the total story.
Query 1: Immediate an API or prepare your individual mannequin?
The selection to categorise by prompting by means of an exterior API or construct your individual classifier closely is dependent upon your use case. First, you have to think about whether or not your classification activity may even be prompted. It’s simple to immediate automotive and kitchen equipment classifiers however how about click-through price (CTR) for YouTube video thumbnails? Right here we’d like a trainable classifier as a result of we actually don’t know what influences the click on resolution. Conversely, if you want to extract structured JSON recordsdata from photographs representing constructing schematics, a easy classifier simply received’t reduce it.
Our actual property use case sits within the center. Many courses like KITCHEN and BATHROOM are simply promptable whereas others, like HALLWAY, LOFT and ALCOVE are fuzzier and tougher to verbalize.
In the event you determine to coach your individual classifiers as we did, you in fact want coaching information, in all probability no less than just a few thousand examples per class. When launching a brand new product, that’s one thing you won’t have. In the event you no less than have entry to plain photographs with out annotations,you may launch with aprompted VLM as your first classifier. Its predictions steadily accumulate into an annotated dataset, which you’ll later use to coach a customized classifier. This will require a cleanup move, for the reason that dataset inherits the VLM’s errors.
Price is one other main query. With a quantity within the thousands and thousands, the totally different classification approaches lead to dramatically divergent value profiles. Utilizing Google’s Agent Platform and the gemini-3.5-flash mannequin, the July 2026 value is roughly $1.50 per 1,000 photos (at 1K decision), so classifying 1,000,000 photographs prices roughly $1,500.
Utilizing our personal classifier on a devoted AWS EC2 g4dn.xlarge occasion with a T4 GPU, we will classify no less than 400 photos per second. At a July 2026 on-demand hourly price of $0.53, classifying 1,000,000 inputs comes out to $0.37 or roughly 1/4000th of the value for the API answer (inference compute solely).
However, if you classify just a few hundred photographs a day, from the price perspective it actually doesn’t matter the way you do it. Prices turn out to be a problem solely at scale.
Along with labels, classification confidence is usually helpful. As talked about above, if we provide kitchen photographs to the consumer, we should always in all probability go together with assured matches. It’s, nonetheless, difficult to derive dependable confidence estimates from a VLM; verbalized confidence estimates are identified to be poorly calibrated [2] and token log-likelihoods from an API incessantly don’t signify the class-probabilities you’re truly occupied with.
In the event you use an API, you may due to this fact must depend on granular tags like PROBABLE/POSSIBLE/UNLIKELY [3], and there’s no assure that these will likely be dependable both. In the event you as a substitute prepare your individual classifier, you get usable per-class scores which might be calibrated when wanted. Desk 1 summarizes how the 2 approaches examine.
| Prompted VLM (API) | Customized classifier | |
| Coaching information | None wanted | A couple of 1000 examples per class |
| Setup effort | Write a immediate | annotate, prepare, deploy |
| Price per 1M photographs | ~$1,500 | ~$0.37 (on GPU) |
| Per-class scores | Unreliable / not uncovered | Express, thresholdable, calibratable |
| Fuzzy courses | Arduous to verbalize in immediate | Learnable from examples |
| Altering the duty | Edit the immediate | Retrain the mannequin |
Query 2: Which basis mannequin to use?
In the event you determine to coach your individual classifier, the one cheap selection for many is to start out with a pretrained open-source imaginative and prescient mannequin, usually a imaginative and prescient transformer. For enterprise use, first test that the mannequin’s license permits business use.
Past that, your enterprise objective ought to drive the selection, as a result of totally different pretraining methods produce totally different representations:
- SigLIP [4] (Google) is skilled on captioned photos, so it attends to caption-worthy issues: canines, automobiles, individuals. Its representations are extremely object-oriented; background and digicam angle obtain far much less emphasis.
- DINO [5, 6] (Meta) is self-supervised with patch-level goals: each area of the picture contributes to the loss, not simply the caption-worthy objects. That makes it a robust candidate when background or structure issues [7]. We put this to the check beneath.
- RADIO / AM-RADIO [8] (NVIDIA) agglomerate representations from a number of ViT basis fashions by means of distillation.
- I-JEPA[9] Meta) is self-supervised like DINO however based mostly on masked prediction.
Query 3: To fine-tune or to not fine-tune?
The best strategy to begin is coaching a linear classifier on prime of frozen ViT representations. There are two main benefits: it’s conceptually easy and lightning quick. You may prepare on a laptop computer in a matter of minutes utilizing a 100k-instance coaching set. Sometimes, this results in very cheap efficiency.
In the event you determine to fine-tune, the perfect apply is to make use of low-rank adapters (LoRA), which freeze the precise ViT spine and inject just a few skinny trainable parameter layers into the mannequin [10]. After coaching, these might be merged with the unique mannequin to keep away from prices at inference time. LoRA retains coaching tractable even on a modest GPU setup, whereas delivering practically the identical efficiency acquire as full fine-tuning.
Since a shallow linear classifier normally performs effectively, fine-tuning can lead to modest positive aspects by way of uncooked F1 rating. Nonetheless, under-labeling is usually a actual drawback whenever you freeze your basis mannequin as we see beneath.
Easy has a price ticket
The foremost drawback with the frozen mannequin is low classification confidence. At a typical 0.5 working threshold, a whopping 35% of photographs obtain no labels from the mannequin. Tuning down the brink helps, however it comes at the price of decrease precision. Per-class thresholds may assist, however downstream purposes want scores that imply the identical factor throughout all 23 courses, and class-specific thresholds would drift with each retraining.
In apply, we settled on a compromise of 0.2, which offers cheap protection and precision. Determine 3 illustrates what this appears like for a single picture: at t = 0.5 nothing clears the bar, whereas at t = 0.2 the 2 appropriate labels come by means of.
A secondary drawback is poor classification on just a few frequent courses like GARDEN and HALLWAY. Edge circumstances additionally trigger issues: when a eating set is seen in a lounge picture, we wish to label it each LIVING ROOM and DINING AREA. Nonetheless, when the eating set is seen solely by means of a doorway, we don’t need the DINING AREA label.
These issues might be addressed by LoRA fine-tuning.
Placing it to the check
We determined to coach our personal classifier and in contrast the 2 customized approaches outlined above: a frozen basis mannequin mixed with a shallow linear classifier, and fine-tuning with LoRA. In each circumstances, we added 23 impartial classification heads on prime of the muse mannequin, one per class.
The enter is a picture vector generated by SigLIP. We moreover experiment with DINOv2 as a frozen baseline to see how caption-training compares to self-supervised coaching. LoRA fine-tuning is completed completely on SigLIP. We used the unique SigLIP mannequin reasonably than SigLIP 2 in these experiments; since we examine a frozen setup towards fine-tuning on the identical spine, the conclusions don’t hinge on the mannequin era.
For analysis, we use micro averaged F1 rating. This emphasizes efficiency on widespread courses like KITCHEN and LIVING ROOM, that are most central for our use circumstances.
Moreover, we consider protection on the check set: how lots of the photographs get no less than one label? Whereas there’s a pure residual of inputs that don’t fall into any of the 23 courses, we wish to discover all of the photographs that may be labeled.
Coaching
We prepare our classifiers on our personal proprietary set of 40k manually annotated photographs, the place every enter will get 1-3 class labels. Our validation information has 1.9k examples; we break up this into 100 growth and 1.8k check examples. Coaching, growth and check photographs come from distinct listings, so photographs of the identical property by no means seem in multiple break up.
For each our frozen baselines, we skilled 23 separate sklearn LogisticRegression fashions.
We skilled LoRA utilizing the PEFT library. Following widespread apply [10], we wrapped the SigLIP ViT self-attention question and worth layers in LoRA adapters, leaving the MLP layers untouched, and used BCE loss on prime of 23 impartial logistic classification heads. This meant coaching solely about 0.6% of the mannequin’s parameters, roughly a 99% discount in comparison with full fine-tuning. It’s also why the entire sweep matches on a single T4.
We did a random 40-trial hyperparameter sweep [11] over the configurations in Desk 2, fixing all different hyperparameters to customary values.
| Hyperparameter | Vary | Distribution |
lr |
1e-5 -> 1e-3 |
log-uniform |
batch_size |
{16, 32, 64} |
uniform categorical |
lora_r |
{8, 16, 32} |
uniform categorical (lora_alpha locked to lora_r) |
For quick and numerically safer coaching, we used blended precision with fp16 autocast and loss scaling [12]. We skilled for 20 epochs and picked the mannequin that delivers the perfect F1 rating on the event set.
All coaching is completed on an AWS EC2 g4dn.xlarge occasion with a single NVIDIA T4 having 16 GB VRAM.
Analysis
When it comes to plain micro averaged F1, variations are modest. At 82.6% F1, the fine-tuned mannequin beats each frozen SigLIP’s 78.4% F1 and frozen DINOv2’s 78.3% F1, however the distinction is just round 4 factors. The frozen SigLIP and DINOv2 classifiers ship basically equivalent efficiency. Frozen fashions are reported at their finest dev-set thresholds (0.2 for SigLIP, 0.35 for DINOv2); the fine-tuned mannequin at its default threshold of 0.5, which marginally understates its finest achievable F1 (83.1%). Desk 3 reveals the total outcomes.
| Metric | Frozen SigLIP (t = 0.2) | Frozen DINOv2 (t = 0.35) | LoRA SigLIP (t = 0.5) |
| Micro F1 | 78.4 | 78.3 | 82.6 |
| Micro precision | 85.1 | 85.1 | 86.2 |
| Micro recall | 72.8 | 72.4 | 79.3 |
| Unlabeled photographs | 9.4% | 10.8% | 3.4% |
The rise in F1 rating is mainly on account of recall, which improves by roughly 7 factors from 72.8% (SigLIP) and 72.4% (DINOv2) to 79.3%. At 85.1%, the frozen fashions’ precision is already very excessive, and it solely improves by about 1 level.
As Determine 4 reveals, these outcomes should not an artifact of the working threshold; the fine-tuned classifier outperforms the frozen SigLIP classifier at each working threshold, exhibiting that fine-tuning doesn’t merely push confidence up however genuinely improves classification efficiency. With solely 100 growth examples, we deal with the chosen thresholds and stopping epoch as coarse decisions reasonably than extremely tuned optima.
The modest positive aspects in micro averaged F1 conceal substantial enhancements for particular person courses, particularly for GARDEN (help in check set: 237) with a powerful 26-point rise in comparison with frozen SigLIP, and DINING AREA (help in check set: 150) with a decent 15-point enchancment.
The GARDEN class is a very fascinating instance, as a result of it’s usually all background, one thing that SigLIP doesn’t do effectively off the shelf, as mentioned above. In such circumstances, fine-tuning can ship dramatic enhancements.
Nonetheless, once we have a look at efficiency for the GARDEN class utilizing the frozen DINOv2 mannequin, a unique sample emerges: frozen DINOv2 F1 rating is 58%, a 15-point enchancment over the frozen SigLIP mannequin. Simply by selecting a extra appropriate basis mannequin, we now have gained greater than half of the efficiency hole in comparison with a fine-tuned SigLIP mannequin. DINING AREA additionally reveals an enchancment of 5 factors F1 rating.
On the identical time, DINOv2 underperforms in comparison with SigLIP on many courses the place semantic understanding of the picture appears extra necessary: it by no means predicts MARKETING (help in check set: 24), and KITCHEN (help in check set: 272) slips 7 factors. Apparently, efficiency additionally degrades on HALLWAY (help in check set: 86), a basically architectural class the place we might have anticipated DINOv2 to excel.
The one class the place fine-tuning degrades efficiency is KITCHEN: an F1 drop of 5 factors. We think about this minor, however that is naturally case dependent.
The true promoting level for LoRA fine-tuning is that it largely solves under-labeling. The frozen SigLIP classifier leaves 9.4% of photographs with out labels and DINOv2 does even worse at 10.8%. SigLIP’s price is 2.4x the pure price (3.9%) of photographs that genuinely fall into none of our 23 courses. LoRA finally ends up at 3.4%, barely beneath the pure price, that means it as a substitute very sometimes over-labels.
Determine 5 reveals that the under-labeling price of the fine-tuned classifier stays low for cheap working thresholds. We are able to commerce a little bit of recall for even greater precision. In distinction, the under-labeling price of the frozen classifier shoots towards the sky if one tries to sharpen precision by elevating the working threshold. As a classifier, it’s due to this fact far much less versatile than the fine-tuned one.
So, when ought to you fine-tune?
There are some things value contemplating. If under-labeling is an issue for you, then fine-tuning might be value it. The issue basically disappeared in our case.
For particular person courses, we did see massive positive aspects, particularly in recall. GARDEN and DINING AREA are actually acknowledged way more typically. Nonetheless, simply selecting an applicable basis mannequin (DINOv2 reasonably than SigLIP) recovered greater than half of the GARDEN hole with out fine-tuning. However, we didn’t observe degradation of precision, so positive aspects are real albeit modest by way of uncooked F1.
With a coaching set of 40k examples, the price for a full hyperparameter sweep turned out to be round $30, which is negligible. Nonetheless, if under-labeling just isn’t a problem, you may choose to make use of a frozen spine, particularly when periodic retraining is required. Coaching 23 classification heads on a CPU takes minutes and wishes no GPU; a full LoRA sweep takes days on a devoted GPU occasion, and that value repeats each time you retrain.
The bottom-effort possibility can be VLM-based classification, however at excessive volumes that turns into a major recurring value. The distinction between $1,500 and $0.37 for 1,000,000 inputs provides up rapidly. Nonetheless, keep in mind that coaching your individual classifier requires annotated information. We use 40k manually annotated photographs. That’s not free both.
For us, the funding has already paid off. The fine-tuned classifier now runs in manufacturing, and the labeling high quality is sweet sufficient that Alma has constructed new performance on prime of it. As a result of the labels are produced mechanically, they’re obtainable at scale for downstream purposes to construct on.
Conclusion
Whichever method you select, picture labeling pays off throughout the actual property itemizing service: search outcomes, suggestions, and a variety of inside use circumstances all enhance. In the event you’re not sure whether or not it’s value it, begin small and immediate a VLM to categorise a subset of your information. From there, a light-weight classification head on prime of an current embedding mannequin will reduce your prices, and in the event you want extra accuracy, fine-tuning your individual mannequin is the pure ultimate step.
References
[1] N. Kisel, I. Volkov, Okay. Janouskova and J. Matas, Multimodal massive language fashions as picture classifiers (2026), arXiv:2603.06578
[2] M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He and B. Hooi, Can LLMs specific their uncertainty? An empirical analysis of confidence elicitation in LLMs (2024), Worldwide Convention on Studying Representations (ICLR)
[3] S. Lin, J. Hilton and O. Evans, Educating fashions to specific their uncertainty in phrases (2022), Transactions on Machine Studying Analysis
[4] X. Zhai, B. Mustafa, A. Kolesnikov and L. Beyer, Sigmoid loss for language picture pre-training (2023), IEEE/CVF Worldwide Convention on Laptop Imaginative and prescient (ICCV)
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski and A. Joulin, Rising properties in self-supervised imaginative and prescient transformers (2021), IEEE/CVF Worldwide Convention on Laptop Imaginative and prescient (ICCV)
[6] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, et al., DINOv2: Studying sturdy visible options with out supervision (2024), Transactions on Machine Studying Analysis
[7] M. El Banani, A. Raj, Okay.-Okay. Maninis, A. Kar, Y. Li, M. Rubinstein, D. Solar, L. Guibas, J. Johnson and V. Jampani, Probing the 3D consciousness of visible basis fashions (2024), IEEE/CVF Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR)
[8] M. Ranzinger, G. Heinrich, J. Kautz and P. Molchanov, AM-RADIO: Agglomerative imaginative and prescient basis mannequin cut back all domains into one (2024), IEEE/CVF Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR)
[9] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun and N. Ballas, Self-supervised studying from photos with a joint-embedding predictive structure (2023), IEEE/CVF Convention on Laptop Imaginative and prescient and Sample Recognition (CVPR)
[10] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang and W. Chen, LoRA: Low-rank adaptation of enormous language fashions (2022), Worldwide Convention on Studying Representations (ICLR)
[11] J. Bergstra and Y. Bengio, Random seek for hyper-parameter optimization (2012), Journal of Machine Studying Analysis, 13
[12] P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh and H. Wu, Blended precision coaching (2018), Worldwide Convention on Studying Representations (ICLR)
