I wanted to make a web app that extracts data from pictures of receipts (or screenshots) in a standard format (total price, list of items purchased, company, date, etc.) for finance purposes. I am extremely new to HuggingFace and the ML community in general so I thought here would be a good place to ask. What are the options in terms of models, and if there isn’t one, how could I make one? I apologize if this question is a lot.
Good news, there’s actually a model built specifically for this on the Hub: AdamCodd/donut-receipts-extract, fine-tuned on receipt data, outputs structured fields (total, items, phone, discount, etc.) directly, no separate OCR step needed since Donut is OCR-free (image straight to structured text). That’s the easiest starting point for exactly your use case.
If you want better accuracy on messy phone-camera photos (skew, bad lighting, crumpled receipts) rather than clean scans, a general vision-language model like Qwen2.5-VL-7B (or the OCR-focused fine-tune syntheticbot/ocr-qwen) handles that better since it’s more robust to real-world image quality, you just prompt it to extract fields as JSON. Donut is lighter/faster and purpose-built, VL models are heavier but more forgiving of bad photos, worth trying Donut first since it’s already tailored to receipts, and falling back to a VLM if your real-world photos are too noisy for it.
If using a model alone isn’t working well, it may be faster to think in terms of a pipeline:
Short answer: I would not start by training a receipt model from scratch.
For a first version, I would define the JSON shape you want and try a model that can read the receipt image and return that JSON directly. NuExtract3 looks like a practical starting point for this because its structured-extraction interface is built around a JSON template.
By “schema” here I just mean something like:
{
"company": "verbatim-string",
"date": "verbatim-string",
"subtotal": "verbatim-string",
"tax": "verbatim-string",
"total": "verbatim-string",
"items": [
{
"description": "verbatim-string",
"quantity": "verbatim-string",
"unit_price": "verbatim-string",
"line_total": "verbatim-string"
}
]
}
For a first prototype, that can be as simple as:
receipt photo
↓
model
↓
JSON
You can stop there if it works well enough on the receipts your app will actually receive. I would only add more machinery when a particular type of error starts repeating.
A practical starting route
define the JSON fields you need
↓
try direct image → JSON extraction
↓
test it on representative receipts
│
├─ works well enough
│ → build the first app version
│
├─ it is misreading text/images
│ → add OCR or a second vision model
│
├─ values look plausible but were not actually printed
│ → add an evidence/grounding check
│
├─ line items are grouped incorrectly
│ → retry/validate only the item block
│
└─ the same domain-specific mistake keeps repeating
→ then consider fine-tuning
A Qwen-family vision-language model (VLM — a model that can inspect an image and follow text instructions) is also worth testing. For example, syntheticbot/ocr-qwen is a Qwen2.5-VL-based model whose model card is specifically aimed at OCR/document use.
I would not assume in advance that one of these is universally better. The useful comparison is which one works on your stores, languages, layouts, screenshots/photos, and line-item formats.
Keep extracted, normalized, and calculated values separate
This is probably the design choice I would make before comparing many models.
printed / extracted
!= normalized
!= derived / inferred
For example, if a receipt visibly contains:
TOTAL 88.000
I would first preserve:
{
"total_raw": "88.000"
}
and only later convert it into whatever numeric representation your finance application uses.
Likewise, if the receipt does not print a subtotal, I would prefer:
{
"subtotal_raw": null
}
rather than letting the extraction model calculate a plausible subtotal and put it into the same field.
If your application wants a calculated subtotal, keep that separately:
{
"subtotal_raw": null,
"subtotal_derived": 76000
}
NuExtract3 has a documented verbatim-string type specifically for extractive text, which is useful here because receipt punctuation and number formatting can be meaningful.
A recent receipt benchmark, ReceiptBench, also treats perception, normalization, semantic reasoning, and structure parsing as distinct problems rather than one monolithic “OCR accuracy” problem.
Runnable example
If a concrete example is useful, I also put a small T4/Colab reference notebook here:
T4 receipt extraction / evidence demo
It is only a reference implementation, not a benchmark.
You can stop at the simple route above. The sections below are mainly for cases where you want to compare options or make the extraction more reliable.
Model choices and when I would use each
NuExtract3: direct structured extraction
NuExtract3 is the one I would probably test first for this particular app shape.
Its documented structured-extraction path takes:
- the document — text, image, or both;
- a JSON template describing what to extract;
- optional instructions;
- optionally, in-context examples.
That maps naturally to:
receipt image
→ company/date/total/items JSON
For receipt fields where you care about exactly what was printed, I would start with verbatim-string rather than forcing every amount into a numeric type immediately.
The documented structured-extraction instructions also support missing leaf fields as null and missing arrays as [], which is useful because “not printed” should be a valid state rather than something the model feels obliged to invent.
See:
If local deployment becomes important later, NuMind also publishes several quantized NuExtract3 variants in its NuExtract3 collection. I would choose among those only after knowing your serving hardware.
Qwen / OCR-tuned Qwen
A Qwen model can be used in two different ways:
- ask it to extract the structured receipt directly; or
- use it as a second reading of the image when the first model/OCR is uncertain.
For reliability, I find the second role particularly interesting.
For example:
receipt
├─ structured extractor
└─ second OCR/VLM reading
↓
compare evidence
The syntheticbot/ocr-qwen model card describes OCR fine-tuning, structured document output, and text localization/bounding boxes.
I would treat that as a candidate to test, not proof that it will beat another model on your receipts.
Conventional OCR can still be useful
OCR means simply “read the text from the image.”
It does not have to be the whole solution.
Even if a VLM produces the final JSON, a conventional OCR pass can help answer questions such as:
Did the returned total actually appear in the receipt?
Was there really a SUBTOTAL label?
Did the extractor miss one product row?
Docling currently supports multiple OCR engines, including RapidOCR, EasyOCR, Tesseract variants, and others.
So one reasonable architecture is:
image
├─ model → structured JSON
└─ OCR → source text / positions
↓
validate
I would not assume the OCR reconstruction itself is infallible either. Keeping the lower-level text/geometry available is useful when table or layout reconstruction groups things incorrectly.
Why a pipeline can be easier than searching for one perfect model
When something fails, I would first ask what kind of failure it is.
1. The text itself is wrong
Example:
48982.68
is read as:
48982.66
That is mainly a perception/OCR problem.
Possible next steps:
- another OCR engine;
- a document-oriented VLM;
- better image capture/preprocessing.
2. The text is readable, but the model returns something that is not printed
Example:
subtotal = 154.08
even though the receipt never printed a subtotal.
That is a grounding/extraction-contract problem rather than ordinary OCR.
I would require evidence for important extracted fields:
returned value
→ should be traceable to visible receipt text
If it is not, leave it unresolved or retry that field.
3. The text is correct, but line items are grouped incorrectly
For example:
Coffee
2 x 4.50
9.00
may be three visual rows but one purchased item.
A model can read all three rows correctly and still build the wrong JSON structure.
That is a structure problem.
The useful response is usually not “replace the entire model immediately,” but something like:
retry the item block
or
use layout/OCR evidence to regroup it
4. The extraction is correct but normalization is wrong
For example:
88.000
can mean different things depending on locale/context.
This is why I would keep the printed string first and normalize later.
5. A value is being calculated rather than extracted
Arithmetic can be useful as a validation signal:
quantity × unit_price ≈ line_total
but arithmetic cannot prove that a subtotal was actually printed.
So I would treat arithmetic as:
consistency check
not:
source evidence
This kind of decomposition is also reflected in ReceiptBench, which splits receipt understanding into perception, format normalization, semantic reasoning, and structure parsing.
Line items are the part I would test most carefully
Header fields such as merchant/date/total are often simpler key/value fields.
Purchased items are where receipts become more awkward.
You can see patterns such as:
product name
quantity × unit price
line total
or:
product
size/modifier
discount
price
and those may span multiple visual rows.
The CORD receipt dataset is a useful example of why this matters. Its receipt annotations distinguish fields such as:
- item name;
- quantity;
- unit price;
- total item price;
- discount;
- sub-items/modifiers;
and also include row/group hierarchy.
I would therefore evaluate line items separately from header fields.
Useful checks include:
item count
missing items
duplicate items
description match
quantity
unit price
line total
A model that gets merchant/date/total perfectly can still be unacceptable for a finance app if it quietly drops one purchased item.
CORD is useful as a structural test/reference, but it is Indonesian receipt data, so I would still create a holdout from the receipts your application is actually intended to process.
Links:
Why I would keep a validation/fallback path
I did a few small local diagnostics while looking at this. They are not a benchmark, but three observations were consistent enough to affect how I would design the app.
Plausible does not always mean printed
Structured models can produce valid, numerically plausible JSON while still filling a field that was not visibly present.
So:
valid JSON
!=
grounded extraction
and:
arithmetic plausibility
!=
proof that the field was printed
Line-item grouping can fail independently of OCR
The model may read every token correctly but associate a quantity/price/modifier with the wrong item.
That is why I would test items separately.
Different reading methods can make different mistakes
A conventional OCR engine and an OCR-oriented VLM can miss different parts of the same receipt.
That makes this kind of conditional path useful:
cheap/default path
↓
is important evidence missing?
│
├─ no → accept
│
└─ yes → stronger second OCR/VLM pass
The point is not to run two large models on every image. The point is to have a second route available for the receipts where the first route is uncertain.
How I would evaluate it before building the web app around it
I would make a small holdout from receipts that resemble the actual input to the app.
Try to include the variation you expect:
different merchants
different countries/languages
screenshots and phone photos
clean and angled/blurred images
simple and multiline items
cash/card receipts
different tax/service-charge formats
Then measure different things separately.
Header fields
For example:
company
date
total
subtotal
tax
cash/change/payment
Items
For example:
item count
missing items
duplicate items
description
quantity
unit price
line total
Grounding
Also track:
value was printed and extracted
value was printed but omitted
value was emitted but not printed
That last category is especially important for finance-oriented extraction.
I would keep a simple error table such as:
receipt
field/item
expected raw text
predicted raw text
failure type
which model/path produced it
After perhaps 10 or so representative receipts, you will often already see which error class dominates. That is only an exploratory set, not a statistically meaningful benchmark, but it is usually enough to decide what to investigate next.
Then branch:
mostly OCR failures
→ try a different OCR/VLM reading path
mostly unsupported fields
→ add stronger source-evidence checks
mostly item grouping
→ improve item parsing/retry
same merchant/layout mistake repeatedly
→ fine-tuning may now be worth it
If you want a more formal receipt-oriented evaluation reference, ReceiptBench and its code are useful. Its evaluation separates different receipt tasks rather than hiding everything inside one score.
When I would consider fine-tuning
There are two different ideas that are easy to mix up when starting out:
training from scratch
means building the model’s capabilities through a very large training process.
fine-tuning
means starting from an existing model and teaching it additional examples from your own task.
For this receipt app, if training becomes necessary at all, I would almost certainly look at fine-tuning an existing model, not training a vision-language model from scratch.
My decision tree would be:
Does an existing model handle most target receipts?
|
+-- yes
| |
| +-- failures are inconsistent/random
| | → improve pipeline/evidence first
| |
| +-- the text itself is unreadable
| | → improve OCR/perception first
| |
| `-- the same mapping/layout mistake repeats
| → fine-tuning becomes interesting
|
`-- no
|
+-- model cannot read the document
| → perception/OCR/data problem
|
`-- model reads it but systematically maps it wrong
→ supervised fine-tuning is plausible
A training example would roughly contain:
receipt image
+
instruction / desired JSON shape
+
correct structured output
I would make sure the training targets already reflect the contract you want:
- printed values remain raw;
- missing fields are actually missing/null;
- derived values are not disguised as extracted ones;
- difficult multiline items are represented consistently.
If possible, also avoid making train/test splits where nearly identical merchant templates occur on both sides. Holding out merchants/layouts can tell you much more about whether the model generalizes.
You also do not necessarily need to update every parameter in the model.
Techniques such as LoRA train a much smaller set of parameters while leaving most of the base model frozen. Hugging Face documents this through PEFT and its LoRA guide.
For vision-language models specifically, the current TRL SFTTrainer supports datasets containing an image or images field, so there is an established Hugging Face path if you eventually reach this stage.
There is also a worked VLM fine-tuning recipe.
But I would only invest in this after the simple baseline tells you what repeated failure you are actually trying to train away.
A few production notes
For a finance-oriented app I would keep some traceability around important fields.
For example:
raw extracted value
source/evidence used
normalization result
derived result, if any
That makes it much easier to inspect a suspicious transaction later.
A conditional fallback is also useful operationally:
normal/cheap path
↓
accept if sufficiently supported
↓ otherwise
stronger model/OCR path
so the more expensive model does not need to run for every receipt.
And since receipts can contain addresses, payment fragments, loyalty/customer identifiers, and purchase history, check the retention/storage/privacy behavior of whatever hosted inference service you eventually use.
If I were building version 1, I would therefore do this:
1. Choose the small set of JSON fields the app actually needs.
2. Try NuExtract3 directly on representative receipts.
3. Keep printed/raw values separate from normalized/calculated values.
4. Check header fields and line items separately.
5. Add OCR/Qwen evidence only for the failure types that actually appear.
6. Fine-tune only if a stable target-domain error remains.
That should get you to a useful prototype without requiring you to solve “receipt AI” all at once.
===========================
JSON SCHEMA (Receipt Extraction)
{
“store_name”: “”,
“address”: “”,
“date”: “”,
“time”: “”,
“items”: [
{
“product”: “”,
“quantity”: “”,
“unit_price”: “”,
“total_price”: “”
}
],
“subtotal”: “”,
“tax”: “”,
“total”: “”,
“payment_method”: “”
}
===========================
OPTIMIZED PROMPT (NuExtract3 / HF)
You are an extraction model. Read the receipt image and return ONLY valid JSON following the schema below.
Do not add text, comments, explanations, or extra fields.
If a field is missing in the receipt, return an empty string for that field.
JSON schema:
{
“store_name”: “”,
“address”: “”,
“date”: “”,
“time”: “”,
“items”: [
{
“product”: “”,
“quantity”: “”,
“unit_price”: “”,
“total_price”: “”
}
],
“subtotal”: “”,
“tax”: “”,
“total”: “”,
“payment_method”: “”
}
Extract all visible information from the receipt image and fill the JSON accordingly.
Return ONLY the JSON object.
===========================
FULL PIPELINE (Conceptual)
-
INPUT
- User uploads a receipt image (JPG/PNG/PDF).
- Optional: user provides language preference.
-
PREPROCESSING
- Convert image to RGB.
- Resize to model-friendly resolution.
- Optional: apply OCR fallback if text is extremely small.
-
MODEL CALL (NuExtract3 or similar)
- Provide the optimized JSON prompt.
- Send the receipt image.
- Model returns structured JSON.
-
VALIDATION
- Check JSON format.
- Ensure required fields exist.
- Normalize numbers (prices, quantities).
- Remove currency symbols if needed.
-
POST-PROCESSING
- Convert strings to numeric types.
- Recalculate totals if needed.
- Validate item list consistency.
-
OUTPUT
- Return final JSON to user.
- Provide optional “confidence score” per field.
-
FEEDBACK LOOP
- User can report missing fields or errors.
- Pipeline adjusts preprocessing or prompt accordingly.
===========================
OPTIONAL MESSAGE TO USERS
If you try this pipeline, I would really appreciate any feedback about your experience.
Your feedback helps improve the extraction accuracy and makes the tool more useful for everyone.