CyberFreezeDev
Loading
Testing which mix of OCR, layout and language models gives document fields a finance team can trust.
The question: which OCR, layout and model combination gives fields that can go straight into accounts, and how doubtful ones reach a person.
For the paperwork Indian businesses actually receive, which combination of OCR, layout understanding and language models produces fields that can go straight into accounts, and how do we send the doubtful ones to a person?
The documents are messy: GST invoices from hundreds of suppliers in hundreds of layouts, phone photos of bills, scanned bank statements, PAN and Aadhaar cards, application forms with handwriting, and PDFs that look like text but are really images. We have already built an OCR pipeline for invoices, IDs and forms with human review for a client, and expenses from bills are part of Fint. This track turns that experience into a repeatable test so we can choose components on evidence.
Results feed our OCR and document intelligence service and our GST and accounting automation work.
A confidently wrong GSTIN flows into returns and ledgers and may surface months later.
A missing field gets noticed and filled in. A confidently wrong GSTIN or taxable value flows into GSTR-1, input tax credit claims and ledgers, and is found months later, if at all.
Automation saves keying time only if staff trust the output enough to stop re-checking.
CA firms and finance teams spend hours keying invoices and statements. Automation only saves that time if staff trust the output enough to stop re-checking every field.
Aadhaar and other IDs carry storage rules and DPDP duties, so pipelines must mask and keep less.
Aadhaar numbers have specific rules on storage and display under the Aadhaar Act and UIDAI regulations, and all of it is personal data under the DPDP Act 2023. A pipeline must mask and limit what it keeps.
The pipeline is built from swappable stages so each one can be tested on its own.
Rules catch errors that no model confidence score will. These come straight from how GST invoices are structured.
A GSTIN that fails its 15-character pattern or check digit is wrong, whatever the model says.
A GSTIN is 15 characters: a two digit state code, the holder's 10 character PAN, an entity number, the letter Z by default, and a final check character. A value that fails the check digit is wrong, whatever the model says.
Same state means equal CGST and SGST; different states mean IGST. A mismatch flags a misread.
If supplier and buyer are in the same state, the invoice should carry CGST and SGST in equal amounts. If they are in different states, it should carry IGST. A mismatch means a misread field or a wrong invoice.
Lines must add up to the taxable value, and tax and total must match.
Line amounts must add up to the taxable value, tax must match the stated rates, and the grand total must equal taxable value plus tax, allowing for rounding.
Dates must be sensible and the same supplier invoice number must not already exist.
The invoice date must fall in a sensible period, and the same supplier GSTIN with the same invoice number must not already exist in the books.
Fields passing every rule and both passes are accepted; the rest go to a person.
Fields that pass every rule and agree across two extraction passes are accepted. Anything else goes to a person, with the reason shown.
| Metric | How we measure it | What counts as success |
|---|---|---|
| Field accuracy | Each extracted field compared to an answer key a person prepared, exact match after normalising spaces and case | Matches on every field that the rules accept |
| Confident errors | Fields that were wrong but passed every check and were not sent for review | None. This is the number we care about most |
| Straight-through rate | Documents that needed no human edit at all | Rises as rules and prompts improve, without confident errors rising |
| Review time | Seconds a reviewer spends per routed document | Lower than keying the document by hand |
| Character error rate | Raw OCR text against a typed transcript, for comparing engines | Used to choose engines, not as the end goal |
| Cost and speed per page | Run time and any API cost per page | Affordable at the client's monthly volume |
Test documents are created by us or come from public samples, with personal details replaced. No client documents are used in published results.
Public knowledge and our working hypotheses. No benchmark numbers are published yet.
Many billing PDFs already hold exact text, so OCR should be the fallback.
Many PDFs from billing software already contain the exact text. Reading that layer avoids OCR errors entirely, so OCR should be the fallback, not the first step.
Fluent output can be wrong; rules and a second pass give a better signal.
Language models can produce fluent, well-formatted values that are wrong. Validation rules and a second independent pass give a better signal than the model's own certainty.
Hypothesis: GSTIN, tax type and arithmetic rules catch most confident errors. We measure what slips through.
We expect GSTIN check digits, tax type checks and arithmetic to catch most confident errors on invoices. We are measuring what slips through anyway.
Hypothesis: for phone photos, image clean-up before OCR matters more than the engine chosen.
Skew, shadows and folded paper should hurt every engine. We expect image clean-up before OCR to matter more than the choice of engine.
Notes will be published in the R&D / Labs notes section of our blog. Only documents we created or public samples are used, with personal details replaced.
| Note | Published when | What it contains |
|---|---|---|
| Method note | Before any scores | Document types in the test set, the answer-key format, the validation rules above and how confident errors are counted |
| OCR engine comparison | When Tesseract, PaddleOCR and docTR have run on the full image set | Character error rate per document type, phone photos and scans reported apart, with engine versions and settings |
| Extraction and routing | After the engine comparison | Field accuracy, confident errors, straight-through rate and review time for each extraction setup |
| Local extraction | When the LLM track has baseline results | The same scores with the extraction step on a local model, so the cost of keeping documents in-house is visible |
Printed text is reliable with modern engines. Handwriting varies a lot by writer and form design, which is why handwritten fields in our pipeline go to a person by default until the test data shows otherwise.
No. It removes typing and moves staff to checking the cases the system flags. In our designs, as in Fint, nothing reaches the books without a person confirming it.
The OCR engines above run locally. Whether the extraction step can run locally is what our local LLM track is testing.
GST invoices, bills, bank statements, PAN and Aadhaar cards, and application forms, in PDF and image formats, using documents we create or public samples.
Last updated