Hundreds of waybills, invoices, and contracts arrive daily as scans. Manual entry is a bottleneck, errors are inevitable. Time and budget losses grow. We solve this with Document AI — automatic structured data extraction. Our approach achieves F1 up to 96% on complex fields and reduces processing time by 10–20 times. Unlike classic OCR that outputs flat text, Document AI understands semantics and visual element layout. The model knows that 12 345.00 to the right of Total is the final amount, not a random number. This is a key element of document workflow automation, where machine learning eliminates routine.
Extraction quality directly depends on annotation and model architecture. We use LayoutLMv3 — state-of-the-art for documents with arbitrary templates. Below we break down key problems and their solutions.
How Document AI Extracts Structured Data?
Stack: PyTorch, Hugging Face Transformers, LayoutLMv3, Tesseract for OCR. For each document type we train a separate model with BIO entity tagging (NER).
from transformers import LayoutLMv3Processor, LayoutLMv3ForTokenClassification from PIL import Image import torch class DocumentFieldExtractor: def __init__(self, model_path: str, labels: list[str]): self.processor = LayoutLMv3Processor.from_pretrained(model_path) self.model = LayoutLMv3ForTokenClassification.from_pretrained( model_path, num_labels=len(labels) * 2 + 1 # BIO tagging ) self.labels = labels self.model.eval() @torch.no_grad() def extract(self, image_path: str) -> dict: image = Image.open(image_path).convert('RGB') encoding = self.processor( image, return_tensors='pt', truncation=True, padding='max_length', max_length=512 ) outputs = self.model(**encoding) predictions = outputs.logits.argmax(-1).squeeze().tolist() return self._decode_entities(encoding, predictions) Data annotation is done in Label Studio with Document AI support — the annotator selects areas on the scan and assigns field types. A minimum of 200–500 annotated documents for baseline quality.
Fine-tuning uses the standard Trainer from Hugging Face:
from transformers import TrainingArguments, Trainer from datasets import load_dataset training_args = TrainingArguments( output_dir='./invoice_extractor', num_train_epochs=20, per_device_train_batch_size=2, per_device_eval_batch_size=2, learning_rate=5e-5, warmup_steps=100, weight_decay=0.01, fp16=True, evaluation_strategy='epoch', save_strategy='best', metric_for_best_model='f1' ) trainer = Trainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset, compute_metrics=compute_ner_metrics ) trainer.train() What Problems Does Document AI Solve?
- Non-standard templates — different suppliers format invoices differently. LayoutLMv3 is robust to variations if trained on a representative sample.
- Low-quality scans — skew, noise, poor lighting. Preprocessing (deskew, denoise) plus data augmentation improve robustness.
- Mixed document types — invoices and contracts in the same stream. We train multi-type models or a classifier before extraction.
- Extracted data validation — checksums, date formats, reference data. Post-processing with business rules filters out errors.
| Problem | Solution |
|---|---|
| Non-standard templates | LayoutLMv3 adapts with enough examples; train on a representative sample |
| Low-quality scans | Preprocessing (deskew, denoise) + data augmentation |
| Mixed document types | Use a classifier before extraction or a multi-type model |
| Extracted data validation | Post-processing with business rules: checksums, date formats, references |
Why LayoutLMv3 Beats Classic OCR?
Classic OCR (Tesseract, Abbyy) outputs only text and coordinates. Document AI based on LayoutLMv3 additionally understands semantics: the model knows that the string "TIN 7701234567" is a tax ID, not an account number. As shown in the work LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking, on the CORD dataset LayoutLMv3 achieves F1 95.5%, while OCR+regex is around 70–80%. LayoutLMv3 is 5–10 times more accurate for complex fields, especially dates and amounts. Deploying such a system typically pays for itself within 3–6 months by reducing manual labor, with significant monthly operational savings. Additionally, reduced processing errors provide further cost benefits.
| Dataset | Field | F1 (LayoutLMv3) |
|---|---|---|
| FUNSD (forms) | Entity extraction | 92.0% |
| CORD (receipts) | Transaction fields | 95.5% |
| SROIE (invoices) | 4 key fields | 96.6% |
| DocVQA (QA on documents) | ANLS | 83.5% |
Typical Mistakes When Deploying an Invoice Extractor
- Too little annotated data — less than 200 scans per type leads to overfitting.
- Homogeneous templates in the sample — the model doesn't see variations and fails on new formats.
- Ignoring augmentation — without rotations and noise, the model crashes on real scans.
- Lack of post-processing — dates in different formats, amounts without separators — the model outputs raw tokens instead of normalized values.
What's Included?
- Custom model — fine-tuned LayoutLMv3 for your document type.
- API container — Docker image with REST API and request examples.
- Documentation — model description, integration instructions.
- Employee training — webinar on data annotation and extractor use.
- Support — 3 months of quality monitoring and retraining if needed.
Get a consultation for your project — evaluation is free.
Data requirements: scans in PDF, JPG, PNG formats. Resolution at least 150 DPI. Ideally templates are similar, but the model adapts to variations with enough examples (from 200).
Step-by-Step Deployment Process
- Data collection and annotation — we select 200–500 scans of your type, annotate fields using Label Studio.
- Model fine-tuning — we tune LayoutLMv3 to your annotation, optimize hyperparameters.
- Post-processing — add validation rules (formats, checksums).
- Packaging into API — model in a Docker container with REST endpoints.
- Integration and testing — connect to your system, test on a holdout set.
- Monitoring and support — track quality, retrain if needed.
Timeline: one document type with annotated data takes 4–6 weeks; for 3–5 types — 8–12 weeks; universal extractor — up to 18 weeks.
Our Expertise and Guarantees
We are a team of ML engineers with 7+ years of experience in Computer Vision and NLP. We have completed over 30 Document AI projects for banks, logistics, and retail. We hold certified experience with Hugging Face and NVIDIA.
We guarantee accuracy on your document type: if F1 is below the agreed threshold on the test set, we refine the model for free. Deployment includes monitoring and the possibility of retraining within 3 months.
Order a pilot project — evaluate the result on your data. If you want to automate document workflow, contact us for a consultation and project assessment.