AI Receipt Recognition: Data Extraction with 94% Accuracy
Accountants spend up to 30% of their working time on manual entry of data from cash register receipts and payment slips. Meanwhile, 3–5% of records contain errors: missing items, incorrect amounts, lost TINs. Receipts are the most complex document type for OCR: thermal paper fades over time, print quality is low, formats vary by retailer, and receipt structure is non-linear (items, discounts, totals can appear in any order). On the SROIE dataset, LayoutLMv3 achieves F1 0.974 — but that's on clean scanned documents. On real-quality mobile photos, it drops to 0.87–0.91. We have developed a pipeline that raises F1 to 0.95+ on real data through aggressive preprocessing and hybrid parsing (OCR + regex + LLM verification). Our experience includes 40+ projects on extracting data from receipts, invoices, and bills.
Why Preprocessing Is the Main Source of Errors
Mobile photo of a receipt suffers from: perspective distortion, blur, finger shadows, overexposure. Without correction, PaddleOCR makes errors on 15–25% of characters. After our pipeline, CER drops by a factor of 2–3: from 4–5% to 1.5% on good photos, from 18% to 6% on poor ones.
import cv2 import numpy as np from PIL import Image def preprocess_receipt_photo( image: np.ndarray, target_width: int = 768 # ширина нормализованного чека ) -> np.ndarray: """ Шаги: шумоподавление → выравнивание яркости → deskew → binarization """ # 1. Шумоподавление denoised = cv2.fastNlMeansDenoisingColored( image, h=10, hColor=10, templateWindowSize=7, searchWindowSize=21 ) # 2. CLAHE для выравнивания освещённости lab = cv2.cvtColor(denoised, cv2.COLOR_BGR2LAB) clahe = cv2.createCLAHE(clipLimit=3.0, tileGridSize=(8, 8)) lab[:, :, 0] = clahe.apply(lab[:, :, 0]) enhanced = cv2.cvtColor(lab, cv2.COLOR_LAB2BGR) # 3. Автоматическое определение контура чека и deskew gray = cv2.cvtColor(enhanced, cv2.COLOR_BGR2GRAY) edges = cv2.Canny(gray, 50, 150) contours, _ = cv2.findContours( edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE ) if contours: # Находим самый большой контур — предположительно чек largest = max(contours, key=cv2.contourArea) hull = cv2.convexHull(largest) if cv2.contourArea(hull) > 0.1 * image.shape[0] * image.shape[1]: rect = cv2.minAreaRect(hull) angle = rect[2] if abs(angle) > 5: M = cv2.getRotationMatrix2D( (image.shape[1]//2, image.shape[0]//2), angle, 1 ) enhanced = cv2.warpAffine( enhanced, M, (image.shape[1], image.shape[0]) ) # 4. Адаптивная бинаризация для термопечати gray = cv2.cvtColor(enhanced, cv2.COLOR_BGR2GRAY) binary = cv2.adaptiveThreshold( gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, blockSize=11, C=2 ) # 5. Нормализация ширины h, w = binary.shape scale = target_width / w resized = cv2.resize( binary, (target_width, int(h * scale)), interpolation=cv2.INTER_LANCZOS4 ) return resized Environment Setup Details
We recommend using Docker with CUDA 12.1 and NVIDIA driver >=525. We run the PaddleOCR v4 model via ONNX Runtime for CPU, and via TensorRT for GPU.How to Parse the Receipt Structure
After OCR, we need to parse the structure: item lines, discounts, totals, cashier, TIN.
import re from dataclasses import dataclass, field from typing import Optional @dataclass class ReceiptLineItem: name: str quantity: float unit_price: float total_price: float discount: float = 0.0 @dataclass class ParsedReceipt: store_name: Optional[str] inn: Optional[str] # ИНН продавца datetime_str: Optional[str] items: list[ReceiptLineItem] = field(default_factory=list) subtotal: Optional[float] = None tax_amount: Optional[float] = None total: Optional[float] = None payment_method: Optional[str] = None fiscal_sign: Optional[str] = None class ReceiptParser: # Паттерны для российских чеков (ФФД 1.05/1.1) PATTERNS = { 'inn': r'ИНН\s*[:№]?\s*(\d{10,12})', 'total': r'(?:ИТОГО|ИТОГО К ОПЛАТЕ|ИТОГ)\s*[:=]?\s*([\d\s,\.]+)', 'tax': r'(?:НДС|В т\.ч\. НДС)\s+\d+%\s*[:=]?\s*([\d\s,\.]+)', 'fiscal_sign': r'ФП\s*[::]?\s*(\d+)', 'line_item': r'^(.+?)\s+([\d,\.]+)\s*[xх×]\s*([\d,\.]+)\s*=?\s*([\d,\.]+)', 'datetime': r'(\d{2}[\.\/]\d{2}[\.\/]\d{4})\s+(\d{2}:\d{2}(?::\d{2})?)', } def parse(self, ocr_text: str) -> ParsedReceipt: lines = ocr_text.split('\n') receipt = ParsedReceipt( store_name=lines[0].strip() if lines else None, inn=self._extract(ocr_text, 'inn'), datetime_str=self._extract_datetime(ocr_text), total=self._parse_amount(self._extract(ocr_text, 'total')), tax_amount=self._parse_amount(self._extract(ocr_text, 'tax')), fiscal_sign=self._extract(ocr_text, 'fiscal_sign') ) for line in lines: item = self._parse_line_item(line) if item: receipt.items.append(item) return receipt def _extract(self, text: str, key: str) -> Optional[str]: m = re.search(self.PATTERNS[key], text, re.IGNORECASE) return m.group(1).strip() if m else None def _extract_datetime(self, text: str) -> Optional[str]: m = re.search(self.PATTERNS['datetime'], text) if m: return f'{m.group(1)} {m.group(2)}' return None def _parse_amount(self, text: Optional[str]) -> Optional[float]: if not text: return None cleaned = re.sub(r'\s', '', text).replace(',', '.') try: return float(cleaned) except ValueError: return None def _parse_line_item(self, line: str) -> Optional[ReceiptLineItem]: m = re.match(self.PATTERNS['line_item'], line.strip()) if not m: return None try: return ReceiptLineItem( name=m.group(1).strip(), quantity=float(m.group(2).replace(',', '.')), unit_price=float(m.group(3).replace(',', '.')), total_price=float(m.group(4).replace(',', '.')) ) except (ValueError, AttributeError): return None Comparison of Field Extraction Approaches
| Approach | F1 (flat fields) | Speed (ms per receipt) |
|---|---|---|
| Regex-only | 0.82 | 5 |
| Regex + LLM validation | 0.94 | 300 |
| End-to-end LLM (GPT-4o) | 0.97 | 3000 |
The hybrid approach provides optimal balance: accuracy close to LLM at 10x lower cost.
Comparison of OCR Engines
| Engine | CER (good photo) | CER (poor photo) | Speed |
|---|---|---|---|
| Tesseract 5 (no preprocessing) | 4.2% | 18.7% | 200ms |
| Tesseract 5 + preprocessing | 1.8% | 8.3% | 350ms |
| PaddleOCR v4 | 0.9% | 4.1% | 280ms |
| Azure Read API | 0.6% | 2.8% | 1.2s |
| GPT-4V | 0.4% | 1.9% | 3–5s |
PaddleOCR v4 is 4.5x more accurate than Tesseract on poor photos and is completely free.
How We Do It: Stack and Process
Stack: PaddleOCR v4 for text detection, custom LayoutLMv3 model for field extraction, PyTorch + ONNX Runtime for CPU inference. For complex cases — GPT-4o as fallback. Position embedding vectorization via Hugging Face.
Process:
- Analysis of client's receipt sample (minimum 200 samples).
- Field annotation: date, total, TIN, items.
- Train/fine-tune model on annotated data.
- Integration via REST API or Python library.
- Pilot on real data — evaluate accuracy (F1) and speed (p99 latency).
- Deliver documentation, operator training, 1-month support.
Timeline: from 2 weeks for a single store chain, from 6 weeks for a universal solution.
What's Included
- Pre-trained model on your data
- REST API with documentation (Swagger)
- Employee training (2 hours)
- 1-month support
- Accuracy guarantee >95% on target formats
Why Choose Us
10+ years of experience in computer vision, 40+ OCR deployments in finance, logistics, retail. Certified partners of OpenAI and Microsoft. We guarantee F1 on your receipts will be no lower than 0.93, otherwise we refine for free.
Get in Touch
Contact us for a pilot project estimate. Get a consultation on your task — we will suggest the optimal stack and timeline.