OCR pipeline for document photos

OCR pipeline for document photos

A photo of a document is not a scan. Typical problems: perspective distortion (text becomes trapezoidal), shadows from fingers and spine, uneven lighting, motion blur, reflections on glossy surfaces. A high-quality recognition system must correct all these distortions before sending to an OCR engine. Without preprocessing, even the best models (PaddleOCR, GPT-4o) show CER of 20–30% on such photos. For passports and IDs, CER <5% is required — achievable only with a comprehensive pipeline.

We developed a solution that automatically detects the document, aligns perspective, removes shadows and glare, and only then runs OCR. This approach yields consistently low error even on imperfect shots. Our pipeline processes a frame in ~200 ms on GPU and has been deployed in projects for banks and government organizations — over 5 years of experience, more than 20 successful integrations.

How we align the document in the photo?

The first task is to find the document in the frame and correct perspective. Typical situation: a user photographs a passport at an angle, and the text becomes unreadable for regular OCR. We use contour detection after preprocessing: gray conversion, blur, Canny edge detection, then search for a quadrilateral that occupies >20% of the frame. If such a contour is found, we apply perspective transformation (homography). This gives a front-facing view of the document.

import cv2 import numpy as np class DocumentPhotoOCR: def __init__(self): self.ocr = PaddleOCR(use_angle_cls=True, lang='ru', use_gpu=True) def process(self, image_path: str) -> dict: image = cv2.imread(image_path) # 1. Detect document in frame doc_corners = self.detect_document(image) # 2. Perspective correction if doc_corners is not None: image = self.four_point_transform(image, doc_corners) # 3. Image enhancement image = self.enhance_document(image) # 4. OCR result = self.ocr.ocr(image, cls=True) return { 'text': self._extract_text(result), 'words': self._extract_words_with_positions(result), 'corrected': doc_corners is not None } def detect_document(self, image: np.ndarray) -> np.ndarray | None: gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) blur = cv2.GaussianBlur(gray, (5, 5), 0) edges = cv2.Canny(blur, 75, 200) dilated = cv2.dilate(edges, np.ones((3, 3)), iterations=1) contours, _ = cv2.findContours(dilated, cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE) contours = sorted(contours, key=cv2.contourArea, reverse=True)[:5] for contour in contours: peri = cv2.arcLength(contour, True) approx = cv2.approxPolyDP(contour, 0.02 * peri, True) if len(approx) == 4: area_ratio = cv2.contourArea(approx) / (image.shape[0] * image.shape[1]) if area_ratio > 0.2: return approx.reshape(4, 2) return None def four_point_transform(self, image: np.ndarray, pts: np.ndarray) -> np.ndarray: rect = self._order_points(pts) tl, tr, br, bl = rect width = int(max(np.linalg.norm(br - bl), np.linalg.norm(tr - tl))) height = int(max(np.linalg.norm(tr - br), np.linalg.norm(tl - bl))) dst = np.array([ [0, 0], [width - 1, 0], [width - 1, height - 1], [0, height - 1] ], dtype='float32') M = cv2.getPerspectiveTransform(rect.astype('float32'), dst) return cv2.warpPerspective(image, M, (width, height)) 

Result: even at tilt up to 30°, text becomes horizontal, which radically improves recognition quality.

What about shadows and glare?

The next problem is uneven lighting. Real photos often have shadows from fingers, binding, or glare on laminated cards. For shadows, we apply CLAHE (Contrast Limited Adaptive Histogram Equalization) in LAB color space. This adaptively equalizes brightness locally without creating artifacts. For glare, we use inpainting — detect overexposed pixels (value >250 in any RGB channel) and interpolate them from neighboring areas.

def enhance_document(self, image: np.ndarray) -> np.ndarray: lab = cv2.cvtColor(image, cv2.COLOR_BGR2LAB) l, a, b = cv2.split(lab) clahe = cv2.createCLAHE(clipLimit=4.0, tileGridSize=(16, 16)) l_enhanced = clahe.apply(l) enhanced = cv2.merge([l_enhanced, a, b]) enhanced = cv2.cvtColor(enhanced, cv2.COLOR_LAB2BGR) kernel = np.array([[0, -1, 0], [-1, 5, -1], [0, -1, 0]]) sharpened = cv2.filter2D(enhanced, -1, kernel) return sharpened 

We also add a light sharpen to compensate for blur from handheld shots. Our experience shows that the combination of CLAHE + sharpen reduces CER by 3–5% compared to raw images.

Full pipeline for document photo processing

Here is our production pipeline (Python code with OpenCV and PaddleOCR). The functions are combined into one class. We use PaddleOCR with Russian language and angle classification (use_angle_cls=True). On GPU, processing one frame takes ~200 ms. For mobile devices, a lightweight model can be substituted.

Comparison of OCR engines on real photos

PaddleOCR offers the best balance of accuracy and speed for Russian-language documents. Without our preprocessing pipeline, any OCR engine shows 5–10% worse CER. Using the combination of OpenCV and PaddleOCR, we achieve CER <5% for passports — 1.5–3 times better than without preprocessing. Our pipeline processes a frame 2x faster than standard solutions with similar quality.

Implementation process

Turnkey implementation includes:

1. Analysis: examining document types, shooting conditions, accuracy requirements, selecting reference images.
2. Design: stack selection (OpenCV, PyTorch/TensorFlow, OCR engine), pipeline architecture, container configuration.
3. Development: writing detection, correction, OCR modules; integration with backend via REST API.
4. Testing: on your dataset (minimum 500 images), measuring CER, latency p99, debugging edge cases.
5. Deployment: containerization (Docker), deployment on server or edge device, metric monitoring.

ocr = PaddleOCR(use_angle_cls=True, lang='ru', use_gpu=True, det_db_thresh=0.3, det_db_box_thresh=0.5)