Industrial Export of Scraped Data: CSV, JSON, and REST API
Scraped data is rarely needed in raw form. An analyst asks for CSV for pivot tables, a developer needs JSON with filtering, and an accounting system expects automatic webhook notifications. Typical situation: the parser collected 50,000 products, but serving them as a single file is impossible—the server runs out of memory, Excel won't open. We solve this with streaming and proper API design. Streaming CSV processes data 10 times faster than loading the entire file into memory and allows exporting catalogs up to 500,000 rows without failures. Streaming reduces server load by 10x, saving up to 50% on server infrastructure costs. Contact us to choose the optimal export format for your task.
Raw data often contains duplicates, invalid records, and junk. Before export, we perform cleaning and normalization to ensure only high-quality data comes out. Over years of practice, we have implemented more than 50 export integrations for projects with millions of records.
How to Build Fault-Tolerant Export for Millions of Records?
For large data volumes, export architecture is critical. We use database cursors, chunked queries with pagination, and backpressure control. In relational databases (PostgreSQL, MySQL), we apply server-side cursors—this avoids loading the entire result into memory. For NoSQL (MongoDB), we use allowDiskUse() and iterators. Typical pattern: read 1000 records at a time, send chunks to the stream. If the client is slow, we pause reading to avoid memory overflow.
Streaming CSV Export for Large Volumes
For CSV with more than 10,000 rows, we don't generate the whole file in memory—we use streaming. Example with Flask using Response and stream_with_context:
import csv import io from flask import Response, stream_with_context def export_csv(site_id: int, filters: dict): def generate(): output = io.StringIO() writer = csv.DictWriter(output, fieldnames=[ 'id', 'name', 'price', 'currency', 'url', 'in_stock', 'scraped_at' ]) writer.writeheader() yield output.getvalue() output.truncate(0); output.seek(0) for product in stream_products(site_id, filters): writer.writerow(product) yield output.getvalue() output.truncate(0); output.seek(0) return Response( stream_with_context(generate()), mimetype='text/csv', headers={'Content-Disposition': 'attachment; filename=export.csv'}, ) This approach allows processing files of any size without increasing memory consumption. We apply it in projects with catalogs of 200,000+ items—guaranteeing stable performance under load.
Why HMAC-Signed Webhook Is Safer than Plain POST?
Sending data via webhook without authentication is a vulnerability: anyone can fake a request to your system. We add a signature using HMAC-SHA256 so the recipient can verify the sender. We use httpx:
import httpx, hashlib, hmac def send_webhook(url: str, secret: str, payload: dict): body = json.dumps(payload).encode() sig = hmac.new(secret.encode(), body, hashlib.sha256).hexdigest() httpx.post(url, content=body, headers={ 'Content-Type': 'application/json', 'X-Signature-SHA256': f'sha256={sig}', }, timeout=10) The same secret is stored by sender and receiver—this eliminates request forgery. The signature is computed from the body, so any data change breaks verification. For critical data, you can add AES body encryption.
Export Format Comparison
| Characteristic | CSV (streaming) | JSON API | Webhook |
|---|---|---|---|
| Purpose | Reports, analytics | System integration | Triggers, alerts |
| Data volume | Any (stream) | Up to 10,000 records per page | Single event |
| Security | Basic auth | API key + rate-limit | HMAC signature |
| Development complexity | 1 day | 1–2 days | 2–3 days (with filters) |
| Flexibility | Fixed fields | Field selection, pagination | Custom payload |
How to Filter Data in REST API?
REST API for scraped data is implemented with full filtering, pagination, and field selection. Example endpoint:
GET /api/v1/scraped-products?site_id=7&in_stock=true&fields=name,price,url&page=1&per_page=100 Response:
{ "data": [ { "name": "Nike Air Max sneakers", "price": 89.99, "url": "https://..." } ], "meta": { "page": 1, "per_page": 100, "total": 4823 } } Authentication uses an API key in the X-API-Key header. Access is restricted by IP and rate-limited (100 requests/min). Sorting by any field and aggregation (sum, average) for analytical queries are supported.
When to Use Each Export Format?
| Format | Use case | Example |
|---|---|---|
| CSV | Analytics, reports, Excel import | Product catalog of 200,000 rows |
| JSON API | Integration with internal systems | Real-time data retrieval via API |
| Webhook | Automation notifications | Alert when a new product appears |
What's Included in Typical Implementation (Deliverables)
- Export source code with comments
- API documentation in OpenAPI (Swagger) format
- Deployment scripts (Docker, docker-compose)
- Integration examples in Python and JavaScript
- 30-day support after delivery
- Optional integration with Sentry for error monitoring
Timelines and Cost
Basic implementation of CSV and JSON API with authentication—1–2 working days. Adding webhook with signature, filters, and field selection—another day. Cost is calculated individually after task analysis. Contact us to discuss details. Order a turnkey export implementation—get a working solution in 2–3 days.