Incremental Product Import Implementation: Only Changes

Imagine you manage an online store with a catalog of 300,000 products. Every night, a full import runs—servers are at 100% load, the database locks up, and customers complain about outdated prices and order processing delays. Yet, in reality, no more than 5% of the assortment actually changes; the other 95% is reloaded pointlessly. This situation is familiar to many e-commerce projects. This optimization can save over $2,500 per month in server and database costs for mid-size catalogs, and for larger ones, savings can exceed $6,000 per month.

We solve this problem with incremental product import: a catalog synchronization method that updates only changed items. Incremental product import is 10 times faster than full reloads and reduces server load by 80%. Over several years, we have implemented this approach for 15 projects—from small shops to marketplaces with catalogs up to 500,000 SKUs. Our expertise ensures you get a working solution quickly. Our certified implementation process guarantees a reliable solution within your timeline.

Incremental import (delta import) is a synchronization method that updates only changed records. This import optimization reduces server load by 80% and cuts import time by 10 times compared to full reloads. For instance, on a project with a 200,000 SKU catalog, we reduced sync time from 3 hours to 18 minutes by using timestamps and hash comparison for reliability. Each product update is processed individually, resulting in substantial resource savings.

Choosing a Change Detection Strategy

The choice depends on the data source capabilities. The table below compares the main approaches:

Let's examine each method in detail.

Timestamp

The most common approach—the supplier supports a filter by modification date: GET /api/products?updated_after=2024-01-15T10:00:00Z. The system remembers the last successful sync time and passes it in the next request. This timestamp-based import method is simple but may miss changes that occur during processing. Therefore, we always record the sync start time, not the end.

Cursor / Change Log

The supplier maintains a change log with an incrementing ID. More reliable than timestamp: no changes are missed during processing. Example: GET /api/changes?since_id=48291. Cursor-based import is more reliable and suitable for APIs with high update frequency.

Hash Comparison

When the source does not support change filtering—we compare the hash of the data row:

$hash = md5(serialize([ $row['price'], $row['qty'], $row['name'], $row['description'] ])); 

The row is processed only if the hash changed. This method works well when data comes as full export, but we want to process only changed records.

Diff Files

The supplier publishes an hourly diff file instead of full price list:

<changes> <updated id="SKU-123"><price>4990</price><qty>15</qty></updated> <updated id="SKU-456"><qty>0</qty></updated> <deleted id="SKU-789"/> <created id="SKU-999"><!-- full data --></created> </changes> 

This method is most accurate but requires supplier support.

Implementation Steps

1. Analyze your data source capabilities (API, export files).
2. Choose the appropriate change detection strategy (timestamp, cursor, hash, or diff).
3. Implement the state tracker in your database.
4. Build the incremental import pipeline.
5. Add deletion detection using anti-joins.
6. Implement double-run protection via distributed lock.
7. Test with your catalog and deploy.

How We Implement Incremental Import

State Tracker

Sync state is stored in the database. We use a dedicated table:

CREATE TABLE import_sync_state ( source_id int PRIMARY KEY REFERENCES import_sources(id), last_sync_at timestamptz, last_cursor varchar(200), last_change_id bigint, items_synced bigint DEFAULT 0, updated_at timestamptz DEFAULT now() ); 

We record the sync start time, not the end. If new changes appear during processing, they will be picked up in the next cycle.

Incremental Import Pipeline

The core class that performs synchronization:

class IncrementalImportJob implements ShouldQueue { public function handle( SyncStateManager $state, SupplierApiClient $client, IncrementalProductSync $sync, ): void { $since = $state->getLastSyncAt($this->sourceId); $state->markSyncStarted($this->sourceId); $stats = ['created' => 0, 'updated' => 0, 'deleted' => 0, 'skipped' => 0]; foreach ($client->fetchUpdatedSince($since) as $item) { $result = $sync->process($item, $this->sourceId); $stats[$result]++; } $state->markSyncCompleted($this->sourceId); $this->logResult($stats); } } 

Detecting Deleted Items

If the source does not send explicit deletion signals, we use an anti-join via a temporary table (for catalogs from 50,000 SKUs):

CREATE TEMP TABLE current_import_skus (sku varchar(100)); COPY current_import_skus FROM STDIN; UPDATE products SET deleted_at = now() WHERE source_id = $1 AND deleted_at IS NULL AND sku NOT IN (SELECT sku FROM current_import_skus); DROP TABLE current_import_skus; 

Double-Run Protection

We use a distributed lock via cache (Redis). If a synchronization is already running for a given source, a new run is skipped. The lock TTL is set to 1 hour, which is enough for most catalogs. This prevents duplicate processing and conflicts.

What's Included in the Work

• Development of a state tracker for sync state storage
• Implementation of the chosen change detection strategy (timestamp, cursor, hash, diff)
• Mechanism for detecting and handling deleted items
• Double-run protection via lock
• Testing on catalogs up to 500,000 SKUs
• Documentation for deployment and monitoring

Estimated Timelines

Exact timelines depend on the supplier API complexity and catalog size. Contact us for a project assessment—we'll prepare a custom proposal and show how incremental import can reduce your infrastructure costs by up to 80%. Request incremental import implementation and get a free consultation on optimizing your catalog sync.