Automatic Product Price Updates from External Sources

Implementing Automatic Product Price Updates from External Sources

Online stores working with multiple suppliers or using dynamic pricing face the same problem: catalog prices become outdated faster than they can be manually updated. The solution is to build an automatic pipeline that pulls current data from external sources and updates prices in the database without operator involvement.

Price Sources and Data Retrieval Methods

External price sources can include:

• Supplier Price Lists — CSV/Excel files via HTTP or FTP
• Supplier APIs — REST or SOAP with token authorization
• YML Feeds — Yandex.Market format containing <price> and <oldprice>
• Google Merchant Feed — XML with g:price field
• Page Scraping — fallback when no API or feed is available

For each source type, a separate adapter implementing a common interface is needed:

interface PriceSourceInterface
{
    /** @return array<string, float> [sku => price] */
    public function fetch(): array;
}

CSV via HTTP Adapter

class CsvHttpPriceSource implements PriceSourceInterface
{
    public function __construct(
        private string $url,
        private int    $skuColumn,
        private int    $priceColumn,
        private string $delimiter = ';',
    ) {}

    public function fetch(): array
    {
        $stream = fopen($this->url, 'r');
        $prices = [];
        $header = fgetcsv($stream, 0, $this->delimiter); // skip header

        while ($row = fgetcsv($stream, 0, $this->delimiter)) {
            $sku   = trim($row[$this->skuColumn]);
            $price = (float) str_replace(',', '.', $row[$this->priceColumn]);
            if ($sku && $price > 0) {
                $prices[$sku] = $price;
            }
        }
        fclose($stream);
        return $prices;
    }
}

Scheduler Architecture

Price updates are background tasks. Standard Laravel approach: Artisan command + scheduler + Queue.

Cron (every N minutes)
  └─> SchedulePriceUpdateCommand
        └─> PriceUpdateJob (queued)
              └─> PriceSourceFactory::make($source)
              └─> PriceUpdater::apply($prices)

Dispatcher Command:

class SchedulePriceUpdateCommand extends Command
{
    protected $signature = 'prices:update {--source=all}';

    public function handle(PriceSourceRepository $repo): void
    {
        $sources = $this->option('source') === 'all'
            ? $repo->getActive()
            : [$repo->find($this->option('source'))];

        foreach ($sources as $source) {
            PriceUpdateJob::dispatch($source)->onQueue('prices');
        }
    }
}

In app/Console/Kernel.php:

$schedule->command('prices:update')->everyThirtyMinutes();

Update Logic with Garbage Protection

Blindly writing any price from a feed is risky. Checks are needed:

class PriceUpdater
{
    private const MAX_CHANGE_RATIO = 0.5;

    public function apply(array $prices, PriceSource $source): UpdateResult
    {
        $updated = $skipped = $errors = 0;

        foreach ($prices as $sku => $newPrice) {
            $product = Product::where('sku', $sku)->first();
            if (!$product) { $skipped++; continue; }

            $oldPrice = $product->price;
            if ($oldPrice > 0) {
                $ratio = abs($newPrice - $oldPrice) / $oldPrice;
                if ($ratio > self::MAX_CHANGE_RATIO) {
                    Log::warning("Price anomaly: $sku $oldPrice -> $newPrice");
                    $errors++;
                    continue;
                }
            }

            $product->update([
                'price'            => $newPrice,
                'price_updated_at' => now(),
                'price_source_id'  => $source->id,
            ]);
            $updated++;
        }

        return new UpdateResult($updated, $skipped, $errors);
    }
}

Multiple Sources and Priorities

When a product appears in multiple feeds, a conflict resolution strategy is needed:

• MIN — use minimum price (aggressive pricing)
• PRIMARY — first source takes priority, others as fallback
• LAST_UPDATED — price from last updated feed

Source configuration in database:

CREATE TABLE price_sources (
    id         serial PRIMARY KEY,
    name       varchar(100),
    type       varchar(30),   -- csv_http | api | yml | merchant
    config     jsonb,         -- url, credentials, column mapping
    priority   smallint DEFAULT 10,
    strategy   varchar(20) DEFAULT 'primary',
    active     boolean DEFAULT true,
    updated_at timestamptz
);

Update via Supplier API

If supplier provides REST API with pagination:

class ApiPriceSource implements PriceSourceInterface
{
    public function fetch(): array
    {
        $client = new \GuzzleHttp\Client(['base_uri' => $this->baseUrl]);
        $prices = [];
        $page   = 1;

        do {
            $response = $client->get('/v2/prices', [
                'headers' => ['Authorization' => 'Bearer ' . $this->token],
                'query'   => ['page' => $page, 'per_page' => 500],
            ]);
            $data = json_decode($response->getBody(), true);

            foreach ($data['items'] as $item) {
                $prices[$item['article']] = (float) $item['price_rub'];
            }

            $page++;
        } while ($data['has_more']);

        return $prices;
    }
}

Implementation Timeline

• Basic pipeline (one CSV source, scheduler, database update) — 2–3 days
• Support for multiple source types + priorities — +2 days
• Dashboard with update history and anomaly alerts — +2 days

Monitoring and Alerts

After each update cycle, write to price_update_logs table:

source_id | total_fetched | updated | skipped | errors | duration_ms | created_at

When errors / total_fetched > 0.05 (more than 5% anomalies) — send notification to Slack or email via Laravel Notification. This allows detecting broken feeds before customers see incorrect prices.