Product Import from YML Feed (Yandex Market Format)

Implementing Product Import from YML Feed (Yandex.Market Format)

YML (Yandex Market Language) is XML schema that suppliers prepare for Yandex.Market. For online store it's invaluable source: structured data with prices, stocks, characteristics and images, already validated by supplier.

YML Feed Structure

<?xml version="1.0" encoding="UTF-8"?>
<yml_catalog date="2024-01-15 10:00">
  <shop>
    <name>Supplier Shop</name>
    <currencies>
      <currency id="RUR" rate="1"/>
    </currencies>
    <categories>
      <category id="10">Electronics</category>
      <category id="11" parentId="10">Smartphones</category>
    </categories>
    <offers>
      <offer id="ABC-123" available="true">
        <name>Smartphone Example Pro 128GB</name>
        <price>29990</price>
        <oldprice>34990</oldprice>
        <currencyId>RUR</currencyId>
        <categoryId>11</categoryId>
        <picture>https://cdn.supplier.ru/images/ABC-123_1.jpg</picture>
        <vendor>Example</vendor>
        <vendorCode>PRO128</vendorCode>
        <description><![CDATA[Detailed description...]]></description>
        <param name="Screen" unit="inch">6.7</param>
      </offer>
    </offers>
  </shop>
</yml_catalog>

Streaming Parser

Large feeds can exceed 500 MB — SimpleXML::load() kills PHP. Parse via XMLReader:

class YmlFeedParser
{
    public function parse(string $url): iterable
    {
        $reader = new \XMLReader();
        $reader->open($url, null, LIBXML_NOERROR);

        // Collect categories first (they're at start)
        $categories = $this->parseCategories($reader);

        // Then iterate offers
        while ($reader->read()) {
            if ($reader->nodeType === \XMLReader::ELEMENT && $reader->name === 'offer') {
                $node = new \SimpleXMLElement($reader->readOuterXml());
                yield $this->parseOffer($node, $categories);
            }
        }
        $reader->close();
    }

    private function parseOffer(\SimpleXMLElement $node, array $categories): array
    {
        $categoryId   = (string) $node->categoryId;
        $categoryPath = $this->buildCategoryPath($categoryId, $categories);

        return [
            'sku'           => (string) $node['id'],
            'available'     => ((string) $node['available']) === 'true',
            'name'          => (string) $node->name,
            'price'         => (float)  $node->price,
            'old_price'     => $node->oldprice ? (float) $node->oldprice : null,
            'currency'      => (string) $node->currencyId,
            'category_path' => $categoryPath,
            'images'        => array_map(fn($pic) => (string) $pic, $node->picture),
            'vendor'        => (string) $node->vendor,
            'description'   => (string) $node->description,
        ];
    }
}

Validation and Caching

class YmlFeedValidator
{
    public function validate(string $url): ValidationResult
    {
        // Check DTD
        libxml_use_internal_errors(true);
        $dom = new \DOMDocument();
        $dom->load($url);
        $xmlErrors = libxml_get_errors();
        libxml_clear_errors();

        foreach ($xmlErrors as $error) {
            $errors[] = "XML error at line {$error->line}: {$error->message}";
        }

        // Check required elements
        $xpath = new \DOMXPath($dom);
        if (!$xpath->query('//offers/offer')->length) {
            $errors[] = 'No offers found in feed';
        }

        return new ValidationResult(empty($errors), $errors);
    }
}

Import Job

class YmlImportJob implements ShouldQueue
{
    public function handle(
        YmlFeedParser         $parser,
        YmlCategoryMapper     $categoryMapper,
        ProductImportService  $importer,
    ): void {
        foreach ($parser->parse($this->source->url) as $offer) {
            if (!$offer['available']) {
                $importer->markUnavailable($offer['sku'], $this->source->id);
                continue;
            }

            $siteCategoryId = $categoryMapper->resolve(
                $offer['category_id'],
                $offer['category_path'],
                $this->source->id
            );

            $importer->upsert(array_merge($offer, [
                'site_category_id' => $siteCategoryId,
                'source_id'        => $this->source->id,
            ]));
        }
    }
}

Implementation Timeline

• Streaming parser YML, basic import prices/stocks/descriptions — 2 days
• Category mapping, currency conversion, images — +1 day
• Feed validation, caching, scheduler, logging — +1 day