Automatic Product Categorization (AI)

Implementing Automatic Product Categorization (AI)

When a catalog is formed from multiple sources—suppliers, XML feeds, manual entry—products arrive with different data structures and often without proper categories. Assigning them to sections manually with hundreds of new products daily is unrealistic.

Automatic categorization through a language model works differently than rules or regex. The model understands meaning, not just keywords: "wireless earbuds with ANC noise cancellation" and "TWS earbuds noise cancelling" will land in the same category without explicit mapping.

Two Categorization Modes

Mode 1: classification into a predefined category tree. Pass the model a list of allowed categories and ask it to choose the most suitable. Deterministic result, easy to validate.

Mode 2: suggesting new categories. The model proposes a category name based on product semantics. Used when initially building a catalog or identifying "orphaned" products.

In practice, the first mode is needed with fallback to the second for products that don't fit any category.

Classification into Existing Tree

interface CategoryTree {
  id: string;
  name: string;
  path: string; // "Electronics / Audio / Headphones"
  children?: CategoryTree[];
}

async function classifyProduct(
  product: RawProduct,
  categories: CategoryTree[]
): Promise<{ categoryId: string; confidence: number; reasoning: string }> {
  // Flat list of paths for prompt
  const categoryList = flattenCategories(categories)
    .map((c) => `${c.id}: ${c.path}`)
    .join("\n");

  const prompt = `
Classify this product into the most appropriate category.

Product:
- Name: ${product.name}
- Description: ${product.description?.slice(0, 300) ?? "—"}
- Brand: ${product.brand ?? "—"}
- Supplier category: ${product.supplierCategory ?? "—"}
- Attributes: ${JSON.stringify(product.attributes ?? {}).slice(0, 200)}

Available categories (id: path):
${categoryList}

Return JSON:
{
  "categoryId": "the id from the list above",
  "confidence": 0.0-1.0,
  "reasoning": "one sentence why"
}

If no category fits well, use the closest parent category and set confidence below 0.5.
`.trim();

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
    response_format: { type: "json_object" },
    temperature: 0,
  });

  return JSON.parse(response.choices[0].message.content!);
}

temperature: 0—for classification tasks, reproducibility is needed, not creativity.

Batch Processing with Smart Prompts

For token economy and speed—classify several products in one request:

async function classifyBatch(
  products: RawProduct[],
  categories: CategoryTree[]
): Promise<Map<string, ClassificationResult>> {
  const categoryList = flattenCategories(categories)
    .map((c) => `${c.id}: ${c.path}`)
    .join("\n");

  const productList = products
    .map(
      (p, i) =>
        `[${i}] "${p.name}"` +
        (p.brand ? ` by ${p.brand}` : "") +
        (p.supplierCategory ? ` (supplier: ${p.supplierCategory})` : "")
    )
    .join("\n");

  const prompt = `
Classify each product into one of the categories. Return JSON array.

Categories:
${categoryList}

Products:
${productList}

Return: [{"index": 0, "categoryId": "...", "confidence": 0.0-1.0}, ...]
`.trim();

  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: prompt }],
    response_format: { type: "json_object" },
    temperature: 0,
    max_tokens: 1000,
  });

  const results: Array<{ index: number; categoryId: string; confidence: number }> =
    JSON.parse(response.choices[0].message.content!).results ?? [];

  const map = new Map<string, ClassificationResult>();
  for (const r of results) {
    const product = products[r.index];
    if (product) {
      map.set(product.id, { categoryId: r.categoryId, confidence: r.confidence });
    }
  }

  return map;
}

10–20 products per request—a reasonable batch. More—the prompt becomes too long and quality drops.

Worker with Queue

const categorizationWorker = new Worker(
  "categorization",
  async (job) => {
    const { productIds } = job.data;
    const products = await db.products.findMany({
      where: { id: { in: productIds } },
    });
    const categories = await db.categories.findAll({ active: true });

    const results = await classifyBatch(products, categories);

    for (const [productId, result] of results) {
      await db.products.update({
        where: { id: productId },
        data: {
          categoryId: result.confidence >= 0.7 ? result.categoryId : null,
          suggestedCategoryId: result.categoryId,
          categorizationConfidence: result.confidence,
          categorizationStatus:
            result.confidence >= 0.7 ? "auto_assigned" : "needs_review",
          categorizedAt: new Date(),
        },
      });
    }
  },
  { connection: redisConnection, concurrency: 3 }
);

Products with confidence < 0.7 go to review queue—a manager assigns their category, and this additionally trains the system via few-shot examples.

Few-Shot Learning from Catalog Examples

When a manager manually corrects a category, this is valuable data. Accumulate it and add to the prompt:

async function getExamplesForCategory(categoryId: string, limit = 5): Promise<string> {
  const examples = await db.products.findMany({
    where: { categoryId, categorizationStatus: "manually_confirmed" },
    select: { name: true, brand: true },
    take: limit,
  });

  if (examples.length === 0) return "";

  return `\nExamples of products in this category: ${examples.map((e) => `"${e.name}"`).join(", ")}`;
}

After 2–3 weeks of system operation with reviews, automatic classification accuracy in a specific catalog grows to 90%+—the model sees real catalog examples.

Quality Monitoring

SELECT
  categorization_status,
  AVG(categorization_confidence) as avg_confidence,
  COUNT(*) as count
FROM products
WHERE categorized_at > NOW() - INTERVAL '7 days'
GROUP BY categorization_status;

If the share of needs_review is growing—possibly new product types appeared that are not covered by the current category tree. This signals the need to expand the catalog.