Why AI formulation optimization fails at pilot
AI formulation optimization stumbles not on the model, but on the input data.
Raw material certificates, incoming inspection cards, warehouse balances, and procurement terms live in different systems, while the calculation uses an averaged ingredient card with a "typical" composition.
The algorithm finds a cheap combination, but on the very first batch, fat content and moisture come in different — and the formulation goes into rework.
The second failure zone is the technologist's manual tables.
Composition constraints, permissible substitutions, and the order of addition live in Excel files with versions "final_final_2": one rule is duplicated across three workbooks with different values, and some thresholds are kept in the lead specialist's head.
Until this is consolidated into a single set of rules, automation reproduces private experience rather than a technological regulation.
Regulatory documents are more complicated. Requirements for mass fractions, acidity, content of additives and allergens are written as text, without dividing into hard limits and target guidelines.
A model that treats all conditions as strict finds no solution at all; a model that treats them as soft produces a formulation that inspection will not pass.
Hence the failure in batch reproducibility: raw materials change from delivery to delivery, and the formulation that was optimal yesterday yields a different product output on new certificates.
Without recalculation from actual values and a protocol by which the decision is repeated, it is impossible to prove the stability of the result.
We break down how this is closed: a unified data layer on raw materials, formalized composition constraints, and batch-level formulation recalculation.
Measurable effects: what changes in production
After implementation, we do not ask "has it become more convenient." We count four numbers on the same line: the share of formulations that went to repeat review, the time to select a composition for specified cost and organoleptic properties, the spread of key indicators between batches, and the share of rejects that were previously closed by batch rework.
We take measurements before the start and after — on the same raw materials and the same equipment, otherwise there is nothing to compare.
| Metric | Before | After |
|---|---|---|
| Share of formulation revisions | three to four iterations per launch | one to two, more often a dosage adjustment |
| Time to select composition | weeks of coordination between laboratory and line | hours per variant, a day per package of variants |
| Spread between batches | some batches go outside tolerance | control points stay within tolerance |
| Class of rejects | composition passed the laboratory but was not reproduced on the line | gone: raw material substitution is recalculated before the run |
The spread shrinks not because raw materials have become more even, but because the model works with corridors.
For each delivery, an allowable range for protein, fat, and moisture is set, and the composition is recalculated for the actual batch analysis, not for the warehouse average.
This also eliminates the most painful failure — a formulation that holds in the laboratory and falls apart at scale when unstable raw materials arrive in a wave.
The accuracy of these numbers rests on experimental designs: we calculate mixture models and response surfaces in JMP, so the contribution of an ingredient to losses during heat treatment or to texture is visible before the run, not after.
The technologist approves the variant and remains the owner of the formulation — the system offers a calculation and boundaries, not a decision.
Three approaches to building a model and their limitations
Formulation is a constrained problem: meet composition standards, preserve taste, texture, and shelf life. Deterministic calculation on balance equations and linear optimization gives a reproducible answer where composition and target indicators are formalized.
But it does not know how a substitution of a raw material batch will behave or why a similar formulation's texture dropped.
A model on historical data learns from test and production records and captures nonlinear relationships that cannot be put into formulas. The price is requirements for data volume and cleanliness and no guarantees on hard constraints.
Search over a database of past decisions helps when data is scarce, but it does not go beyond what has already been tried.
Options and their limitations
| Option | When suitable | Limitations |
|---|---|---|
| Deterministic calculation | Standards, composition, and target indicators are formalized | Does not describe organoleptic properties and behavior when raw materials are substituted |
| Model on historical data | Test and production records have been accumulated | Requires data volume and cleanliness; hard constraints are specified separately |
| Hybrid approach | Some constraints are hard, and there is already enough data | Calculation and model need to be synchronized; setup is more expensive |
| Search over a database of past decisions | Little data, but there is an archive of analogs | Does not go beyond already tried formulations |
The mode is selected according to data maturity: with a thin archive, one starts with search over past decisions; with accumulated records, the model is connected; the hybrid is the target state.
At the pilot, both loops are assembled on the same data, and it is visible which one is stable for your product range.
Implementation stages: from data audit to handover
The implementation of a formulation optimization system proceeds in stages, and the order here is not arbitrary: each next step relies on the result of the previous one, and we do not proceed until it is accepted.
This removes the typical risk when the calculation is ready but technologists do not trust the output and continue selecting the composition manually.
- Data audit. We analyze what has been accumulated on raw materials, batches, and laboratory analyses, and how comparable the reference data is. The output is a list of gaps that we close before the start.
- Solution design. We fix which indicators are included in the calculation and where each comes from, and what the system returns to the technologist: composition variants and expected characteristics.
- Constraint setup. We translate technological and regulatory frameworks into machine rules: permissible component shares, compatibility, total limits by indicators, priority of hard frameworks over soft ones.
- Validation on historical batches. We run the system over past production runs and compare what it proposes with the actual composition. We count the share of decisions that fell within tolerances and analyze discrepancies separately.
- Technologist training. We show how to read the output, change frameworks for new raw materials, and send feedback into the system rather than into email.
- System handover. We hand over the constraint configuration, documentation, access credentials, and the data update regulation. After that, the team works independently.
At each step, you retain a verifiable artifact: a data list, a calculation design, a set of rules, a report on past batches.
Configurations, data schemas, and revision regulations
A formulation optimization solution lives longer than the team that implemented it. Therefore, we hand over not only a working service but also everything needed for independent operation: model settings, composition constraints, and integration points with accounting systems. Below is the package that remains with the customer.
- Solution configuration. The set of input features, permissible ranges for raw material components, weights of the objective function, and separate calculation profiles for different product lines.
- Data schema. Where raw material batches, compositions, and laboratory analysis results come from, how they are brought to a unified form, and where the version history is stored.
- Formulation access rules. Roles of technologist, analyst, and business unit manager; separation of rights for reading, editing, and approval; action log for each formula.
- Revision regulation. Frequency, triggers (supplier change, quality deviation), who initiates, how the decision is recorded, and how a rollback to the previous version is performed.
- Technical documentation. Model description, quality metrics on a holdout sample, known limitations, and cases where technologist review is mandatory.
- Instructions for technologists. How to read recommendations, what to do in case of conflict with regulatory requirements, and how to pass feedback to the model.
We accept each item together with the customer's working group: by project closure, you have not only the service but also the documents by which it can be developed in-house.
Case study: revising formulations without stopping the line
Before implementation, formulation revision ran as a separate project: the technologist collected data on raw material batches, manually recalculated component ratios, and sent trial mixes to the laboratory.
One cycle from task to approved version took 18–25 business days, and some trials were rejected for moisture and organoleptic properties — the deviation share reached 12%.
While trials were running, the line produced product according to the previous formulation, and the warehouse accumulated stock for several composition versions at once.
What was changed
We assembled a digital product model: component specifications, permissible corridors by composition and indicators, constraints from Codex Alimentarius and internal regulations.
Ratio selection is calculated immediately across all raw material batches; the laboratory checks only versions that passed the constraints. The formulation is versioned, so rolling back to the previous version is switching a link, not recalculating from scratch.
Metrics before and after
The line was not stopped throughout implementation: trial mixes ran in planned windows, and the transition to a new formulation happened during normal changeover. The reduction in revisions had a side effect: fewer raw material write-offs when changing composition versions.
What to do if regulatory documents contradict the data?
A discrepancy between the process sheet and batch history is not an export error but a working situation. The document describes the permitted corridor: mass fractions, permissible substitutes, limits by safety indicators.
Batch history shows what actually happened in the shop: raw materials arrived with different moisture, the line held a different temperature, a batch went out with a viscosity deviation.
A single source of truth cannot be chosen forever: if you blindly trust the document, the system proposes formulations that are not reproducible with current raw materials, while trusting only the data replicates deviations that once passed through oversight.
The selection criterion is the class of discrepancy. Within the document's corridor, a deviation is accepted as normal and taken into account during selection.
Going beyond the limit for a safety indicator blocks the calculation completely, no matter how often such batches are produced. Intermediate cases go into a queue for technologist review, indicating the parameter and the magnitude of the exceedance.
Each decision is recorded in a discrepancy log: parameter, magnitude, selected priority, who confirmed. This provides reproducibility — a disputed formulation can be retrieved and explained why it looks the way it does.
Solution limitations and technical concerns
A formulation optimization model relies on batch history and laboratory measurements, not on process physics. Its accuracy rests on data quality: if raw materials were weighed "by eye" and deviations were recorded after the fact, the recommendation will rely on noise.
The first stage is profiling: we find gaps, bring units and moisture to a unified basis, mark batches with manual edits, and exclude them from training.
Some factors do not enter the model in principle — supplier change, line wear, operator decisions. We do not hide this behind precision to tenths: the solution shows the recipe's sensitivity to deviations and provides ranges, not a single "ideal" value. A trial batch in the laboratory remains a mandatory step — this is built into the process.
We discuss the responsibility zone before the start. Before handover, we are responsible for data correctness, calculation reproducibility, and model documentation; after, the customer's process service is responsible for the operating regulation. We fix the boundary in writing.
What to do if batch history is incomplete?
We restore gaps using indirect signs, narrow the search area, and mark zones where the recommendation relies on an assumption. An honest range is more useful than a confident answer on incomplete data.
Who is responsible for the recipe after implementation?
The enterprise technologist approves the formula — the model has no right to release a batch. We are responsible for calculation and support, and the quality service is responsible for acceptance; this is specified in the regulation.
Let's discuss the task: what is needed from you at the start
To come back with an estimate and work plan rather than general wording, we need a compact set of input data for assessment. Below is what actually affects the scope of the first iteration of the formulation optimization model. Some things may not be in digital form — tell us, and we will advise how to assemble the minimum.
- Current formulations in structured form — composition, dosages, permissible corridors for component substitution; an export from the accounting system will work.
- Analyses for raw materials and finished products — at least for the indicators that the technologist controls in each batch.
- Technological constraints of the area: order of addition, processing time, temperature modes, equipment that cannot be changed.
- What we optimize — stability of an indicator, product yield, share of expensive components; the goal determines the model design.
- Acceptance criteria — how you recognize the result as working: tolerance by indicator, share of formulations that passed technologist review.
- Pilot boundaries on one line — one product group, one area, observation period; we expand after the pilot converges.
- Responsible technologist on your side — the person who confirms constraints and accepts the result.
This is enough to calculate labor costs and come back with a stage-by-stage work plan. Describe the task — area, products, what is not adding up now. We will discuss the task concretely: the format of a pilot on one line and the acceptance procedure.
