AI formulation optimization: where manual composition selection breaks down
A formulation of a dozen components with five to seven dosage variants of each yields hundreds of thousands of combinations. A lab tests dozens of samples per round; the rest remains unexplored — and the first acceptable composition goes into production, not the best one.
Manual composition selection is limited not by the chemist's qualifications, but by the throughput of component enumeration.
The search proceeds sequentially: one ingredient is changed, the rest are fixed, and the target characteristic is observed.
This loses pairwise and three-way interactions — components that are neutral individually, together produce the desired effect on viscosity, adhesion, or curing time. The more functional additives in a formulation, the more costly each missed combination.
A separate gap is between the laboratory sample and scale-up. A composition that is stable in a flask at one hundred grams behaves differently on the line: heat transfer, order of addition, and mixing time change. The transfer consumes several iterations, and each is a new round of lab trials.
Finally, the formulation development cycle is rarely recorded in full. Series of experiments remain in scattered files and logs, some results are not indexed, and when an engineer changes, the context leaves with them — the company pays for the same experiments again.
It is precisely at these three points — enumeration, transfer, and lost data — that AI formulation optimization works. We break down how this is addressed below.
What composition modeling delivers: timelines, stability, less scrap
Empirical formulation selection is limited by the trial cycle: mixing, holding, testing — and only then a conclusion about the composition. One iteration takes weeks, and a noticeable portion of results does not transfer from the lab to the shop floor.
The model learns from lab logs and certificates of industrial batches: components, input ranges, mixing and drying regimes, actual output properties.
Then the experiment planner suggests the next point in the composition space — the one that gives maximum information about the system, not the one closest to the previous. We calculate property predictions on thermodynamic models, e.g.
DWSIM, so the composition is checked for component compatibility and behavior within the operating temperature range before raw materials are loaded.
- Formulation selection cycle: months of trials compress to weeks — some physical experiments are replaced by computation.
- Share of successful trials: most proposed compositions fall within the target property region, not just a few.
- Batch reproducibility: the model maintains tolerance for raw material variability, not only for the average point.
- Property stability: prediction is built per batch, not per single sample — batch-to-batch variation narrows.
A separate class of failures is addressed — "passed in the lab — failed in the shop": loading volumes, equipment regimes, and temperature profiles are entered into the model, so the formulation is evaluated under production conditions, not in a flask.
At the output, the engineer receives not a single "optimal" point but a tolerance window — with what raw material variation the batch remains within specification.
Composition search modes: full enumeration, directed search, hybrid
The search mode is chosen based on the dimensionality of the composition space and on how strict the component constraints set by the process engineer are.
Full enumeration of the matrix guarantees that the best option is not missed, but the number of combinations grows as the product of gradations for each ingredient: ten components with five dosages yield hundreds of thousands of rows, and synthesizing and measuring each still takes time and material.
Directed composition search instead of enumeration builds a "composition — property" model and suggests the next point for testing, cutting off obviously unsuitable regions by constraints.
Hybrid formulation selection combines both approaches: coarse enumeration outlines the working region, and directed search concentrates experiments within it.
| Option | When it fits | Limitations |
|---|---|---|
| Full enumeration of the matrix | Few components, wide dosage range, property needs to be described across the entire region | The number of combinations grows multiplicatively and hits the lab's throughput already at six to seven ingredients |
| Directed composition search | Many components, expensive synthesis, there are a priori component constraints | Requires starting data for the model; in a sparse region it converges to a local result |
| Hybrid formulation selection | Mixed constraints: some components are discrete, some continuous | Harder to configure: control of both search stages and separate validation of the result |
Implementation stages: from laboratory data audit to model handover
We start not with a model but with a review of what has already been accumulated. Laboratory logs over many years are usually heterogeneous: units of measurement changed, some batches are duplicated, and some components have long gaps.
Until this is sorted out, any training yields a beautiful metric on paper and a discrepancy with actual product output.
- Laboratory data audit. We consolidate experimental records into a single table, normalize units, and label raw material batches and process conditions. We record which composition ranges are actually confirmed by experiments and which the model must not extrapolate.
- Feature schema. We determine what describes the composition: mass fractions of components, incoming raw material properties, process parameters. We separately mark target indicators and constraints on permissible corridors.
- Training on experiment history. We split the dataset chronologically, not randomly — otherwise the model peeks into future experiments and shows unattainable accuracy. We tune the trade-off between prediction accuracy and robustness.
- Validation on held-out compositions. We test formulations the model has never seen. We calculate not the average error but the share of compositions that fell within tolerance on the first calculation — this is the working metric.
- Industrial composition verification. We run selected formulations on real equipment and compare actual results with the prediction. Discrepancies are returned to the dataset as new data.
- Handover to the team. We hand over the model, the regulations for its use, and the procedure for retraining on fresh experiments so the system develops without us.
Project artifacts: constraint map, regulations, documentation
Delivery of a formulation optimization project is not a demonstration of a working model but a package with which the process engineer will launch the next batch without our involvement.
Below is what remains with the customer in a closed loop: configurations, schemas, rules, and regulations.
- Constraint and specification map — process limits for each component, raw material compatibility, safety requirements, and permissible process ranges. Each constraint has a source indicated: internal regulation, raw material batch certificate, or customer requirement.
- Feature schema and calculation rules — which raw material properties and process parameters are included in the model, how derived values are calculated, where data comes from, and how gaps in logs are handled.
- Model configurations and experiment log — training parameters, feature set, quality metrics on the held-out set. This makes it possible to retrain the model on new batches and compare the result with the previous run.
- Batch reproduction regulations — how to turn a model recommendation into a process sheet: loading order, control points, actions when an indicator goes out of tolerance.
- Data access rules — role separation, who sees raw material logs, who sees calculation results, how formulation changes are recorded, and who confirms them.
- Model documentation — dependency map, expected accuracy by formulation classes, and applicability boundaries: where the prediction is reliable and where laboratory verification is needed.
- Instructions for process engineers — short scenarios for each step: set constraints, get options, evaluate the prediction, enter the actual batch result back.
Handover to process engineers is a separate stage: review of disputed formulations, training of an internal administrator, and acceptance checklist verification. Until the process engineer has reproduced a batch independently, the project is not considered closed.
Case study: formulation selection cycle shortened, share of successful trials increased
How it was set up before implementation
At the customer — a manufacturer of industrial coatings — the formulation was selected by enumeration: the process engineer set a matrix of component fractions, laboratory technicians prepared weighed samples, and the samples went for adhesion, viscosity, and resistance testing.
One formulation took 40–60 laboratory trials, the formulation selection cycle took 10–12 weeks, and about every third batch passed acceptance testing on the first pass.
Some rejections surfaced only on the scale-up unit — when weeks of work had already been spent and the cause had to be searched for anew.
What was changed
We collected historical test protocols, batch certificates, and process logs into a single table "composition → conditions → result," and labeled gaps and discrepancies between series.
On top of that, we trained two models: one to predict target indicators from component fractions, and the second to estimate the probability of passing acceptance tests.
Constraints were built into the search: raw material availability, component compatibility, and fraction ranges beyond which the technology cannot handle.
Then active selection works: the next trial is placed where the model either promises the best result or is most uncertain. The prediction is always verified by a real weighed sample — the system ranks trials but does not issue a ready formulation in place of the laboratory.
What became
The number of trials per formulation decreased from 40–60 to 12–14, the share of successful first-pass trials increased from one third to more than two thirds, and the time to reach a formulation went from 10–12 weeks to 3–4.
The yield of usable product during scale-up rose from 68% to 91%: the model suggests which ratios behave unstably when moving from a laboratory sample to a reactor.
The team verified each recommendation against actual tests — the discrepancy between prediction and measurement remains within limits at which the trial stays meaningful. The database is replenished with new series, so ranking accuracy grows as data accumulates.
How to test a formulation for robustness to raw materials of varying quality?
A formulation cannot be tested for raw material robustness using an "average batch": a model trained on certificate values selects a composition for the center of tolerance.
This is typical model overfitting — the metric on the held-out set holds, but when a batch deviates by one and a half sigma in moisture or dispersity, properties go beyond the tolerance boundary.
Therefore, a distribution is fed into the model, not a point.
We take raw material property variation from the incoming raw material inspection log: for each item, we build intervals for impurities, moisture, and correlations between them, not just "minimum — maximum." We perform enumeration within the intervals using the Monte Carlo method — thousands of virtual batches, with the formulation calculated separately for each.
The choice of method depends on what is already in the archive and how strict the property requirements are.
| Approach | When it fits | Limitations |
|---|---|---|
| Historical batches from incoming inspection | there is an archive covering several seasons | rare outliers are not included |
| Synthetic variation by tolerances | raw material tolerances are formalized | does not see correlations between impurities |
| Optimization by worst batch | viscosity, adhesion, color are critical | narrows the formulation space |
| Interval property estimation | risk is needed, not a single number | requires more data |
The acceptance criterion is not average error but the share of batches within tolerance: a formulation is robust if it maintains target properties on 90–95% of simulated batches.
The final check is on blind batches not involved in training: it shows that the model relies on process physics, not memorized combinations.
Weak points of the approach: model accuracy, data quality, support
Formulation prediction accuracy is not universal: it holds within the region of compositions and regimes covered by past experiments. Beyond its boundaries, the model extrapolates, and instead of a plausible number you get a confident but incorrect answer.
Therefore, we explicitly delineate the applicability zone and provide an interval prediction indicating what data it is based on.
The second limitation is the quality of history. Paper experiment logs hold value but in a form unsuitable for computation: different units, abbreviations, unrecorded raw material batches and conditions.
Converting to a structured table and resolving contradictions take a significant part of the project — this is not preparation but work without which the model will not train.
What if there is little historical data?
Then we rely on transfer from related compositions and on a plan of controlled experiments: a few dozen points in the required range often provide more than hundreds of scattered records.
The applicability zone narrows honestly — we state directly where the prediction is reliable and where laboratory verification is needed.
Who is responsible for the result and how does the model live after launch?
Support is part of the work, not a separate service: we monitor the discrepancy between prediction and actual output, record when raw materials or conditions go beyond the trained region, and initiate retraining.
We have a responsible engineer and an escalation procedure, and a disputed recommendation is tested on a control batch before being transferred to production. Responsibility for the result is distributed across these steps and does not dissolve in a report.
Let's discuss the task: what data is available and what is needed at the output
To come back not with general words but with a working scheme, we need a minimum of inputs: what is already available on compositions and test results, what constraints are set by regulations and production rules, and what is considered a successful result.
Describe the task in free form — tables, exports from the laboratory log, and internal documents are not needed for the first conversation.
In response, we send not a commercial proposal "for everything" but specifics for your case:
- data review for the start: what is actually needed from test registries and formulation certificates, and what will not have to be collected;
- a list of regulatory constraints that the system must comply with, and a way to check each proposed formulation option for compliance with them;
- acceptance criteria — by which indicators and on what set of compositions we consider the result accepted, rather than "looks plausible";
- a staged work scheme with points where you accept an intermediate result or stop the project;
- timelines for the first stage and what will be visible at it before investment in the full system;
- the composition of transferred materials: configurations, data preparation scripts, and regulations for retraining on new tests;
- a rollback plan: how to return to the current selection procedure if the result turns out worse than expected.
Describe the task and attach what you have — we will come back with an estimate, a work scheme, and a first-stage plan.
