AI analysis of advertising creatives: why CTR forecast diverges
The divergence between forecast and actual starts not in the model, but in the problem statement. Decisions about the fate of a layout are made based on campaign averages: if an ad group has hit the target cost, no one isolates a weak creative within it.
It picks up impressions at the expense of neighbors' budget and burns out after launch, when an edit costs more than a new shoot.
For CTR forecasting to be workable, the object of evaluation should be the creative itself, not the campaign.
AI analysis of advertising creatives decomposes the layout into features: composition and focal point, object in frame, volume and readability of text, contrast to background, first frame of video, alignment of the promise with audience and placement.
The model then compares these features with the actual response of similar ads under comparable impression conditions.
The second source of error is mixing effects. A high CTR can be driven not by composition but by audience, bid, or placement; a low CTR by a narrow sample at launch.
Therefore, layout features are separated from impression context, and pre-launch creative evaluation is adjusted for whom and where it will be shown.
We treat the divergence between forecast and actual as a managed metric. The model is calibrated on past launches: predicted and actual response are compared, features that pull the error are filtered out, and weights are rebuilt until the variance becomes predictable.
Then weak layouts are filtered out at the preview stage, not after budget has been spent — and advertising creative effectiveness is evaluated by the ad itself.
Below we break down how this works in practice and what exactly breaks most often.
What pre-launch creative response forecasting provides
The hypothesis testing cycle shrinks from weeks to days
A creative response forecasting model is built on historical impression data and evaluates a layout before campaign launch, not after a week on live traffic.
The designer and media planner get an assessment during edits — replacing a headline or frame takes minutes, not placement overspend. The hypothesis testing cycle shrinks due to volume: in one pass, not 2-3 layouts are checked, but the entire pool, sorted by expected response.
A unified creative scoring scale across channels
One model evaluates layouts for feed, banners, and video and brings them to a common scale — comparison no longer depends on whose report is more convincing.
Previously, layouts from different channels were compared using non-comparable metrics and “by eye”; now the team has one ranked list. The scale is calibrated on the client's data, so scores are comparable across campaigns and seasons.
The failure class “budget spent on a knowingly weak creative” goes away
The most expensive error is when the placement has already spent budget, and the creative delivers response below baseline. Forecasting filters out such layouts before launch: only those passing the expected response threshold enter the campaign.
The threshold is configured for the task — softer for a reach campaign, stricter for a conversion campaign — so filtering does not cut working options. The first runs show where the model errs, and it is further trained on the client's own data.
Forecasting modes: batch, streaming, and hybrid
The forecasting mode is chosen not by the “strength” of the model, but by the rhythm of launches and the volume of accumulated history. If creatives are approved in batches once a week and tens of thousands of impressions have already been accumulated for them, the logic is one.
If a designer uploads a layout and expects a verdict in the same minute, it is another, because there is no history for that layout yet and the model relies on the features of the ad itself.
Below is a comparison of forecasting modes by applicability and limitations. We discuss these boundaries before work starts so that the mode does not have to be changed mid-campaign.
| Mode | When it fits | Limitations |
|---|---|---|
| Batch response forecasting | Regular recalculations across a portfolio of creatives, with impression history over months | The answer comes on schedule, not on demand; a fresh layout waits for the next run |
| Streaming forecasting on creative upload | A verdict is needed at upload time: the designer edits the layout and immediately sees the score | Relies on the ad's own features; on rare formats and new placements, the score is less stable |
| Hybrid evaluation model | Different launch rhythms: mass campaigns plus targeted tests | Requires support for two loops and unified calibration between them |
Hybrid evaluation model is not a fallback option “just in case,” but a working scheme for a mixed rhythm. The batch loop is retrained on accumulated history and maintains calibration; the streaming loop gives a quick answer for a just-uploaded layout.
We measure the divergence between the loops separately: if the streaming forecast systematically overestimates response on narrow formats, this is visible before launch, not after budget has been drained.
Comparison of forecasting modes should be done by two parameters: how often you launch creatives and how much history has already been labeled.
Switching between modes does not mean rewriting the system — features, labeling, and quality metrics remain shared; only the way the request is submitted changes. We break down how to choose a mode for your launch rhythm.
How a creative evaluation model is built for your campaigns
A model cannot be trained from nothing: first you need to understand what data about past launches exists at all, what state it is in, and whether there are enough observations for a stable forecast.
Below is the order of work, where each next step relies on the result of the previous one; doing them in parallel is pointless.
Order of work
- Audit of historical launches. We merge exports from ad accounts with product analytics, look for gaps, duplicates, and campaigns that were run outside common rules. The output is a data map: what and for what period is suitable for training.
- Labeling of historical launches. We bring creatives to a unified form and record the target metric based on actual impressions, not a manager's assessment. We separately mark launches where conditions changed on the fly — the model should not treat them as a reference.
- Feature collection and selection. From layouts, texts, audiences, and placements we form a set of features and filter out those that are only accidentally related to the result. A feature that cannot be obtained before launch does not enter the model.
- Training and calibration of the forecast. We train the model on one period and align the forecast scale: if the system promises a creative above-average results, that score must be confirmed in practice with the same frequency.
- Validation on a holdout sample. The model has not seen the control period during either training or tuning. We look not at average accuracy but at ranking quality: whether the system correctly sorts creatives from strong to weak.
- Handover to the client's team. We provide the code, data schema, retraining regulations, and applicability boundaries — for which campaign types the forecast can be trusted and where manual verification is needed.
This order yields a forecast that is reproducible on new launches, not only on those the model was trained on.
What we hand over with the model: artifacts and regulations
Handing over a model is not a file with weights, but a kit by which your team can train, verify, and explain the CTR prediction without us. Below is what physically remains with you after the solution is delivered.
- Training and inference configurations: parameters, cross-validation scheme, random seed fixing — a repeat run gives the same metric, not an “approximately similar” one.
- Feature schema: which fields of impression and click logs enter the model, how visual and textual features of the creative are calculated, field order, normalization and missing-value handling rules.
- Campaign data access policies: roles and access levels, anonymization before training, export retention periods, logging of dataset access.
- Evaluation model documentation: purpose, inputs and outputs, prediction range and calibration, applicability conditions, and known limitations — so the result is not interpreted more broadly than it is.
- Retraining regulations: triggers (distribution drift, metric drop on the holdout sample, new creative format), frequency, mandatory checks before rollout.
- Quality monitoring: set of tracked metrics, notification thresholds, degradation report, and procedure for rollback to the previous model version.
- Operating instructions: how to assemble the dataset, run training, pass acceptance checks, and update the model in the production loop.
The kit closes the main risk — the situation where the service works, but anything in it can only be changed through us.
Case study: creative forecasting cut idle launches
What it was before
The client is the retail chain “Severny Dom”: the flow of new layouts was continuous, and the launch decision was made based on the marketer's experience.
Each test launch lived two to three weeks and took impressions from proven combinations; the target CTR was held by eight to nine layouts out of thirty.
Weak creatives were identified after the fact, already after impression spend, and the decision on the next batch was again made blindly.
What was changed
We assembled an archive of several thousand past creatives, labeled them by CTR, seasonality, and placement, and added image, text, and format features.
The model evaluates each new layout before launch and builds a priority queue; once a week the forecast is checked against actuals, and the cutoff threshold is recalculated on fresh data.
What became
The share of hits in the top quartile by CTR is about eighty percent on layouts the model did not see during training. Trial launches decreased threefold: only promising options are tested, while the rest wait in queue or go for rework.
About two thirds of knowingly weak layouts do not go into circulation — they are filtered out by preliminary evaluation, not by impression spend. Preparation of a decision on a batch of layouts shrank from three days to an hour.
What to do if you already have your own evaluation model deployed?
Yes, it is possible to integrate into a working model, and there is no need to retrain it from scratch. We do not replace your loop but expand the feature set and compare forecasts on the same data.
Previous decisions remain in force until the new branch shows a consistently better result on the holdout sample.
Integration with the existing model runs as a parallel loop: your version continues to produce scores, and the new one works alongside it on the same stream of creatives. Forecasts from both versions are written to a common log — we keep versioning and tracing in MLflow.
We compare them by unified metrics: ranking quality, forecast error, and share of matches at the top of the list.
The main gain comes from feature augmentation: to your vector we add visual and textual embeddings of the creative, frame structure, first-seconds dynamics, and historical placement metrics.
Training runs on the same labeling and the same forecast horizon — otherwise comparison of forecasting approaches becomes incorrect.
Next we choose from three schemes: fine-tuning your model on the extended vector, a separate add-on over its score, or a hybrid with an independent forecast. The criterion is the volume of history and data stability, not habit with the current scheme.
We build in reversibility from the start: versions are fixed, the switch threshold is set in advance, and return to the previous logic takes one step. Current decision-making processes do not stop.
Technical doubts: data, accuracy, integration
Before starting, engineers and department heads ask the same questions: is there enough accumulated history, how is response forecast accuracy measured, where campaign data physically resides, and will the model displace the usual partial-audience test. We answer substantively — without promises to “predict response with percent-level accuracy.”
The model ranks creatives relative to each other and to the campaign baseline, rather than producing an absolute number.
Therefore we record quality on a holdout period: how often the best option by forecast turns out to be best in fact, and how stable the order is in the upper part of the list.
What response forecast accuracy should be expected?
We look at two indicators: the share of correctly ordered creative pairs and calibration — whether the stated response probability matches the observed frequency. Practical value appears when the model confidently separates the top third of options from the bottom.
How much historical data is needed?
Minimum — several thousand impressions per creative and full campaign cycles, including different seasons and placements; otherwise the model learns on a biased sample. When history is scarce, we transfer dependencies from similar lines of business and fine-tune on your data.
Where is campaign data stored?
Raw logs with impressions and clicks remain in your loop; anonymized aggregates and features go outside. Access separation, access logging, and retention periods are agreed before work starts.
Will the model replace partial-audience testing?
No, it is a filter before it. The model filters out knowingly weak options and reduces the number of hypotheses, while partial-audience testing confirms the result on live traffic — it remains the final arbiter.
Let's discuss the task: what to prepare for the first review
The task review starts with inputs — these are the same data on which the model will later be trained. The more complete the exports and the clearer the limitations, the sooner we can say what accuracy gain is realistic for your inventory and what is not.
- Export of creatives from past campaigns: images, videos, texts, placement sizes — from several hundred ads, so the model catches patterns rather than memorizing isolated cases.
- Impressions, clicks, conversions, and spend for each ad broken down by placement and date — without these slices it is impossible to build an honest target metric.
- Composition of audience segments and impression geography: they affect CTR more than the visual itself, and this must be accounted for in features.
- Access to ad accounts in read mode or export via API — we retrieve data regularly without interfering with current campaigns.
- Limitations: tested hypotheses, prohibited formats, what cannot be changed in the media plan.
- Forecast horizon: pre-launch evaluation or on-the-fly recalculation for bid adjustment.
Based on this set, we will return with a work plan for creative forecasting: labeling volume, validation scheme, quality metrics, integration points with ad accounts, and the procedure for a test launch on one campaign.
Describe the task — which creatives you analyze, how many ads are in circulation, and what you have already tried. We will return with an estimate and a plan.
