Why developing an AI model for candlestick patterns fails
Most often, developing an AI model for candlestick pattern analysis breaks not on the architecture but on the data.
Pattern labeling is done by people or scripts, and where one considers a "double bottom" completed, another sees a sideways range: the pattern boundaries, the allowable retracement depth, and the confirmation moment are interpreted differently.
A model trained on such labeling averages out the contradictions and produces a signal that cannot be confirmed.
The second problem is rare formations.
When training on historical quotes, "head and shoulders" or a wedge occur many times less often than trending sections, and pattern recognition shifts toward the frequent class: the system predicts continuation of movement and misses the reversal.
Adjusting the threshold does not help here — what is needed is precise labeling of rare formations, class weighting, and separate validation on them.
The third source of failure is timeframe scale. The same candlestick configuration on a lower and higher chart has different statistics: on the lower it is noise, on the higher it is a significant formation, so a model trained on one scale loses accuracy when transferred. A feature calculated on an unclosed candle adds signal delay and false triggers on the last bar.
In practice this looks like this: the system either lags on entry or stays silent where the reversal was obvious in hindsight. Below we break down how this is closed at the level of data preparation, labeling, and model validation.
What changes after implementing a pattern recognition model
Labeling historical data is the most time-consuming stage: an analyst manually marks formations on the chart, and on a long history this means thousands of marks with different interpretations of boundaries.
After implementing the model, the share of manual labeling drops to spot-checking — the engineer reviews only disputed fragments where the model's confidence is low.
Pattern recognition accuracy on the holdout set stays within a range where discrepancies with the analyst's labeling remain acceptable for trading logic.
Robustness to noise in quotes is addressed separately. The model is trained on data with artificially introduced gaps, outliers, and liquidity breaks, so it does not mistake a random series of shadows for a formation.
This removes the class of errors that was most costly in practice — false level breakouts, when a signal triggered on a thin market and reversed within the next candle.
Reproducibility of the result is ensured by training on PyTorch with fixed versions of weights, parameters, and dataset. Any run can be repeated with the same result, and comparison of two model versions relies on identical input conditions rather than the analyst's memory.
Validation on new data is built into the process: the model is run on a period that did not participate in training, and metrics are compared with the previous version. The signal does not degrade when volatility changes, and update decisions are made based on numbers.
Ready-made solution or training on your data: comparing approaches
Comparison of approaches begins not with choosing an algorithm but with the question: how similar are your candlestick charts to what the system has already seen.
A ready-made recognition system goes into operation quickly, but no one has checked its quality on your instrument and timeframe.
Fine-tuning for your market transfers the already learned candlestick shape features to your data — but only if the history is labeled uniformly and without gaps.
Below are three working options that are actually chosen between, and what limits each of them.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made recognition system | Need a working result in a short time, formations are standard | Logic is closed, labeling and trigger thresholds cannot be adjusted; quality on your market is not confirmed |
| Fine-tuning a ready-made model for your market | You have your own observation history and 2–3 thousand labeled examples | Requires labeling discipline and a separate validation period where market conditions did not change |
| Training from scratch on your data | Non-standard formations, your own entry and exit rules, need full transparency of the decision | The longest path: requires a large and clean labeled dataset |
In practice, the middle path is most often taken: a ready-made recognition system as a base layer for standard formations, fine-tuning for patterns of your market, while entry and exit rules remain with you.
Such a hybrid gives a working result early on and leaves room for accuracy growth as labeling accumulates. In a task review, we look at your history and say directly which option will produce results on your data specifically.
Work stages: from data audit to handover into operation
We go from data to model, not the other way around: first we check that the history has enough examples of each pattern, and only then do we start training.
Each stage is closed with an artifact — a report, a labeling scheme, or a validation protocol — so an error is uncovered at the current step, not during operation.
- Audit of historical data. We check the depth of history for each instrument and chart scale, continuity of quotes, gaps, and breaks. The result is a suitability report: for which patterns the sample is sufficient, and where it needs to be expanded.
- Labeling scheme. We fix a formal definition of each pattern: candle composition, proportion tolerances, behavior at pattern boundaries. Without this, labelers and the model understand the same pattern differently.
- Labeling quality check. We calculate agreement between labelers on a control subsample and review disputed cases. We set the agreement threshold before work starts, not after.
- Training. We train the model on labeled history, select parameters on validation, and separately work through rare patterns and class imbalance.
- Validation on a holdout period. We run the model on data it has not seen, including sections with different volatility. We look at precision and recall for each pattern separately, not an average score.
- Deployment. We connect the model to the data stream, set up prediction logging, and monitor divergence from actual market behavior.
- Handover into operation. We provide a support procedure: when to retrain, what to do when metrics decline, and how to roll back to the previous version without downtime.
What the client receives: package and documentation
The package is handed over so that the system can be launched, validated, and developed by your own team. Therefore it includes not only the trained model but also the rules by which it was prepared and the schemes by which data flows before and after it. Below is what exactly is recorded in the transferred documentation.
- Trained pattern analysis system: model artifact, fixed calculation configuration, and a control set of examples on which the result is reproduced exactly.
- Labeling rules: a dictionary of candlestick combination patterns, criteria for class assignment, and review of disputed cases — so that new labeling does not diverge from the training one.
- Data processing scheme: quote sources, procedure for bringing to a unified form, handling of gaps and outliers, feature calculation window.
- Metric description: how precision and recall are calculated for each pattern, false signal share, metric on the control set, and thresholds below which the result is considered non-working.
- Launch instructions: procedure for bringing up the environment, entry points, validation on control data, and typical startup errors.
- Support and update procedure: retraining frequency, signs of data shift that send the model for retraining, and procedure for handling divergence.
- Applicability boundaries and acceptance protocol: on which instruments and timeframes the model has been validated, what limitations are known, and by which examples the client accepts the work.
Case study: pattern recognition accuracy before and after fine-tuning
The client is an algorithmic trading team: signals based on candlestick patterns were sent to analysts for confirmation. Candlestick history was collected from TradingView exports and an internal trade archive.
A model trained on "by eye" labeling confidently found only frequent formations, while rare ones — "head and shoulders", double bottom, narrow consolidations — were confused with each other. Every second signal went to manual recheck.
| Metric | Before | After |
|---|---|---|
| Recall for rare pattern recognition | 58% | 91% |
| Share of signals for manual review | 54% | 12% |
| Accuracy spread by month | ±11 p.p. | ±2 p.p. |
What was changed
First, the labeling rules were rewritten: strict criteria for body and shadow proportions, minimum formation size relative to instrument volatility, and a separate class for "looks similar but not a pattern".
Conflicting labeler examples were reviewed again, after which the model was fine-tuned on balanced classes and rare formations were augmented with noise and scale distortions.
Validation on holdout data
Validation was split by time, not randomly — otherwise adjacent candles of the same movement end up both in training and validation, and metrics look better than they are.
The model did not see the control period; additionally it was run on instruments outside the training set: recall dropped by several points, but the spread between months remained within two points.
How pattern labeling differs from technical analysis
Pattern labeling and technical analysis solve different tasks, and they must not be mixed in the task statement. Technical analysis is a decision-making method: levels, trends, indicators, and context that the analyst interprets.
Labeling is the target variable for the model: a window of candles for which it is unambiguously known whether there is a formation or not and by which rule this was decided.
Technical analysis translates poorly into a label: a level becomes obvious only after a breakout, and different experts evaluate a "strong candle" differently.
The canonical description of formations by Steve Nison is formulated in words — "small body", "long shadow" — and without numerical tolerances becomes a subject of dispute between labelers.
Therefore, the task statement begins with a formal rule: a window of candles in OHLCV terms, body and shadow proportions relative to volatility, minimum formation size, behavior when patterns overlap.
Only then is the label reproducible, and the model learns from a pattern rather than the tastes of a particular labeler.
The difference is visible in the result. If the task says "the model must understand the trend", labeling diverges between experts, quality does not converge, and it is unclear what to fix — data or architecture.
If the geometry of the formation and the event horizon are described, disputed cases can be rechecked, and quality can be compared with a baseline rule.
Practical conclusion: technical analysis sets hypotheses, labeling sets a measurable goal. Until the labeling rule is fixed, any assessment of model quality is random.
Common technical doubts before starting work
Engineering doubts at the start usually concern not the model itself but the data and the boundaries of its applicability. We have collected four questions most often raised at the first meeting and answer them substantively — without generalities.
How do you check that the model is not overfitted to past data?
We split the history by time, not by random sampling: training runs on an earlier section, validation on a later one that the model has not seen. Additionally, we run several independent time windows and look at the spread of metrics between them.
If quality holds across all windows, not only the last one, this is a sign of robustness rather than memorization.
What happens when the market regime changes?
We build features as relative, tied to scale and volatility, so a sharp shift in levels does not break the pattern logic. On top — a rolling training window and distribution drift monitoring: when input data moves away from the training profile, a signal to rebuild arrives.
Degradation on regime change is inevitable — we build it into the plan rather than promising eternal accuracy.
In what format are data and labeling needed?
A time series of candles and event labeling are needed: pattern boundaries, its class, and labeler confidence. We fix the scheme at the start and validate it automatically — gaps, duplicates, and time breaks are filtered out before training.
What is critical is not so much the length of history as the number of examples of rare patterns: the model learns worst on them.
Who owns the data and the result?
The data remains yours and does not leave your perimeter: processing runs on your infrastructure or in an isolated environment under your control.
We transfer rights to the model, code, and documentation to you; we do not use your exports in our products and do not leave copies after delivery.
Let's discuss the task and come back with an estimate and plan
To come back with a timeline estimate and work plan, we need not a retelling of the idea but the data and boundaries of the task. The more precise the input, the fewer assumptions in the estimate — and the higher the chance that acceptance metrics match expectations.
- Quote history for the required instruments: OHLCV format, intervals, sample depth, and places of data gaps.
- Description of the patterns the model must find: verbal criteria and examples of "signal present" / "no signal".
- Labeling or the rule for obtaining it: who marked chart sections and how disputed cases were resolved.
- Application boundaries: on which intervals and instruments the model works, what we consider a missed signal and what an error.
- Acceptance metrics: share of correct triggers, allowable detection delay, share of false signals on historical data.
- Target environment: where the model will compute and what limitations exist on latency and data volume.
- A responsible person on your side with the authority to approve labeling and accept the result.
Next we assemble a baseline model for comparison, check labeling quality, and come back with a timeline estimate and stage-by-stage work plan — from data preparation to degradation monitoring on new periods.
Describe the task in free form: we will discuss it substantively and tell you which inputs are missing.