ML Models for Finance: Where Scoring and Antifraud Break Down
Application scoring and antifraud models almost never break during training — they break in production, where the data is no longer the same as what the model was trained on.
On a holdout sample, the metrics look decent, but in the stream the approval rate shifts, the manual review queue grows, and risk analysts stop trusting the system's decisions.
The first weak spot is features.
Application scoring lives on credit bureau data, streaming events, and customer-level aggregates, and any of these branches can lag: the export arrives once a day, device counters have not yet been collected, the merchant category has not been pulled in.
The model gets half the picture and understates the score for an application that is acceptable by risk profile.
In antifraud, the picture is different. The model works on top of a set of hard rules, and the rules override its output: if a stop list triggers, the application goes to a human regardless of the low fraud probability.
Over time, rules accumulate and intersect, and the share of applications in manual review grows faster than the stream itself.
The third cause is the feedback loop. Labels for the antifraud model come only from confirmed cases, and those are confirmed that passed manual review.
If only applications with a high score are reviewed, the next training cycle will reinforce this bias: ML models for finance will never see the fraud they themselves missed.
Risk management rests on three things: score predictability, explainability of a rejection, and decision speed.
When they diverge — monitoring does not catch data drift, the reason for a rejection cannot be reconstructed, a queue accumulates in manual review — the system formally works but stops managing the flow.
Next, we break down how this is addressed: from feature quality control and score calibration to triaging the manual review queue.
Measurable Effect: Decision Speed and False Rejection Rate
Manual review sets a ceiling on speed: an application waits for triage, during peak hours the queue grows, and a decision is made after minutes. The model calculates the score on the event stream and returns an answer in seconds, leaving only borderline cases to a human. The risk function stops being a bottleneck — throughput grows without expanding headcount.
The false rejection rate is the second measurable lever. Hard rules trigger on legitimate customers: a large amount from a person with a short history, a purchase right after a change of region.
The model ranks by probability, and the cutoff threshold is tuned to the target false rejection rate and recalculated on a control sample. A separate loop for repeat inquiries removes most of the friction.
Fraud losses fall due to a class of errors that disappears — schemes that look harmless in isolation: amount splitting, chains of linked accounts, device substitution. The relationship graph and customer-level aggregates are visible to the model where rules are powerless.
MLflow stores model versions, feature sets, and metrics, so the result after release is compared with the baseline on the same sample, not "by eye." If a metric drops on live traffic, rollback to the previous version takes minutes. Such a tracking loop makes changes verifiable for audit and predictable for the risk function.
Implementation Options: Rules, Statistics, Learning from Data
The approach is chosen not by "modernity" but by the volume of labeled history, requirements for decision explainability, and the cost of an error.
Rules and expert thresholds launch immediately and are fully explainable, but each new fraud scheme has to be described manually, and the false positive rate grows along with the number of rules.
Learning from historical data finds non-obvious feature combinations, but is sensitive to distribution shift: a scheme that was not present in past data looks normal to the model.
That is why in production operations a hybrid scheme is more often assembled: rules cover obvious cases and act as a safeguard, the model ranks the remainder by probability, and portfolio statistics set the cutoff thresholds.
| Option | When It Fits | Limitations |
|---|---|---|
| Rules and expert thresholds | No accumulated history, full explainability of every verdict is required | Scale poorly: rules conflict with each other, and the false block rate grows |
| Statistical models | Features are stable, interpretation of each factor's contribution is important | Do not capture nonlinear relationships, quickly become outdated when customer behavior changes |
| Learning from historical data | Labeled history in hundreds of thousands of events, retraining delay is acceptable | Requires data drift control, vulnerable to sample poisoning and new schemes |
| Hybrid scheme | Production operation with a high cost of error and selective explainability | Harder to maintain: two logic branches, consistency monitoring of verdicts is needed |
We take as a basis the option that withstands the error matrix of the specific process: where it is more expensive to miss an event, and where it is more expensive to block a legitimate customer.
Work Stages: From Data Review to Decision Handover
The order of stages is set not by methodology but by how the model will live in production: who runs it, what data it uses at decision time, and what happens when the input data distribution shifts.
Therefore, the work proceeds in iterations — the result of each step either confirms the next one or returns us a step back.
- Data review and problem statement. We profile sources: look at field completion, discrepancies between systems, data arrival lag, and signs of target variable leakage. We fix what exactly we predict and at what point in time this information is actually available.
- Feature preparation. We build a training mart, describe the calculations, and version them so that the same values are produced both during training and when the model runs.
- Baseline solution and comparison. We build a simple reference model: it shows whether there is any signal in the data at all and serves as a bar below which there is no point in adding complexity.
- Validation on an independent sample. We evaluate quality on a holdout period that does not overlap in time with the training period. This shows whether the result holds on new data, not just on familiar examples.
- Calibration and threshold selection. We bring scores to real probabilities and select the trigger boundary for the cost of type I and type II errors.
- Model implementation. We connect the scoring service to the customer's process and launch it in shadow mode — the model scores, a human makes the decision; versions and parameters are fixed so the result is reproducible.
- Handover to operations. We hand over retraining regulations, validation sets, input data drift dashboards, and the procedure for rolling back to the previous version.
What the Customer's Team Retains After Handover
Handover is complete not when the model is shown in a demo, but when the customer's team can explain any of its decisions, repeat training on the same data, and disable the model without us.
Therefore, we deliver a package of artifacts, not a set of scripts: each decision has a described applicability boundary and conditions under which it must be revisited.
What remains in hand:
- Model documentation: data sources, feature schema, holdout metrics, known limitations, and scenarios where a decision cannot be made without manual review.
- Retraining regulations: triggers — metric decline, input data distribution shift, changes in business rules; frequency, responsible person, and procedure for validating a new version.
- Decision handling policy: thresholds for automatic approval, rejection, and manual triage; how a specialist's disagreement is recorded and how this signal returns to the training data.
- Quality monitoring scheme: control slices, input stability and output quality metrics, calculation frequencies, alert thresholds, and the on-call procedure.
- Version map and rollback: which dataset and parameters produced each version, how to deploy the previous one, and what happens to decisions already made.
- Access matrix: who is allowed to change rules, who can retrain, who can only view, and how this separation is checked.
- Acceptance criteria: which metrics each version is accepted by and at which points we connect further.
Case Study: Scoring and Antifraud in a Single Application Stream
Applications came in a single stream but were processed by two independent services: scoring calculated the probability of default, antifraud — the fraud risk, and each saw only its own part of the features.
In practice, this produced two problems: the antifraud false positive rate reached 12%, and some confirmed fraud passed within already approved applications.
| Metric | Before | After |
|---|---|---|
| Antifraud false positive rate | 12% | 3.4% |
| Applications sent to manual triage | 18% | 5% |
| Application decision time | up to 40 minutes | up to 2 minutes |
| Recall of confirmed fraud | 71% | 91% |
| Release of a new scoring model version | 6–8 weeks | 10 days |
What Was Rebuilt
Both loops were merged into a single stream: features are collected in a common storage at the application event level, antifraud and scoring work from the same mart, and they exchange decisions before returning a response to the customer.
Antifraud thresholds are configured by segment — they differ for payroll clients and new borrowers, so the reduction in false positives is not bought by an increase in misses.
Model versions are fixed in MLflow, and rollback to the previous version does not require rebuilding the stream.
The before and after metrics were taken on the same load period so the comparison would be fair. The application stream grew approximately threefold without expanding the manual triage team, and the share of rejections that a customer disputed and won was cut in half.
What to Do If the Company Already Has Rules and Manual Reviews in Operation
Rules and manual reviews are not something to shut down when implementing ML. They already contain expertise: limits, stop factors, industry regulations developed over years.
The reasonable order is not replacement but parallel operation: the old loop remains primary and continues to make decisions, while the new one scores in shadow mode on the same application stream.
The choice between rules and machine learning is made based on the nature of the feature. Hard deterministic conditions — sanctions lists, age and limit restrictions, mandatory stop factors — remain rules: they must be explainable to an auditor and regulator.
Machine learning takes over where there are too many rules and they have begun to conflict: feature intersections, nonlinear dependencies, segments where correct manual review turns into a queue for several days.
According to the CRISP-DM methodology, this is a separate step — data analysis and problem statement, not algorithm selection.
A transition without stopping the process is built on comparison. We take the history of closed cases with a known outcome, run it through the models and through the current rules, and look at where the scores diverge and why.
Two metrics decide the fate of the solution: what share of risky cases is intercepted and what share of legitimate customers goes to the manual queue. As long as the new scheme loses on either of them, it does not come out of the shadow.
Borderline cases are moved to a separate loop. The model returns a score, not a verdict: in the uncertainty zone, the case goes to manual review with a ready set of features and reasons that highlighted it.
The threshold is configurable, the share of such cases is a controllable parameter, usually single-digit percentages, and it can be adjusted without changing either the rules or the model.
Then the loop narrows gradually: a rule is disabled only when the model consistently overrides it on the holdout sample and in the live stream. Rollback means returning the threshold and enabling the rule back, without a release or stopping application intake.
Technical Concerns to Resolve Before Starting
Technical concerns around ML in finance rarely concern the algorithms themselves — they concern data, decision explainability, model stability on new data, and how all of this will fit into an already operating process.
Below are the four questions most often asked before starting, and how we address them engineering-wise.
Will our data be enough for the model to work?
Volume is rarely the bottleneck — quality is critical. We start with profiling the extract: missing values, desynchronization between sources, rare classes such as isolated fraudulent transactions.
Then comes cleaning, stabilization of reference data, and drift control so the model does not "go blind" when customer behavior changes.
How can the model's decision be explained to an auditor or customer?
For finance, explainability is not an add-on but a requirement. We provide feature importances for each decision, local explanations for the specific case, and surrogate rules by which a human can reproduce the decision. The output is a wording that can be shown in a disputed situation.
Will the model overfit and fall apart on new data?
This is caught at the validation stage, not in production. We train on the past and validate on the future — a time-based holdout period instead of a random sample.
After launch, we monitor quality and input data drift; retraining follows the regulations and a degradation trigger.
How can the model be integrated into the current process?
The model does not replace rules but works on top of them. First comes suggestion mode: the specialist sees the score and explanation and makes the decision themselves.
Once accuracy on confident outcomes is confirmed, those cases go to automation, while borderline ones remain with a human. Integration is via a service layer, without restructuring processes.
Let's Discuss the Task and Return with an Estimate and Work Plan
Estimation of a machine learning model project for finance begins not with algorithm selection but with reviewing the task and data: what the target variable is, how quickly confirmation of the correct answer arrives, which errors are more expensive — a missed delinquency or a rejected solvent customer.
The metric, trigger threshold, and quality validation scheme depend on this. Therefore, discussion of the task comes before we name timelines.
During the review, we close specific questions, each of which changes the work plan:
- Data and labeling. Where issuance and transaction histories are stored, whether the outcome for each case is confirmed, and how much time passes before the label appears.
- Metric and threshold. What error rate is acceptable in the production stream, and how the losses from a miss compare with those from a false trigger.
- Validation scheme. A time-based holdout period instead of a random split — otherwise quality on paper is higher than in production.
- Explainability. Which decisions need to be justified to auditors and the customer, and which features are permissible to use.
- Implementation format. Batch scoring, scoring at transaction time, or a prioritized manual review queue.
- Monitoring and rollback. Input data drift control, scheduled retraining, and rollback to the previous model version.
- Resources. What is required from your side in terms of data, access, and process knowledge.
Describe the task — we will return with a project estimate and work plan: stages, team composition, areas of responsibility.
