Apache Airflow ML pipelines: where model retraining breaks
Model retraining is often started manually: an engineer applies a fresh data slice and sequentially triggers feature preparation, training, and validation.
While there are three or four steps, the scheme relies on one person's memory, but as soon as you add data mart calculations or scheduled retraining, the launch order breaks down.
The model trains on incomplete data, and the discrepancy surfaces already in production, when the quality metric drifts for no apparent reason.
When a failure occurs midway through the pipeline, everything starts over: intermediate artifacts are not saved, and a rerun takes as long as the first.
Manually restarting tasks turns into a separate operation — you need to remember from which step to continue, which branches have already run, and whether you might overwrite a successful result that will still be useful.
A separate pain is experiment reproducibility. A week later, no one can say which data slice produced the current metric, which hyperparameters were in the config, and how the previous run differed from the one before it.
There is nothing to compare models against, and rolling back to a previous version turns into retraining blind.
All four failure points — manual start, rigid step order, loss of state on failure, and non-reproducible experiments — are addressed by a single training orchestration layer. We break down how this is covered.
What managed orchestration of training and model retraining provides
Regular training becomes manual work when steps are launched from memory: data export in one window, training in another, deployment — when an engineer is free.
The Apache Airflow scheduler replaces this arrangement with a schedule: training runs on a calendar or when fresh data appears.
The first measurable effect is model preparation time. The chain of ML tasks “data validation, training, quality evaluation, deployment” runs without pauses for coordination, and the run log shows which stage consumes more time.
The second is the share of tasks with manual intervention: only retrying a failed step and confirming deployment remain.
Data quality control is moved to the beginning of the chain. Schema, completeness, and value range checks do not let training proceed if the dataset fails validation: a model computed on garbage does not enter the registry. This is the class of failures that stops happening.
Retraining reliability rests on retries and failure notifications.
A failed task is restarted, the state of previous steps is preserved, and a notification goes to the work chat indicating the task and log — the on-call engineer sees the cause without reading the entire run history.
Thus orchestration moves model training from routine into a mode where intervention is needed for decisions, not for starting the chain.
Launch options: scheduler, scripts, hybrid scheme
The choice between a scheduler, scripts, and a hybrid scheme is a matter of the number of dependencies and restart frequency, not taste. We compare approaches along two axes: when the option is appropriate and what limitations it brings to maintenance.
Manual scheduled scripts hold up while the chain is linear and failures are investigated by eye.
As soon as branching appears, waiting for data from another system, or the need to retry only the failed step, manual checks grow around it: logs are scattered, the resume point is not saved, and a delayed run overlaps with the next one.
A hybrid scheme separates responsibilities: heavy calculations and one-off tasks remain where they already work, while the scheduler takes on schedules and dependencies — in Apache Airflow, this is the task graph and the state of each run.
| Option | When it fits | Limitations |
|---|---|---|
| Scheduled scripts | Linear process of 1–3 steps, one data source, infrequent runs | No mid-chain retry or run history, competing copies during delays |
| Task scheduler | Branching, multiple sources, partial graph recalculation, regular reporting | Deployment and maintenance of a state store, discipline in graph description |
| Hybrid scheme | Existing working scripts and external services, partial migration | Two places of truth — explicit rules are needed for what runs where |
There is no universally correct option: for one data mart, a scheduler is overkill; for a chain of ten dependent recalculations, scripts become a source of failures.
That is why in assessment we always break down how many branching points the process has and how often it will need to be restarted.
How we build orchestration: project stages
Before setting up orchestration, you need to understand how runs live now: where calculations rely on a manual start, where only one person knows the schedule, and where a failed nightly recalculation is noticed only from complaints in the morning. Then comes a six-step route that we follow on Apache Airflow.
-
Run discovery. We build an inventory: what tasks exist, who owns them, where data comes from and goes to, and where manual steps are. For each, we record duration, frequency, and failure points.
-
Dependency scheme. We build a graph: what actually depends on what, where parallel execution is acceptable, and where strict sequence is required. We remove unnecessary links — they lengthen the critical path and mask real failures.
-
Schedule setup. We tie runs to the moment data appears, not to a convenient hour. We align windows with source schedules so calculations do not run on an incomplete export and do not compete with other processes for resources.
-
Restart rules. We set the number of retries, the delay between them, and divide tasks into safe-to-rerun and those requiring a manual decision. Separately, we describe which failures page the on-call engineer and which resolve themselves.
-
Access and environments. Connections to storages, credentials, and permissions are moved into managed connections, without keys in code. We separate production and test environments so checks do not touch production data.
-
Result acceptance. We run a historical interval on real data, compare the result with a reference calculation, and see where tasks hang and what takes longer than expected to recalculate. We hand over the configured scheme together with on-call regulations and incident analysis.
Configurations, regulations, and handover documentation
An ML pipeline orchestration project ends not with a demonstration of a working graph, but with a package your team can continue working with independently. Below is what is physically handed over and where it lives after delivery.
- Working Apache Airflow configurations: scheduler and executor parameters, pools, queues, concurrent task limits — as files in the repository, without manual edits through the web interface.
- Task dependency scheme: a map of graphs with data sources, entry points, result passing between tasks, and binding to schedules and external triggers.
- Access rights and roles: separation by teams and environments, separate service accounts, the procedure for granting and revoking access to secrets and storages.
- Run documentation: how to add a new task, how to release a schedule change, what to check before moving to production.
- Failure response regulations: failure classification, restart procedure without losing already computed results, notification channels, response times, and escalation points.
- Observability settings: which metrics and logs are collected, where notification thresholds are set, and who receives them.
- Knowledge transfer materials: a walkthrough of the system design on the live environment together with your engineers.
The package remains workable: new tasks and rules can be added within your team without coming back to us for every change.
Case study: model retraining without manual restarts
The data preparation and training chain relied on manual runs: an engineer followed a checklist to bring up export, cleaning, and training, monitored the nightly window, and passed artifacts between steps.
Any failure — a failed export, a changed source schema, insufficient resources — stopped the entire chain until morning. As a result, more than a third of runs completed incompletely, and a mismatch between data and model versions surfaced already at the quality check stage.
What we changed
We described the preparation and training chain in Apache Airflow as an ML pipeline of dependent tasks with a schedule, retries, and timeouts.
Steps were made idempotent: each writes its result to versioned storage, so a retry after failure picks up ready intermediate artifacts instead of recomputing everything.
Before training, we added freshness and completeness checks; after training, we added recording of the dataset and model version.
Metrics before and after
The share of successful runs grew from 62% to 97%, and manual restarts decreased from several per week to isolated cases per month.
Failure response dropped from hours to minutes: the task retries automatically, and the engineer receives a notification only when retries are exhausted.
Checks cover all critical transitions between steps, so data and model desynchronization no longer reaches quality validation, and rolling back to a previous version is performed with a single run.
How Airflow differs from a custom scheduler
A custom scheduler holds up while there are two or three pipelines, dependencies are linear, and runs happen once a day. The logic usually lives in one script with sequential calls, the schedule is in cron, and knowledge of step order remains in one engineer's head.
The solution works, but it hits growth limits: each new step lengthens the script, and one day no one dares touch it.
The first recurring failure occurs on restart. A task fails midway — the run repeats entirely, duplicating already loaded data, and troubleshooting is manual.
To this are added waits via a pause inside the process, lack of retries with increasing delay, logs in different places, and zero history: why yesterday's run ended differently from today's cannot be reconstructed.
The orchestrator changes the model — dependencies between steps are described explicitly, not derived from line order.
Hence practical consequences: restarting only the failed task, automatic retries, running for missed intervals, state log for each step, environment isolation, and reuse of ready-made operators instead of copying code. Apache Airflow covers this with standard tools, not homemade add-ons.
Existing runs do not need to be migrated entirely. The working path is to wrap current scripts as tasks, keep the previous launch method as a safety net, and run both in parallel for some time, comparing results. Then the monolithic script is cut into steps with their own parameters and reruns.
The criterion for choosing a tool is simple: one linear pipeline is served by cron, but as soon as dependencies, different environments, and reproducibility requirements appear, a custom scheme costs more than orchestration.
Technical questions before project start
Before the project starts, the technical team asks not questions about the tool, but about how ML pipeline orchestration on Apache Airflow will fit into the working environment: what will happen to already written scripts, who will get access, and how calculations will behave on failure. Below are direct answers to what most often comes up during assessment.
Will existing scripts have to be rewritten for the new tool?
No. Existing scripts are connected as separate tasks; we describe their dependencies, schedule, and retry rules. Along the way, we remove what hinders migration: hardcoded paths, credentials inside code, manual launches from a developer's machine.
How is access differentiated within the team?
Access is separated by roles: pipeline developer, on-call engineer, log reader — each sees only their own area. Secrets are stored separately and injected at runtime rather than lying in code. The log shows which task and which account were used to run it.
What happens if a task fails midway?
The calculation resumes from the interruption point: computed fragments are not recomputed and are not duplicated in storage.
The number of retries and behavior after them are set by policy; if the failure repeats, the pipeline stops and notifies the on-call engineer instead of replicating the error across the entire data mart.
Will recalculation time increase after migration?
Recalculation is triggered by schedule and by event, and heavy steps are parallelized across executors. During assessment, we calculate by what factor the recalculation window will shrink for your volumes and verify this on a pilot section before the main implementation.
Let's discuss your training orchestration task
To come back with an estimate and work plan rather than general words, we need to see your run scheme as a whole: how training steps are connected, where they stop, and what happens to data between them. Below is a list of what is worth having on hand when we sit down to discuss the task.
- How many tasks are in the chain and how often it runs: on a schedule, on an event, or after a new batch of data appears;
- Where exactly it breaks: failed task, recalculation, manual restart — what is currently done by hand;
- How data is passed between steps — files, tables, intermediate exports — and who ensures that runs do not mix with each other;
- What is already automated and what relies on scripts and an on-call engineer;
- What data quality checks must pass before training starts and what to do if they fail;
- How you learn about a failure — email, chat, call — and how long after the failure begins;
- Whether rollback to a previous model version is needed and how long it should take.
Next, there is no need to describe everything at once. We will take one scenario — the most frequent or the most painful — and assemble it in Apache Airflow from start to finish: a trial run on one scenario will show how the system behaves on your data and schedule. Based on its results, we will return with an estimate and a work plan for the remaining chains.
