Auto-Retraining System for ML Crypto Trading

ML models for crypto trading degrade quickly: market regimes shift in hours, correlations break, regressors drift. After the latest Bitcoin halving, one client's models saw directional accuracy drop from 60% to 38% in a week—a loss of over $100,000. Manual retraining takes hours, but the market doesn't wait. We build automated retraining systems that detect degradation and launch a new training cycle without human intervention, preserving trade uptime. This automated retraining system uses multiple triggers to ensure timely model updates. Capital savings from timely response reach 30% (on average $50,000 for a $150,000 portfolio). With over 5 years of experience and 50+ deployed systems, our team has the expertise to deliver robust automation.

Trigger Types for Retraining

Performance-based trigger checks how often the sign of the prediction matches the sign of the actual price change over a rolling window. If accuracy falls below 52% with at least 100 predictions—retraining is triggered.

Optimal Retraining Frequency

Retraining frequency depends on market volatility and feature stability. For models on minute timeframes, retraining may be needed every 24–48 hours; for daily models, every 7–14 days. But a rigid schedule ignoring feature drift is risky. By combining a performance trigger and PSI, we reduce capital losses by up to 15% compared to scheduled-only retraining.

How We Detect Feature Drift?

For feature drift we use the Population Stability Index (PSI). According to Wikipedia - Population Stability Index, PSI is an industry standard for comparing distributions. We compute PSI for each feature between the last 30 days and a reference period. If PSI exceeds 0.25 for any feature—a trigger fires. In practice, PSI triggers detect drift 2× faster than simply checking accuracy.

Combining Triggers: Why One Is Not Enough

Performance triggers react only to accuracy drops—that's already a consequence. Feature drift often appears days before metric decline. By combining both approaches, we catch the problem early and reduce capital losses by up to 15%.

Automating Retraining in 5 Steps

1. Real-time monitoring—daily checks of triggers: performance, PSI, schedule.
2. Pipeline launch—Prefect/Airflow triggers a DAG when any trigger fires.
3. Data loading and preparation—load last 365 days from ClickHouse/PostgreSQL.
4. Training with walk-forward validation—5 folds, 60-day test, 24h gap. All experiments logged in MLflow.
5. Validation and hot swap—the new model is compared to the current one on accuracy, Sharpe ratio, and drawdown. If it passes, it replaces the old one without stopping signals.

How We Build the Retraining Pipeline

import mlflow from prefect import flow, task @task def fetch_training_data(symbol, lookback_days=365): """Load data for retraining""" end_date = datetime.utcnow() start_date = end_date - timedelta(days=lookback_days) # Load from ClickHouse/PostgreSQL return load_ohlcv_data(symbol, start_date, end_date) @task def prepare_features(raw_data): """Feature engineering""" from feature_pipeline import FeatureEngineer engineer = FeatureEngineer() return engineer.create_all_features(raw_data) @task def train_and_evaluate(features_df, target_col, model_config): """Train model with walk-forward validation""" from training import WalkForwardTrainer trainer = WalkForwardTrainer( n_splits=5, test_size=60, # 60 days test set gap=24 # gap between train and test (hours) ) with mlflow.start_run(): model, metrics = trainer.fit_evaluate(features_df, target_col, model_config) # Log metrics in MLflow mlflow.log_metrics(metrics) mlflow.log_params(model_config) mlflow.sklearn.log_model(model, 'model') run_id = mlflow.active_run().info.run_id return model, metrics, run_id @task def validate_and_promote(model, metrics, run_id, min_metrics): """Check quality and decide on deployment""" passes_validation = ( metrics.get('directional_accuracy', 0) >= min_metrics['accuracy'] and metrics.get('sharpe_ratio', 0) >= min_metrics['sharpe'] and metrics.get('max_drawdown', 1) <= min_metrics['max_drawdown'] ) if passes_validation: # Register as new Production version client = mlflow.tracking.MlflowClient() model_version = client.create_model_version( name='crypto_predictor', source=f'runs:/{run_id}/model', run_id=run_id ) client.transition_model_version_stage( 'crypto_predictor', model_version.version, 'Production' ) return True, model_version.version return False, None @flow(name="model_retraining_pipeline") def retrain_model_pipeline(symbol, model_config, min_metrics): raw_data = fetch_training_data(symbol) features_df = prepare_features(raw_data) model, metrics, run_id = train_and_evaluate(features_df, 'target', model_config) promoted, version = validate_and_promote(model, metrics, run_id, min_metrics) return {'promoted': promoted, 'version': version, 'metrics': metrics} 

Why Hot Swap Is Critical

Upon successful training, the old model must be replaced without stopping trading. We use asynchronous locking:

class ModelHotSwapper: def __init__(self): self.current_model = None self.model_version = None self._lock = asyncio.Lock() async def swap_model(self, new_model, new_version): """Thread-safe model replacement""" async with self._lock: old_model = self.current_model old_version = self.model_version self.current_model = new_model self.model_version = new_version # Log model change logger.info(f"Model swapped: {old_version} -> {new_version}") # Unload old model del old_model async def predict(self, features): async with self._lock: return self.current_model.predict(features) 

Scheduling and Orchestration

Prefect or Airflow runs a daily check pipeline at 00:00 UTC:

1. Check performance trigger
2. Check PSI drift trigger
3. Check schedule trigger (if > 7 days since last training)

If at least one trigger fires → retraining pipeline is launched. Upon successful training → hot swap model → notification via Telegram.

Walk-forward is a rolling cross-validation for time series. We split history into 5 sequential segments: each trains on earlier data, tests on the next 60 days. Between train and test windows there is a gap of 24 hours to prevent data leakage from temporal autocorrelation. This simulates real-world model performance.

Deliverables

• Design and implementation of trigger logic (performance, PSI, schedule)
• Integration with your data store (ClickHouse, PostgreSQL, S3)
• Prefect/Airflow DAG setup with monitoring and alerts
• MLflow experiment tracking and model versioning
• Zero-downtime hot swap implementation
• Documentation and team training
• 3-month support guarantee post-implementation
• Auto-deploy on successful validation

We offer turnkey implementation in 2–6 weeks. Contact us to evaluate your project and get a free initial assessment.