Training XGBoost, LightGBM, CatBoost on Tabular Data

The client changed requirements late in the project: instead of binary classification, multiclass was needed. We had to rebuild the pipeline from scratch. Gradient boosting came to the rescue — XGBoost, LightGBM, and CatBoost. In a bank credit scoring project, we faced class imbalance and 500+ features. Boosting achieved an AUC of 0.92 without deep learning. These algorithms are workhorses for tabular structured data. They dominate Kaggle and production where neural networks falter: little data, many categorical features, need for interpretability. They are effective on tabular data with thousands of features, require no normalization, and are robust to missing values.

A typical scenario: data is a table with missing values, categorical columns with high cardinality, and an imbalanced target. Gradient boosting with proper tuning beats linear models and random forest by 3–5% in AUC. We tested hypotheses on historical data — gains are stable.

What Problems We Solve

Suboptimal hyperparameters reduce AUC by 3–5%. We use Optuna with early stopping: 100 iterations in 10–30 minutes on 1M rows.

Categorical features with high cardinality. One-Hot Encoding creates sparsity. CatBoost handles this natively; for LightGBM we apply Bayesian target encoding.

Interpretability for business. SHAP analysis decomposes the prediction into feature contributions — a mandatory requirement for credit scoring or medicine. As noted in the original paper: XGBoost: A Scalable Tree Boosting System (Chen & Guestrin, 2016).

Class imbalance is another typical problem. Boosting with weighted samples and early stopping helps maintain quality on rare events.

Why These Algorithms?

LightGBM wins in speed: it trains 2–3 times faster than XGBoost on datasets from 100K rows. Whether you choose XGBoost, LightGBM, or CatBoost, proper tuning is essential. XGBoost is more stable on sparse data and gives smoother predictions. CatBoost requires no encoding of categories — just specify the cat_features list. In an ensemble, they cover each other's weaknesses: stacking yields 0.5–2% AUC improvement over the best single model. Computing resource savings of up to 40% when using LightGBM.

How We Do It

Stack: Python 3.11, LightGBM 4.0, XGBoost 2.0, CatBoost 1.2, Optuna 3.5. For large data we use Dask or Spark, where boosting runs via distributed API. Typical case: churn prediction with 1M rows and 200+ features. After tuning num_leaves=127, learning_rate=0.03, subsample=0.8, AUC increased from 0.82 to 0.87. The pilot project pays off through improved prediction accuracy.

How to Avoid Overfitting?

Early stopping and regularization are key. Parameters reg_alpha, reg_lambda, min_child_samples control model complexity. We use 5-fold cross-validation with monitoring of the metric on validation.

Performance Comparison

SHAP Analysis in Detail: SHAP analysis decomposes the prediction into contributions of each feature, allowing understanding of which factors influence the result. This is critical for business tasks with transparency requirements, such as credit scoring or medical diagnostics.

Process

1. Analytics. Study distributions, outliers, correlations.
2. Feature engineering. Generate features based on business logic (moving averages, cross-tables).
3. Hyperparameter search. Optuna with 100–200 iterations, StratifiedKFold cross-validation.
4. Training and validation. Evaluate AUC, precision-recall, calibration.
5. Interpretation. SHAP summary plot and dependence plots for top-10 features.
6. Deployment. Export to ONNX or PMML, REST API on FastAPI, drift monitoring.

Implementation

LightGBM: Full Pipeline

import lightgbm as lgb from sklearn.model_selection import StratifiedKFold from sklearn.metrics import roc_auc_score import optuna def train_lgbm_with_cv(X: pd.DataFrame, y: pd.Series, n_splits: int = 5) -> lgb.LGBMClassifier: def objective(trial): params = { 'n_estimators': trial.suggest_int('n_estimators', 100, 1000), 'num_leaves': trial.suggest_int('num_leaves', 20, 300), 'max_depth': trial.suggest_int('max_depth', 3, 12), 'learning_rate': trial.suggest_float('learning_rate', 0.005, 0.1, log=True), 'subsample': trial.suggest_float('subsample', 0.6, 1.0), 'colsample_bytree': trial.suggest_float('colsample_bytree', 0.6, 1.0), 'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10.0, log=True), 'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10.0, log=True), 'min_child_samples': trial.suggest_int('min_child_samples', 5, 100), } cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) scores = [] for train_idx, val_idx in cv.split(X, y): model = lgb.LGBMClassifier(**params, random_state=42, verbose=-1) model.fit( X.iloc[train_idx], y.iloc[train_idx], eval_set=[(X.iloc[val_idx], y.iloc[val_idx])], callbacks=[lgb.early_stopping(50, verbose=False)] ) pred = model.predict_proba(X.iloc[val_idx])[:, 1] scores.append(roc_auc_score(y.iloc[val_idx], pred)) return np.mean(scores) study = optuna.create_study(direction='maximize') study.optimize(objective, n_trials=100, n_jobs=4) best_model = lgb.LGBMClassifier(**study.best_params, random_state=42) best_model.fit(X, y) return best_model def explain_model(model, X: pd.DataFrame): import shap explainer = shap.TreeExplainer(model) shap_values = explainer.shap_values(X) shap.summary_plot(shap_values, X, plot_type="bar") top_feature = X.columns[np.abs(shap_values).mean(0).argmax()] shap.dependence_plot(top_feature, shap_values, X) 

CatBoost with Categorical Features

from catboost import CatBoostClassifier, Pool def train_catboost(X_train: pd.DataFrame, y_train: pd.Series, X_val: pd.DataFrame, y_val: pd.Series, cat_features: list[str]) -> CatBoostClassifier: train_pool = Pool(X_train, y_train, cat_features=cat_features) val_pool = Pool(X_val, y_val, cat_features=cat_features) model = CatBoostClassifier( iterations=1000, learning_rate=0.03, depth=6, l2_leaf_reg=3.0, bootstrap_type='Bayesian', bagging_temperature=1.0, eval_metric='AUC', use_best_model=True, early_stopping_rounds=100, random_seed=42, verbose=100 ) model.fit(train_pool, eval_set=val_pool) return model 

How to Configure Stacking for Maximum Accuracy?

from sklearn.ensemble import StackingClassifier from sklearn.linear_model import LogisticRegression stacking = StackingClassifier( estimators=[ ('lgbm', lgb.LGBMClassifier(**lgbm_best_params)), ('xgb', XGBClassifier(**xgb_best_params)), ('catboost', CatBoostClassifier(**cat_best_params, verbose=0)), ], final_estimator=LogisticRegression(C=0.1), cv=5, stack_method='predict_proba' ) stacking.fit(X_train, y_train) 

Common Mistakes When Training Boosting Models

• Ignoring categorical features: use CatBoost or proper encoding.
• Insufficient validation: StratifiedKFold for imbalanced datasets.
• Overfitting: early stopping and regularization (reg_alpha, reg_lambda).

What's Included (Deliverables)

• Optimized model with documented hyperparameters.
• SHAP report with top-10 features and their impact.
• REST API or ONNX export.
• Deployment to production + monitoring (data drift, metric degradation).
• Client team training.

Estimated Timeline

From 5 working days for a prototype to 3 weeks for a production pipeline. Contact us for a consultation on your project. Order a pilot project to evaluate metric improvements. Cost is calculated individually. Get a consultation on your project — our engineers will help you choose the optimal approach. Typical pilot project cost starts from $5,000.