AIOps for Capacity Planning: Forecast Resources with Prophet

AIOps Setup for Capacity Planning (Resource Forecasting)

After launching a seasonal project, you notice that cluster nodes are hitting memory limits and the autoscaler isn't fast enough to handle the load — users get 503 errors. We've encountered this on projects with 500+ nodes: user churn, reputation damage. This is the problem of reactive scaling. We solve it with AIOps: we build forecasts based on historical metrics from Prometheus and machine learning. Instead of the crude "add 20%" approach, we use Meta's Prophet model — it predicts resource needs 30 days ahead with ±15% accuracy. Proper forecasting saves up to 30% of infrastructure budget, and business metrics (sales, number of users) improve accuracy by another 20–40%. A typical client with 50 nodes saves $5000 per month on cloud resources.

Which Metrics We Forecast and Where Data Comes From

The resources we plan first:

• CPU and memory on cluster nodes (Kubernetes node utilization)
• Disk space (especially for databases and logs)
• Network throughput
• Storage IOPS (critical for PostgreSQL, Elasticsearch)
• Number of pods and cluster limits

How AIOps Builds Forecasts: Prometheus + Prophet

Typical architecture for self-hosted:

Prometheus (Thanos/VictoriaMetrics for long-term storage)
↓
remote_read API
↓
Python forecasting service (Prophet / statsmodels)
↓
Forecasts written as metrics → Prometheus pushgateway
↓
Grafana dashboard with forecast overlay

Example script that reads node CPU utilization and builds a 30-day forecast:

from prophet import Prophet
import pandas as pd
from prometheus_api_client import PrometheusConnect

prom = PrometheusConnect(url="http://prometheus:9090")
data = prom.get_metric_range_data(
    metric_name='avg by (node) (1 - rate(node_cpu_seconds_total{mode="idle"}[5m]))',
    start_time=datetime.now() - timedelta(days=60),
    end_time=datetime.now(),
    chunk_size=timedelta(hours=1)
)

df = pd.DataFrame({"ds": timestamps, "y": values})
model = Prophet(seasonality_mode="multiplicative", weekly_seasonality=True)
model.fit(df)
future = model.make_future_dataframe(periods=30, freq="D")
forecast = model.predict(future)
# forecast[["ds", "yhat", "yhat_lower", "yhat_upper"]]

yhat_upper (upper bound of the confidence interval) is the "pessimistic" scenario, from which we recommend planning resource procurement.

Why Business Metrics Matter?

Technical metrics alone don't explain when load increases are expected. If you add business data — number of active users, order volume, marketing calendar (sales dates, seasonal peaks) — the forecast becomes more accurate. We create a regressor in Prophet for each business event:

model.add_regressor('is_black_friday') model.add_regressor('monthly_active_users') 

We take MAU data from the analytics database, events from the business calendar. The model itself determines how strongly each factor influences resource consumption.

What Mistakes to Avoid During Implementation?

A typical mistake is using only a week of data for forecasting. That's insufficient to identify seasonality. We recommend at least 4–8 weeks, ideally 3–6 months. The second mistake is ignoring business regressors. Without them, the model won't predict peaks from marketing campaigns. The third is relying solely on the forecast mean (yhat). Always use the upper bound (yhat_upper) for resource ordering to avoid shortages.

Kubernetes: From Forecast to Automatic Scaling

Capacity planning doesn't have to end with manual server procurement. In Kubernetes, forecasts can be connected to KEDA (Kubernetes Event-Driven Autoscaling) via the prometheus scaler — pods will scale proactively, before load arrives.

For node pools in managed clusters (EKS, GKE, AKS), we configure Cluster Autoscaler or Karpenter with a buffer calculated from the forecast. Karpenter is more convenient: it orders nodes of the needed type and size for the predicted load.

What Our Work Includes

• Audit of current metrics and SLOs, assessment of historical data depth (minimum 4–8 weeks, ideally 3–6 months)
• Setup of long-term storage if missing (Thanos, VictoriaMetrics, Grafana Mimir)
• Development of a forecasting pipeline using Prophet, optionally with business metrics
• Integration with KEDA and Cluster Autoscaler / Karpenter
• Configuration of Grafana dashboards with forecast overlay and alerts ("node will run out of memory in 14 days")
• Training your team on dashboard usage and forecast interpretation
• One month of post-project support

Example savings: for a client with a Black Friday peak load, we configured a forecast based on 12 months of metrics. Result: 25% cost reduction by eliminating overprovisioning (previously kept +40% as buffer). The Prophet forecast showed that a 15% buffer with business regressors was sufficient.

Final Work Process

1. Analytics: audit of metrics, SLOs, business calendar
2. Design: architecture selection (Thanos vs Mimir, Prophet vs statsmodels)
3. Implementation: pipeline deployment, dashboard setup, K8s integration
4. Testing: A/B forecast test on historical data, verification with real-time metrics
5. Deployment: production connection, alert configuration

Timelines and Cost

Basic forecast on existing stack — from 2 weeks. Full integration with business metrics and autoscaling — up to 6 weeks. Cost is calculated individually, depends on data volume and integration complexity. Order an infrastructure audit today — contact us to discuss details. Get a consultation and accurate estimate for your project.

Our experience: 7+ years in DevOps, 50+ infrastructure scaling projects, certified AWS and Kubernetes specialists. We guarantee forecast accuracy within 15% of actual consumption.