SMART Monitoring Setup for Predicting Disk Failures

Disks warn of their death in advance. But by default, no one reads this data.

On one project, we recorded a silent increase in Reallocated Sectors across the entire Ceph cluster. The monitoring system reacted only when two OSDs went down simultaneously — a night of recovery and several hours of production downtime. After implementing SMART monitoring with Telegram alerts, such a situation never recurred. In 1-2 days, we set up turnkey SMART monitoring integrated with your alerting system. Our experience — over 7 years and 50+ projects, guaranteeing timely problem detection. Savings from timely disk replacement can reach 90% of data recovery costs.

What is SMART and which attributes matter

SMART (Self-Monitoring, Analysis and Reporting Technology) is built-in diagnostics for HDDs and SSDs. Each drive maintains statistics, some of which are reliable failure predictors. According to Wikipedia, SMART attributes can predict up to 60% of failures.

How to distinguish normal from critical values?

Normal is 0 or close to 0 for error attributes, except NVMe Percentage Used (should be < 100%). Growth of any attribute over a day is a signal to replace. For example, a 5% increase in Raw_Read_Error_Rate in one day is a reason to expedite replacement. For NVMe, watch Available Spare: if it drops below 10%, the disk may fail within weeks. For HDDs, a growth of 10 or more Reallocated Sectors Count over a short time almost guarantees failure within a month.

Why smartctl_exporter is 5x better than smartd

We recommend smartctl_exporter for infrastructures with 5+ servers — it is 5x more effective for centralized monitoring. For single machines, smartd with email notifications suffices.

Tooling: smartmontools + smartctl_exporter

On Linux, the base tool is smartmontools. For monitoring via Prometheus, we use smartctl_exporter:

# Install smartmontools
apt install smartmontools -y

# Manual disk check
smartctl -a /dev/sda

# Run extended self-test (takes hours, does not interrupt operations)
smartctl -t long /dev/sda

smartctl_exporter periodically polls all disks and exports metrics in Prometheus format:

---
# docker-compose.yml services: smartctl-exporter: image: prometheuscommunity/smartctl-exporter:latest privileged: true volumes: - /dev:/dev:ro ports: - "9633:9633" restart: unless-stopped 

The privileged: true or cap_add: [SYS_RAWIO] flag is mandatory — smartctl requires direct device access.

Alerting Rules

groups:
  - name: disk.smart
    rules:
      - alert: SmartReallocatedSectorsIncreasing
        expr: |
          increase(smartctl_device_attribute{attribute_name="Reallocated_Sector_Ct", attribute_value_type="raw"}[24h]) > 0
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "Reallocated Sectors increasing on {{ $labels.device }} ({{ $labels.instance }})"
      - alert: SmartCriticalError
        expr: |
          smartctl_device_attribute{
            attribute_name=~"Reported_Uncorrectable_Errors|Current_Pending_Sector_Ct",
            attribute_value_type="raw"
          } > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Critical SMART attribute on {{ $labels.device }}"
      - alert: SmartDeviceFailed
        expr: smartctl_device_smart_healthy != 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "SMART test failed on {{ $labels.device }} ({{ $labels.instance }})"

for: 0m for critical attributes — we notify immediately without waiting. This is not a flapping candidate.

smartd as an alternative without Prometheus

If Prometheus is not in the project, we configure smartd (the daemon from the smartmontools package). It monitors disks itself and sends email on issues:

# /etc/smartd.conf
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../6/03) \
-W 4,45,55 -m [email protected] -M exec /usr/share/smartmontools/smartd-runner

-W 4,45,55 — alert at temperature above 45°C (warning) and 55°C (critical). -s S/../.././02 — short test nightly at 2:00 AM, -s L/../../6/03 — long test every Saturday at 3:00 AM.

Setup Process

We implement SMART monitoring in several stages:

1. Analysis of the current drive fleet — HDD, SSD, NVMe, their quantity and location.
2. Choice of approach: smartctl_exporter (if Prometheus is present) or smartd (if not).
3. Installation and configuration: setting up the daemon, adding to Prometheus scrape config.
4. Creating Grafana dashboards with key metrics and trends.
5. Configuring alerting rules in Alertmanager with integration into Telegram/Slack.
6. Documentation and training for your team.

Timeline: setup on one server — 2-3 hours; on a fleet of 20 servers via Ansible — one business day with testing.

What's included in the work

We provide:

• Installation of smartctl_exporter or configuration of smartd for your scenario.
• Integration with Prometheus and Alertmanager, Grafana dashboards.
• Documentation on attributes and alerting rules.
• Training for your engineers on responding to alerts.
• 1 month of support after implementation.

Contact us for a consultation — we can help set up disk monitoring turnkey and avoid sudden failures.