Why UPS monitoring does not trigger at the moment of outage
The rack is powered through an uninterruptible power supply, but the switch to batteries is discovered not by the server but by the on-call engineer — and often only after connectivity has already been lost.
The channel between the UPS and the operating system is either not up or only half up: the device reports its state, but this data does not reach the node that must shut down correctly.
The causes are usually mundane.
The driver is bound to the wrong port after the device was renamed, the monitoring service — NUT or apcupsd — is running on the hypervisor while guest machines are not notified about battery operation, or the daemon started without a connection to the UPS and silently runs idle.
From the outside the picture looks fine: the last known state is cached, and new events do not arrive.
A separate trap is battery discharge. The low-charge threshold is set to a remaining capacity that should be enough for a graceful shutdown, but under load the capacity drops faster, and the self-test does not run on schedule.
The level does not have time to reach the threshold — and the server shuts down suddenly. Polling infrequently, once every few minutes, does not catch short sags at all.
The flip side is false alarms: narrow thresholds without hysteresis, battery tests during peak load hours, notifications for every flicker. The on-call engineer gets used to the noise and misses a real power outage.
Next we will break down how to assemble an alerting and correct shutdown chain that does not lie in either direction.
What managed server power control provides
A UPS without monitoring keeps the load until the batteries are fully discharged and goes down together with it.
At that moment the server does not know the line is gone: the filesystem journal remains unclosed, database transactions remain incomplete, and data hangs in a buffer that can no longer be flushed. Managed power control turns this into a predictable scenario with a known reaction time.
Technically it looks routine: the NUT service polls the UPS, publishes events, and runs scripts. Power loss, switching to batteries, low charge, line restoration — each event becomes a notification and an action on the server, not an incident discovered in the morning.
- Threshold instead of zero. The rule "five minutes remaining" lets services close normally rather than cut off at the last percent of charge.
- Managed shutdown. The script stops databases, flushes queues and cache, unmounts volumes — hard power loss is replaced by a correct stop.
- A whole class of failures disappears. A damaged journal, lost indexes, night-time restore from backup — consequences of a single event that is now handled according to procedure.
- Downtime is measured in minutes. The server comes back up by itself after power returns: the downtime is set by the script, not by battery discharge.
- Event notifications. The responsible person gets a signal at the moment of transition, rather than arriving to cold hardware.
- Regular verification. Once a quarter — a test line shutdown: we make sure thresholds, scripts, and alerts trigger.
Put together, the checklist changes not the equipment but the system's behavior during a failure: power, data, and alerts stop being independent storylines.
Options for connecting a UPS to the monitoring system
The scheme for collecting events from the power source determines which failures you will see at all.
Consumer models report only "power lost / restored," while overheating, wear and battery replacement, and operation at the charge limit are available only over a digital channel — USB, serial port, or polling via a network card.
Open drivers such as NUT cover all these options with a single configuration, so the choice is not the tool but the topology: where the agent lives, who polls the UPS, and what happens to events when the network or the host itself goes down.
| Scheme | When it fits | Limitations |
|---|---|---|
| Direct connection to the host | One UPS and one server, simple circuit | The agent runs on the protected host: if it hangs, both events and the correct shutdown command are lost |
| Local agent on each host | Multiple servers, independent shutdown matters | The driver monopolizes the port, settings are duplicated, there is no overall view of the UPS fleet |
| Network polling | UPS with an expansion card, many consumers | Models without a card are not suitable, some parameters are unavailable, collection depends on network availability |
| Intermediate node | Multiple UPSes and servers, a single collection point | A separate gateway host is needed, plus redundancy for the node itself |
The intermediate node usually wins: the UPS is connected to it directly, and the servers receive events over the network — the failure of any of them does not prevent collecting statistics and sending shutdown commands.
Implementation procedure: from survey to handover
The work proceeds in a single pass: by the end, the customer is left not with a "configured daemon on a server" but with a reproducible procedure — what is polled, under what conditions the reaction triggers, and how to verify it without stopping services.
- Infrastructure survey: inventory of power sources and powered nodes — model, connection type, availability of the management interface, number of servers on each UPS.
- Load scheme: map nodes to sources, mark criticality and acceptable downtime, determine the shutdown order — what to shut down first, what to keep until the last.
- Polling design: polling layer (NUT or apcupsd), sampling interval, charge and voltage thresholds, delays — so the reaction does not trigger on short sags.
- Reaction scenarios: on-call notification, delayed service shutdown, cancellation of the scenario when power returns, logging of every event.
- Trigger verification: a test shutdown in an agreed window, measurement of actual battery runtime, log review, threshold adjustment based on the results.
- Observability: metrics go to the common monitoring system, and silence from the polling agent is considered a separate cause — a missing poll is visible just like an outage.
- Handover to operations: configurations in the repository, power scheme and on-call instructions, review of typical events with the team.
The output is a clear "node → source → reaction" matrix and a verified scenario that will work the same way during a planned test and a real emergency.
Configurations, diagrams, and procedures: what we hand over
A working service is only half the result. The other half is a package of artifacts that allows UPS monitoring to be maintained and handed over between shifts without us: from configuration to the procedure for actions when switching to batteries.
- Configuration files for the monitoring and notification service with comments: which parameters to change when replacing batteries and changing the UPS model, and which should remain unchanged.
- Connection diagram: which servers are behind which UPS, how power and signal lines are routed, where communication goes over the network and where over a serial port.
- Notification rules with grading: switching to batteries, low charge, critical charge, power restoration, loss of communication with the UPS — each with its own route and urgency level.
- Access roles: who can see status, who restarts the service, who changes thresholds and notification rules, who is responsible for the UPS's own credentials.
- On-call shift instructions: the procedure for each type of notification, including correct server shutdown before battery discharge and startup after power restoration.
- Verification procedure: how to make sure monitoring is alive without removing load — a test event, checking logs and the time of the last poll.
- Acceptance checklist with expected behavior for each scenario: battery replacement, UPS model change, network loss, unavailability of the monitoring server itself.
We hand over in a form suitable for transfer to another engineer or contractor: configurations under version control, diagrams and procedures in the team's shared documentation.
Case: eliminated false shutdowns on 40 servers
The fleet is 40 servers in four racks at two sites. Some hosts polled the UPS via NUT, some via apcupsd, and for the rest monitoring amounted to a ping.
Configs lived separately on each machine, events were not collected anywhere, and the on-call engineer learned about an outage from a phone call.
During a power sag, the shutdown scenario did not have time to coordinate hosts: some shut down the load immediately, others wrote to disk until the very end, and some virtual machines went into a hard stop.
Recovery took up to 40 minutes of manual reconciliation — which nodes to bring up and in what order — and events about switching to batteries were lost: the local log was overwritten in a loop and never left the host.
What was changed
UPS polling was consolidated to a single point: an agent on each host reports charge, temperature, load, and remaining runtime to a common collector; APC models are read by the native driver, and the rest over the network interface.
Events no longer stay only on the host: they go to a queue and from there to storage, so restarting the service itself does not erase history. Shutdown scenarios are coordinated by groups — databases and file services first, then the rest, with a delay for completing operations.
What the result was
False shutdowns stopped: the threshold is calculated by charge and remaining runtime, not by the mere fact of switching to batteries. Recovery after a failure is 6–8 minutes, the startup order is set by policy, and the state of all nodes is visible on a single dashboard.
There are no event losses: over the quarter, not a single missed transition, including outages in racks where UPSes are installed in pairs and switch over one after another.
How does network polling differ from a local agent on the server?
The difference is where the UPS driver lives. With network polling, state is read from a managed card via SNMP or HTTP: the collector retrieves charge, load, and voltage every few seconds.
A local agent — for example, a driver from Network UPS Tools — runs on the server itself and holds the device via USB-HID or a serial port.
Network polling adds scheme dependencies on the network: switch, route, port, and credentials between the collector and the UPS. But nothing is installed on hosts and no permissions are granted — a single collection point sees all devices as ordinary network devices.
But if the card is powered by the same UPS, the polling channel also disappears during a power outage.
A local agent instantly sees the switch to battery and can itself stop services and shut down the host before the charge is exhausted.
The price is dependencies on the server side: device passthrough into a virtual machine, port permissions, udev rules, and checking that a second driver has not grabbed the UPS.
Event delivery reliability is limited in different ways. A local agent reports current state even when communication with the monitoring system is broken — only visibility is lost.
Network polling is discrete: a short peak between polls will not make it into the sample, and if the channel is lost, events are not buffered and are recorded only as unavailability.
Therefore, choosing the collection mode is a distribution of roles: a local agent on critical hosts where automation is needed, network polling for an overview of the rack fleet.
A hybrid on a single UPS requires explicitly assigning the device owner, otherwise collectors conflict over the port.
Common technical doubts about a power control system
Doubts about power control are usually engineering-related rather than organizational: the source fleet is heterogeneous, no separate host has been allocated for the service, and no one is ready to test the shutdown policy on live racks. Below are the things asked most often before work starts.
We have a mixed UPS fleet — can you bring everything into one picture?
Yes. For each source, a data collection method is selected: USB interface, serial port, manufacturer's network module, or dry contact.
NUT brings heterogeneous equipment to a common variable model — battery charge, "on line / on battery" mode, load, temperature — so in monitoring and notifications the fleet looks uniform. Where the source reports only a binary flag, the state still makes it into the overall picture.
There is no separate host — can the service be placed alongside an existing one?
Yes. The service is installed on an already running node and needs minimal resources.
More important is something else: the initiator of the correct shutdown must not itself depend on the controlled power, so we assign the leading node role to a physical machine, not a VM under the same hypervisor. The remaining hosts connect to it as clients.
Events are lost when communication breaks — what do you do about that?
Every state change is first written to the node's local log, and only then sent to the monitoring system. When the network is unavailable, records accumulate and are delivered after communication is restored; retries are rate-limited, and duplicates are filtered by event ID. Separately, we configure which events are considered critical and require immediate notification.
How can scenarios be tested without actually shutting down racks?
Testing proceeds step by step: first a test switch to battery with delay monitoring, then simulating power loss for a selected node without load, then running the shutdown policy on a test server and checking that services come back up by themselves after power is restored. This verifies timers, shutdown order, and notification delivery — without risk to production.
Let's discuss the task: we'll come back with an estimate and work plan
To discuss a UPS monitoring task concretely, a short input is enough — we will clarify the rest ourselves. Send what you already have on hand: the composition of the server fleet and UPS models, how they are connected now, and what should happen during a power outage.
- List of servers and nodes powered by UPS — with each one's role in the circuit.
- UPS models and their interfaces: network port, USB, or dry contact only.
- Current monitoring method: manual rounds, the manufacturer's standard utility, or none.
- Network diagram: which segments, whether there are isolated sites and whether they can see each other.
- Where notifications should go: email, on-call shift chat, alerting system.
- What to do on discharge: shut down services correctly, transfer load, or only report.
- Constraints: reboot windows, equipment access procedure, logging requirements.
In response, we will return an estimate and plan: a data collection scheme for each type of equipment, a list of installation and configuration work, and a verification procedure — with a test power removal on one node to see the entire chain from event to notification. We will break the work into stages so it can be launched in parts.