NTP setup: why server clocks drift apart
The quartz oscillator in a server is not a reference: its frequency depends on the temperature in the rack, CPU load, and how the hypervisor provides ticks to the virtual machine.
Without external correction, clocks drift by seconds per day, and the offset accumulates silently — services do not go down until the discrepancy crosses a critical threshold.
First to fail is authentication. Ticket-based protocols like Kerberos consider a ticket invalid if the timestamps on the client and controller differ by more than the allowed window — usually a few minutes.
Time-related authentication errors look like login denials where everything worked yesterday, and logs contain a dry entry without a reason. Next, shared resources, inter-service communication, and certificate expiration checks fall over.
Event log desynchronization breaks incident analysis: events from different nodes line up in the wrong order, the cause looks like the effect, and time correlation between the application, load balancer, and database collapses. Replication with timestamps behaves the same way — some operations are discarded as outdated.
NTP configuration returns a common time scale to nodes: the source hierarchy provides a reference, Chrony smoothly adjusts frequency instead of moving the hands, and monitoring offset and drift shows drift before it reaches applications. Below is how this is arranged and what exactly we check on the test bench.
What a unified time scale in infrastructure provides
Clock desync is not cosmetic but a source of failures that are hard to reproduce. Oscillators drift, virtual machines lose track of time during migration between hypervisors, and after a reboot a node comes up with arbitrary time.
First class — authentication: directory service tickets and short-lived sessions are rejected if the timestamp falls outside the allowed window. Second — distributed data: time-versioned replication, locks, and queues begin to conflict. Third — orchestration and certificate issuance: nodes with drifted clocks drop out of the cluster.
A separate cost is failure analysis. When application logs, metrics, and load balancer logs are on different scales, the incident chronology cannot be reconstructed: events from neighboring services cannot be correlated. A unified time scale restores correlation and cuts root cause search from hours to minutes.
We build a hierarchy of time sources and configure Chrony: several external references, a local server as the source for other nodes, polling discipline, and drift accounting.
Clock offset goes to monitoring with alerts, rather than being discovered after the fact, and configuration is rolled out via templates — a new node gets correct time on first boot.
The result is an eliminated class of failures, stability of distributed services, and correctness of metrics and logs that can be relied on during analysis.
Synchronization options: pools, own sources, hardware clocks
The choice of scheme is determined by three things: required accuracy, whether internet access is allowed, and how many sites need to be kept in a single time.
For a typical server fleet, an external pool is sufficient; if the perimeter is closed or time affects legally significant events, the scheme is built from an own source.
| Option | When it fits | Limitations |
|---|---|---|
| External time server pool | Regular services and internal systems without audit requirements | Accuracy depends on the communication channel; if internet is unavailable, clocks drift; public nodes do not guarantee availability |
| Own time server | Fleet of ten or more machines, need a single reference point and predictable latency | Requires redundancy and maintenance; without a second node, it becomes a single point of failure |
| Hardware time signal receiver | Need independence from the network and accuracy higher than network | Tied to installation location, antenna installation in a reliable reception area, separate hardware |
| Cascading synchronization scheme | Multiple sites: one node takes time externally, the rest from it | Error at the upper level propagates through the cascade; latency control needed at each step |
| Isolated perimeter | Closed networks without external access | Time is set by an internal reference: restart requires correct initialization; drift accumulates between verifications |
In practice, schemes are combined: a hardware receiver provides time to its server, which distributes it via cascade to sites, and segments without network access live in an isolated perimeter with periodic verification.
NTP setup: work stages from audit to handover
Work proceeds in one order — from fleet inventory to regulations by which the time service lives on without us. The order is fundamental: Chrony and ntpd are configured for the network topology, and without a segment map the configuration turns out random.
- Audit of current synchronization. We collect the fleet — servers, network equipment, hypervisors — and see who is already responsible for time: service startup parameters, allowed sources, rules on the port. Here we also record discrepancies between nodes — they show where the hierarchy has broken down.
- Time source scheme. We design levels: external pools as the reference, internal servers for distribution to segments, isolated perimeters — with their own source. This removes the main mistake when hundreds of nodes independently poll external pools.
- Service deployment. We install and configure Chrony (or ntpd, if compatibility requires it) using a single template: access, correction step, behavior when sources are unavailable.
- Clock offset verification. We compare each node with the reference, look at drift and how the service reaches tolerance after startup. Nodes that do not fit within tolerance are analyzed separately — usually the cause is virtualization.
- Load testing of the time service. We check whether source servers can handle simultaneous requests from the entire fleet and what happens if one of them fails.
- Regulations and monitoring. We describe how to connect new nodes, what to alert on — discrepancy, source loss — and how to act when time drifts.
- Handover to operations. We show your team diagnostics, hand over configurations and regulations, and stay in touch for the first period.
Documents and artifacts that remain with you
We assemble artifacts not "for a checkmark" but as a working set by which the time service can be brought up on new hardware, handed over to the on-call shift, and checked without our involvement.
Everything is in text form suitable for a version control system: configurations, diagrams, and instructions sit alongside the rest of the infrastructure and change through ordinary change review.
- Time service configurations — working Chrony or ntpd files with comments: which parameters control source selection, how the mode is set when communication is lost, what was changed relative to the standard profile.
- Source and redundancy scheme — a table of external and internal sources with levels, polling order, and failover rule, so that failure of one does not lead to offset accumulation.
- Access rules for the time server — which subnets and nodes may request time, which ranges are closed, how access is granted to a new service and revoked when a node is decommissioned.
- Offset monitoring regulations — which metrics are collected, at what frequency, which values are considered normal, and which are grounds for intervention during a shift.
- Instructions for the on-call shift — step-by-step actions when clocks diverge across a group of servers: how to distinguish a source problem from a node problem, how to shift load to the reserve, and what to record in the journal.
- Rollback procedure and acceptance protocol — previous version of configurations, the command to return to it, and control offset measurements before and after work, by which you accept the result.
Case: how clock discrepancy was eliminated in a distributed perimeter
What it was
A distributed perimeter: several sites, with authentication services, a message queue, and log collection between them.
Clock synchronization had once been set up as an afterthought: some nodes went to external sources, some lived on local clocks, and somewhere time was corrected manually after a virtual machine migration.
The offset spread between nodes reached tens of seconds, and events from logs of different sites physically could not be arranged into a single chronology.
This produced two classes of problems. First — authentication failures: the service issued a pass, and the receiving node considered it expired due to clocks running ahead; such denials accounted for about 4% of all incidents.
Second — analysis cost: to restore the order of events, an engineer spent 3-4 hours, and in some cases the cause remained unproven.
What was changed and what became
Sites were migrated to Chrony with a unified level scheme: local reference nodes at each site, external sources only for them, and other machines synchronized within the perimeter.
The configuration was distributed via configuration management so that manual edits on individual servers became impossible, and offset monitoring with an alert on threshold breach was added.
Metrics before and after: share of authentication failures due to time discrepancy — from 4% to zero over a quarter of observation, offset spread between nodes — from tens of seconds to fractions of a second, time to analyze authentication denials — from 3-4 hours to a few minutes, and more often analysis is not required at all.
The event chronology is assembled from logs without manual reconstruction. If external sources are unavailable, the perimeter maintains accuracy on local references and does not drift apart during the communication failure.
What to do if a time server already operates on the network
A working time server does not need to be stopped to switch to a new scheme. Time service migration is done over the existing one: new nodes are brought up alongside, they receive the same external sources and for some time only record the discrepancy without adjusting clocks.
First you get an honest picture of the fleet, then you enable correction — migration without downtime arises not as a trick, but as a consequence of the old and new perimeters operating in parallel for some time.
Switching to Chrony is justified where fast convergence after downtime and resilience to intermittent channels are needed: the service holds frequency more accurately and behaves more gently when some sources are unavailable.
Then it is not about the daemon, but about a heterogeneous server fleet: some machines still run an old time client, others live in containers, still others are virtual, and their clocks depend on the hypervisor.
Bringing everyone to the same rules is possible only through a unified configuration layer: one source list, the same time zone, the same update order and method.
Staged switching proceeds in descending order of criticality. First we migrate environments and groups that can survive a service restart in a standard window — on them we also check behavior when sources fail and with a noticeable clock rate difference.
Only after that do we touch nodes where incorrect time means authorization failure, replication denial, or log discrepancy between services. As long as the previous sources are alive and the old daemon is not removed, the group can return to the previous scheme in minutes.
Upon completion, each node has one time service and one set of settings, and offset and the share of synchronized machines are visible on graphs. Describe your topology and group order — we will return with a migration plan and estimate.
Common technical doubts about time synchronization
Doubts about time synchronization arise not during setup, but during handover to operations: you need to understand system behavior when a source fails and the fate of already running applications. Below are questions raised in almost every project.
What accuracy is achievable on virtual servers?
On physical hardware, the NTP/Chrony pair keeps discrepancy within microseconds. In a virtual environment, scheduler jitter and host-side delays are added, so the real order of accuracy is milliseconds, and it depends on hypervisor neighbors.
We measure offset before migrating services so that accuracy is based on measurement, not on datasheet figures.
What happens if the time source stops responding?
The client continues to count time using its own reference oscillator and slowly drifts, remaining operational. In the configuration we keep several independent sources, surface unavailability to monitoring, and when the reference returns, time is pulled smoothly. A reverse jump is disabled intentionally — otherwise queues and timestamped records break.
Is it acceptable to rely on public pools?
As a backup level — yes, as the only source — no: pool availability and its server composition are outside your control. For critical services we raise a reference layer inside the perimeter and leave external sources for its failure. This also removes the question of sending service traffic outside.
How long does migrating a server fleet take?
The configuration is rolled out by management tools in one cycle — that is hours, not weeks. The main time goes to windows for applications sensitive to a time jump and to the order of work: first the reference layer, then clients by groups. A fleet of hundreds of machines is migrated without stopping services.
Let's discuss your time synchronization task
To make the estimate for NTP setup on servers specific rather than "plus or minus", we need inputs on the infrastructure. They determine the choice of time source, the distribution scheme within a site, and how many nodes to migrate to the new configuration.
The more precise the inputs, the fewer assumptions in the work plan and the more realistic the timelines.
- Fleet composition: number of servers, operating systems and their branches, physical or virtual machines, whether there are clusters and containers. Virtualization often adds its own drift, and this changes the pull-in scheme.
- Topology and network restrictions: segments, firewalls, NAT, availability of external sources, presence of a local reference server. This determines whether we pull each node outside or build an internal hierarchy.
- Accuracy requirements: what spread is acceptable for logs, billing, cluster quorums, certificate checks, and authentication protocols. The difference between "seconds" and "milliseconds" means different sources and different monitoring scope.
- Time source: public pools, own signal receiver or precise time domain, whether source authentication is needed. We separately record whether any nodes need increased stability.
- Current state: what is already configured, who currently distributes time, whether discrepancies are caught, whether there have been cases of manual clock correction. This hints at where hidden problems have accumulated.
- Work windows: whether a short step offset is acceptable or only smooth pull-in, when the service can be restarted and configuration touched. For databases and clusters this is critical.
- Monitoring and responsible parties: where to send offset and drift metrics, who confirms the result at acceptance. Without this, desynchronization will return unnoticed.
Describe the task with these inputs — we will return with a work plan, estimate, and sequence of actions.