Network diagnostics: what breaks and how it is found
Intermittent application timeouts are the most unpleasant class of failures: the service responds in fractions of a second, then hangs for half a minute, and on a repeated request works normally.
Delays grow during peak hours, some connections drop, but the application logs are clean — the break occurred lower down, on the route. The incident is closed with the wording “it resolved itself,” although the cause has not gone anywhere.
An unstable connection rarely manifests as a complete outage. Packet loss of a fraction of a percent is almost invisible in monitoring, but it forces the transport to retransmit data: throughput drops severalfold, queues grow, and the application falls into retries and timeouts.
One server from the pool begins to respond more slowly than the others, and the load balancer spreads the problem across the entire service.
Network diagnostics is built from the outer perimeter to the application.
traceroute and mtr show on which segment latency grows and where packets are lost; iperf measures pure throughput between two points and separates the network from the code; nmap checks port availability and filtering rules; tcpdump on both ends of the connection lets you compare what was sent and what arrived, while Wireshark breaks down a single slow session step by step.
The task is to find the bottleneck on the route, not to raise timeouts in the config and wait for the next peak. Next, we break down how this is closed in practice: from taking measurements to changes on the network side.
Measurable effects: delays, loss, recovery time
While the cause is being sought based on complaints, the count is in hours.
Constant traffic capture on problem segments and route tracing move incident localization time from 4–6 hours to 20–30 minutes: the data has already been captured, and all that remains is to correlate it with the time of the complaint, rather than reproducing the failure again with a request to “reproduce the problem.”
Packet loss and response delay cease to be one-off measurements at the moment of an incident.
Continuous measurement along key routes with explicit thresholds gives notification earlier than users — the line maintains 0.1–0.5% loss instead of 2–5%, which for years was blamed on “bad internet” and the subscriber's weak equipment.
The class of failures is also changing. The category of “searched, did not find, rebooted” goes away: in its place is a specific cause tied to a segment and time, and a clear owner of the change.
The share of incidents that are closed within the team without escalation to the provider or vendor grows from half to 85–90%, and repeat requests for the same cause almost disappear.
Infrastructure observability also works over the long haul.
The duration of degradation for users is reduced severalfold: they learn about channel narrowing or rising response latency from a metric, not from support calls, and manage to switch the route before the business notices it.
Each analyzed incident leaves behind an alerting rule — the next similar case is caught automatically.
Network diagnostics formats: analysis, monitoring, procedure
The diagnostic format is chosen not by urgency, but by what is needed at the output: the cause of a specific failure, a confirmed fact of availability, or the current state of the path.
A one-off incident investigation answers the question “what broke now,” constant observation — “when it will break again,” and a check procedure — “has it degraded unnoticed.”
How to choose a format
| Option | When it fits | Limitations |
|---|---|---|
| One-off incident investigation | Degradation or outage has already occurred, and the cause is needed, not statistics | Sees only the moment of capture: an intermittent failure will have to be caught again |
| Data capture from equipment | It is unclear on which segment packets are lost or latency grows | Requires access to nodes and a maintenance window; recording depth is limited by the buffer |
| Availability monitoring | A signal is needed earlier than a user reports the problem | Shows the fact of failure but not the cause — investigation still requires capture |
| Inspection procedure | Regular verification of routes, loss, and configurations on a schedule | Catches only what manifested at the time of the check |
For each scenario, we fix a metric and a threshold beyond which investigation is triggered: without a threshold, observation turns into noise, and the procedure into a formality.
In practice, approaches are combined — data capture from equipment provides the initial picture, then the depth is chosen. The customer receives not a scattering of exports, but a network diagram with the failure point marked and a clear sequence of actions.
Work stages: from capturing symptoms to handing over the procedure
Network diagnostics begins not with launching an analyzer, but with recording symptoms: without a timeline and an understanding of what changes preceded the failure, data capture yields a voluminous dump without an answer.
Below are the diagnostic stages by which we conduct work: from collecting symptoms to the procedure that remains with your team.
- Collecting symptoms and timeline. We clarify what exactly is degrading — latency, loss, drops — on which nodes and at what intervals, how it is reproduced, and what changed before the first manifestation.
- Network and traffic path diagram. We build a diagram of traffic flow: segment boundaries, load balancers, address translation, tunnels. This shows where a packet can be lost or go around the expected route.
- Data capture points. We choose key points — at the ingress and egress of the problem segment, on the node itself and on a neighboring one. Dumps and measurements are taken synchronously, with a unified time reference.
- Inspection plan. We rank hypotheses by probability and cost of verification: first we rule out cheap versions, then move on to load measurements and tracing.
- Hypothesis testing. We compare data from different points, check interface counters, confirm or discard a version. Each hypothesis gets an unambiguous conclusion — “confirmed” or “excluded.”
- Recording the result and handing over the procedure. We describe the cause, the configuration change made, the set of control checks, and the thresholds at which it is worth returning to diagnostics.
Output materials: diagrams, measurements, procedures
Diagnostics ends not with a report, but with artifacts that your team works with further without us. Below is the minimum set: with it, you can repeat a measurement, explain service behavior, and onboard a new engineer into the loop.
- Route diagram: addressing, gateways, filtering rules, external exits, and zones between which traffic is allowed.
- Traffic maps: who communicates with whom, over which ports, what share is service traffic and what share is application traffic.
- Recorded measurements: latency, loss, and their spread across key directions — with date, node, and capture method.
- Data capture rules: from which nodes, with what, and in what window to take traffic snapshots so that comparisons are correct.
- Catalog of typical failures: symptom, probable cause, what to check first.
- Sequence of actions during degradation: order of checks from channel to application, with stopping points.
- Escalation route: whom to go to if the bottleneck is on the provider's or external service's side.
- Knowledge transfer: a review with your team and a recording of the review so that the loop remains manageable.
We prepare documents with a version and date: a diagram and measurements without a time mark cease to be evidence within a month after handover.
Case study: how periodic drops were localized
A service on several nodes dropped connections three to five times a day: sessions broke in the middle of an operation, degradation lasted from five to twelve minutes and passed on its own.
Standard monitoring systems recorded only a spike in latency, and application metrics did not show where exactly packets were being lost. Users blamed it on “unstable internet,” and the team blamed load.
What the measurements showed
We launched continuous measurements along all segments of the route: mtr gave loss broken down by hop, tcpdump on both sides of the connection — retransmits and duplicate acknowledgments, analysis of dumps in Wireshark — a timeline of each drop.
Neighboring segments were clean; loss occurred only on one intermediate node and always coincided in time with traffic peaks. iperf between adjacent segments confirmed that throughput dropped threefold on this very segment.
The cause turned out to be a mismatch in port mode and buffer overflow: under peak load, the node silently dropped packets without exposing error counters in the usual metrics.
After bringing the port settings to a matched mode, replacing the cable assembly, and enabling overflow control, the picture changed: the retransmission share fell from 3% to 0.02%, and drops stopped completely.
Before remediation, degradation lasted up to 12 minutes and recurred daily; after — zero incidents over a quarter of observation.
Separate graphs of retransmissions and per-hop loss were brought into the monitoring dashboard: the next such deviation is visible at the growth stage, not at the moment of client complaints.
How to understand that the bottleneck is in the application, not the channel?
Separation of responsibility is built on two independent sets of measurements: processing time inside the application and travel time across data transmission segments.
If the first indicator grows while the second remains steady, the bottleneck is in the code or in the queue to the database, and optimizing the channel is pointless. The reverse picture points to the network, and any application change there will change nothing.
Measurements on the application side are taken with its own tools: request processing duration by stage, waiting time for external calls, share of time in blocking I/O operations, size of returned responses.
These data do not depend on what happens between the server and the client — that is why they work as a reference during comparison.
Data transmission segments are checked separately.
Route tracing shows at which node loss and latency growth appear; a traffic analyzer (Wireshark or tcpdump) gives the number of retransmissions, window overflow, drops, and re-established connections; throughput measurement between two points answers the question of how much the channel actually delivers.
It is important to capture this at both ends: a one-sided picture lies because of asymmetric routing.
The verification methodology is simple in idea and requires discipline: measurements are taken simultaneously, tied to a common time, and compared across the same requests.
Growth in retransmissions or loss on a specific segment with unchanged processing time — the channel is the culprit. Growth in processing time with a clean channel and no loss — the application or an adjacent service it calls is the culprit.
The result is not an opinion, but a written analysis with a point of responsibility and evidence for each segment.
A dispute with the provider or the owner of an adjacent service ceases to be a matter of taste, and the work plan gets priorities: what is confirmed by measurements is fixed first.
Frequent technical questions about incident analysis
Questions at the start are almost always not about tools, but about risk and reproducibility. An intermittent failure, capture on a live service, third-party network segments, and a closed loop come up every time — below are short answers for each.
The failure is not reproduced — how will you catch it?
An intermittent failure is caught not by a one-off launch, but by long-term capture with a ring buffer: the collector keeps a fresh window and overwrites the old one, so after an alert fires, the picture before the event remains.
Alongside, we compare error counters and latency by node — the failure is localized through observations, not by trying to guess the moment.
Will services have to be stopped during capture?
No. Capture is done on port mirroring or at the ingress point, without interfering with traffic; if necessary, depth is limited to headers. We perform bandwidth and loss measurements between nodes in low-load windows so as not to distort the picture.
Part of the network is not under our control — what can actually be measured?
The area of responsibility is determined by measurements from both ends: comparing the trace, per-hop loss, and latency spread shows where degradation begins.
Where there is no access to equipment, the conclusion is built from traffic behavior and points on your side — this is enough for a substantive conversation with the provider.
How often should measurements be repeated?
A one-off snapshot answers “what was,” a series — “how it is changing.” We keep permanent metrics on critical routes, while detailed data is captured for an incident and repeated after changes — this shows whether the fix helped.
Can work be carried out in a closed loop?
Yes: capture and analysis are deployed on your side, collectors write locally, and only aggregated metrics or a report go outside. Identifiers and packet contents are masked if required by policy.
Let's discuss your task and come back with a check plan
In the first conversation, it is more important for us to understand the failure picture than to receive a ready-made list of commands.
What exactly breaks: communication between services drops, latency grows for some users, a tunnel periodically goes down, or throughput does not match expectations. The answers determine whether we take traffic dumps, compare loss along the route, or measure speed between nodes.
So that the work estimate is based on facts, not guesses, we need initial data to start:
- network diagram and list of segments between which the problem traffic flows;
- read access to configurations, connection points to nodes, and windows when checks can be launched;
- description of symptoms: when it started, how often it recurs, what monitoring records;
- which hypotheses have already been tested and what was changed in configurations before the first failure;
- site constraints: production environment, work approval procedure, prohibition on restarting services during load hours;
- who on the customer side accepts the result and answers clarifying questions.
After that, we come back with a work estimate and a check plan: what we look at and in what order, which hypotheses we close at each step, where short windows are needed, and what will go into the report.
We agree on the start order in advance — data collection proceeds without stopping services, and everything that requires intervention is scheduled in a separate window.
Ready to discuss the task — describe the symptoms and constraints, and we will come back with a work estimate and a check plan.