Error tracking in DevOps: where errors get lost in prod
When a payment processor goes down in prod, the first to learn about it is the customer, not the on-call engineer: they see the error in the UI and write to support.
Error collection from production is not connected to the release pipeline — the application writes the stack to a local file, restarts, and the exception context is lost along with the process.
Scattered service logs make the picture harder. Each component writes in its own format, some to container output, some to the node disk, which is replaced on the next deployment.
To reconstruct the path from request through the queue to the failed task, an engineer manually stitches together identifiers across several stores — that is hours of work.
A separate difficulty is environment drift. On the staging environment, an exception in the application reproduces with different data, without real volumes or the state of external integrations.
A defect that reproduces consistently in prod is “not caught” in the test environment, and time goes into comparing environments rather than fixing it.
Centralized error tracking solves this differently: exceptions go to a single store directly from production, are grouped by type and location in code, and are tagged with release, environment, and user session.
A tool on the level of Sentry adds a context snapshot to the stack — request parameters, release, transaction ID — and links the failure to a specific deployment.
Then incident triage happens from a single issue card: which release it appeared in, how many users it affected, whether the frequency is growing after the latest rollout.
Based on these signals, a rollback or emergency fix is decided before the count reaches hundreds of support requests. Below is how to build such a pipeline.
What centralized error collection gives you: measurable effects
Centralized error collection closes the main blind spot: a production failure lives until the first complaint, not until the first event in the observability system.
Instead of log streams and chat threads, there is a single error collection point where every crash has a service, environment, release, and stack trace.
Speed of incident detection and triage
Crash notifications arrive in the on-call channel at the moment of the first error, so failure detection time drops from hours to minutes, and for recurring scenarios to seconds.
Grouping of recurring errors merges thousands of similar events into one issue card: the engineer investigates the cause instead of scrolling through a stream.
Classification of errors by exception type and service owner automatically routes most events — the on-call engineer receives them with ready context instead of manually searching logs.
A single picture across all services and environments
When all services and environments send data to one registry, comparing failures across releases and environments becomes a couple-minute operation.
It becomes visible that the test environment fails more often than production — regressions are caught before deployment, not through a customer report.
Event stitching rules are configured for the project (see event grouping rules), so similar failures from different versions do not get mixed together, and the share of recurring crashes drops severalfold. Sentry and its analogues solve this at the platform level, not with custom handlers on the server.
Deployment options: cloud, self-hosted, hybrid
The choice of scheme is determined not by interface convenience, but by where error data is allowed to be stored and how strict the environment isolation needs to be. Below are four alternatives that teams realistically choose between at the start of implementation.
| Option | When it fits | Limitations |
|---|---|---|
| Cloud error collection | No local storage requirements, need a quick start and ready-made alerts | Events leave the perimeter; sensitive fields must be masked on the application side |
| Self-hosted deployment | Regulatory requirements, closed perimeter, personal data in traces | Operations are on us: event database, retention, updates, backups, on-call |
| Hybrid scheme | Aggregates and metrics can be exported, while raw events must stay inside | Two synchronization environments, more complex event routing and access separation |
| Built-in observability platform module | You already have a log and metric stack, need minimal new components | Weaker grouping of recurring errors and release-based triage |
Selection criteria
We look at four things: event volume and required retention period for error data, depth of environment isolation, need to link an error to a release and commit, and who is on call for collection and triage.
We build routing so that switching between schemes is reversible: the application sends events to an internal receiver, and it decides where to send them — to the cloud or to a self-hosted store. Then changing the option does not require rebuilding services.
Implementation stages: from audit to handover to support
The order of steps matters more than speed: if you enable collection before the audit, some failures will remain invisible — current logs do not show them either. So first we describe what we already observe, and only then expand coverage.
Preparation and error flow diagram
- Audit of current observability. We analyze which failures are caught now — server exceptions, client crashes, background job failures — and where data is truncated or not collected at all.
- Error flow diagram. We trace the path of an event from service to storage and mark the areas where it is lost during a network failure or process restart.
- Collection setup. We connect all environments, define grouping rules so that one failure does not split into hundreds of records, and rate-limit sending — the stream must not overload the store.
- Enrichment with build and environment data. We link the event to a specific build and environment: the error is immediately associated with changes instead of being searched manually in logs.
Acceptance: how we verify data is not lost
- Loss verification. We run test failures in each environment and compare how many were sent and how many arrived; we separately examine behavior when the store is unavailable — the queue must survive the failure, not lose events.
- Handover to support. We give the on-call team alerting rules, access, and triage procedure, and assign an owner for each service — after that the system lives without us.
Output artifacts: configurations, diagrams, policies
A report on the work done is useless if, after the team leaves, the configuration cannot be reproduced. So we hand over not a presentation, but a working set — files, diagrams, and policies that allow the system to be developed by your own team or another contractor.
- Collection configurations: client library and server-side event ingestion settings — logging levels, grouping rules for similar failures, noise cutoff thresholds.
- Error flow diagram: the path of an event from a service failure to an issue card in the dashboard — where it is enriched with context, where it is stored, how long it lives.
- Environment separation rules: separate projects and keys for development, testing, and production — test noise does not enter the production feed.
- Data access policies: team roles, who can see reports with personal data, how masking of sensitive fields is configured.
- Incident triage policy: who receives the notification and within what time, how the cause is recorded, and by what criterion an error is considered closed.
- Instructions for connecting new services: step-by-step procedure with ready configuration templates and a checklist to verify events arrive.
How the documentation being handed over is structured
Each document is tied to the live system: diagrams are assembled from actual settings, not drawn separately, so when the configuration changes, the discrepancy is visible immediately.
We establish versioning — changes to a procedure or access policy go through approval, and the change history stays with you.
Separately, we hand over a decision log: why this grouping level was chosen, how thresholds are configured, and what trade-offs on event volume were accepted. This is what is usually lost during handover and forces the next team to ask again or break a working scheme.
Case study: how failure detection time was reduced
The client is a product made up of several services and a mobile app. Production failures were found by users: a signal came as a support email or an app store review, and from there reached the on-call engineer.
We implemented centralized error collection with Rollbar, linking events to release, environment, and component.
What it was like before implementation
Failure detection time was measured in hours and depended on when a user decided to write: some crashes were never found at all. Recurring crashes lived for weeks — the same failure arrived with different wording and was filed by different people.
The share of investigated errors stayed around one third, the rest were closed as “cannot reproduce,” and support load grew with the number of requests.
What changed after
Events are grouped into traces by stack trace and tagged with environment, release, and service — every error has an owner. The on-call engineer receives a signal in the channel within the first minutes, before the first support requests.
Separate grouping by release shows which rollout introduced the failure and allows rolling it back before the flow of complaints grows.
Metrics before and after:
- time from failure to signal — from hours to 2-3 minutes;
- share of errors with a clear owner — from one third to 9 out of 10;
- repeat requests for already known failures decreased threefold;
- recurring crashes stopped coming back: each has a closure rule.
How centralized error tracking differs from logging
Logging answers the question “what happened”: it is a continuous stream of events from all services, which is later used to reconstruct the picture of a failure.
Error collection answers a different question — “what is broken and for whom right now.” The first tool writes everything indiscriminately and grows linearly with load; the second records only exceptions and immediately brings them to a form that supports decision-making.
The key difference is exception grouping. An event is not stored as a separate record: the system calculates a signature by type and location in the stack trace and folds similar occurrences into one issue.
Ten thousand failures in one part of the code become one card with frequency and number of affected users, not ten thousand lines among which you need to find a pattern.
Next comes event correlation. A cascading failure — a failed external call, followed by timeouts in three adjacent services — looks in logs like independent records in different streams.
A single request ID links them into one chain, and the primary cause becomes visible, rather than its downstream effects.
A separate value is linking an error to a release. The event is tagged with the build version and a point in the change history, so a new problem is separated from a known one, and a regression is visible immediately after rollout. This gives a quick answer: roll back or fix forward, and which exact part of the code is affected.
The tools do not replace each other. Logs are needed for context around a failure; error collection is needed for prioritization: what is on fire, whom it affects, and when it appeared.
Technical concerns before starting: frequently asked questions
Before starting, the technical team checks not “does the system work,” but how it will behave under failures, spikes, and regressions. Below are answers to the questions most often raised during approval: from receiver unavailability to access separation.
Almost all of them are solved by configuration and deployment architecture, not by modifying application code — that is why they are discussed before the start, not after the first failures.
What happens if the external receiver is unavailable?
Event sending is asynchronous and does not block request processing. Events are buffered locally and are sent after connectivity is restored; when the buffer overflows, the oldest events are dropped, not the newest. Users notice nothing at the time.
What volume of data goes outside?
Only the stack trace, request context, and build version — without database contents or personal data. The stream is limited by sampling rate and filters, so its volume is predictable and does not grow with traffic.
How do you cut off noise and duplicates?
Events are grouped by signature: exception type and top stack frames. A thousand identical failures become one record with a counter, while predictable extension errors and connection drops are discarded by filters.
Can it work in a closed environment?
Yes: the receiver and storage are deployed inside the perimeter, and not a single event goes outside. Updates are delivered through an internal registry, and the same deployment serves isolated environments.
How do you separate team access to error data?
Permissions are granted by projects and roles: a team sees only its own services, while the platform team sees a summary across all. Changing rules and deleting events are separated from reading, and personal data is absent from events by default.
Let’s discuss the task and come back with an implementation plan
We build the first conversation not as a presentation, but as gathering inputs: without them, estimating the scope of work turns into guesswork, and the implementation plan into generalities.
Below is what we clarify in order to come back with a concrete proposal, including deploying Sentry or collecting events in a self-hosted environment.
- Composition of services and environments: which applications, background jobs, and mobile clients should go into the system, how many environments there are, and which of them send events.
- Data placement requirements: whether reports can be sent to an external service or an environment inside the perimeter is needed, and how we mask sensitive fields in stack traces.
- Team roles: who owns the project, who triages alerts, how we divide access and configure routing of events by responsible party.
- Alerting procedure: which failures we consider critical, where we send the signal — chat, email, on-call — and what threshold so that alerts do not turn into noise that people stop reading.
- Readiness criteria: what share of paths is covered by tracking, which error types we catch, how we verify that an event arrived and was grouped correctly.
- Handover format: configuration documentation, event flow diagram, on-call instructions — what will allow the team to work with the system without us.
- Access: how we issue keys, where we store secrets, how we revoke access when team composition changes.
Describe the task in free form — service composition, current pain, what you have already tried. We will come back with an estimate of the scope of work and a stage-by-stage implementation plan, with readiness criteria for each stage.