Urgent bug fixing: where production breaks after release
A failure after a release rarely looks like "the whole site went down." More often, production breaks after a release in a targeted way: a site error for customers shows up for some users — those who returned with a saved cart, for a specific role, on a mobile browser, or in a region with a different time zone.
In the test environment, the same scenarios pass green because there is neither real data volume nor live session behavior.
Until the picture is assembled, the team spends hours diagnosing the failure in production instead of fixing it. Logs are read in fragments, the problem is reproduced manually, a symptom is fixed in one scenario — and a day later the same failure appears in a neighboring one.
The toll of such an hour is not only lost orders, but also lost trust: customers have already written to support, and there is no clear answer to "what is broken and when will it be fixed."
Diagnosis is blocked not by code complexity, but by the lack of context. There is no end-to-end request tracing, errors from different services sit in different logs, and the release is rolled out as a whole rather than in parts.
That is why it is impossible to quickly answer the main question: which change and which data state turned a working scenario into a failure.
We break down how urgent bug fixing becomes a predictable procedure.
First we capture the degradation and roll back the dangerous area, then narrow the cause using traces and metrics, close the issue with a targeted patch, and add a check — so the same failure does not return with the next release.
What managed incident analysis gives instead of manual fixes
Incident analysis begins not with a fix, but with measurement: how many users are affected, which scenarios and groups of operations return errors, what changed in recent releases.
This sets priority — first we close what hits payment, login, and checkout, not what sounds loudest in complaints.
Time from signal to working version becomes a manageable value: the typical path is rollback to the previous stable build in minutes, while the fix is prepared without haste or guesswork.
We fix the root cause, not just treat the symptom. A request state snapshot, call stack, and logs from Sentry provide the failure point, and a repeating scenario becomes a check that fails before the fix and passes after.
This also reduces repeat errors: the failure does not return a month later somewhere else, because a test catches it, not a user.
Subsequent releases stop being a lottery. The fix goes through the same pipeline as a planned rollout: a regression suite for critical scenarios, release to a share of traffic with observation, automatic rollback when the error rate grows. This way, release stability does not depend on who is on night duty.
The load on the support team does not decrease by itself — an entire class of failures that were manually triaged every week goes away.
Similar requests collapse into one incident with a known cause and a clear sequence of actions, instead of correspondence and repeated diagnostics from scratch.
Incident response options: fix, rollback, module isolation
Which option to choose is determined by two things: where the defect boundary is and how expensive a pause in service operation is. Release rollback is the fastest move if the failure came with the latest rollout and the previous build is stable.
A targeted fix in production is justified when the cause is clear and localized to one area of code, while a rollback would revert working functionality along with the defect.
| Option | When it fits | Limitations |
|---|---|---|
| Release rollback | The defect appeared with the latest rollout; the previous build is in a working state | Irreversible data schema changes, loss of new improvements, rollback does not work with coupled releases |
| Targeted fix in production | The failure is localized, the cause is confirmed, the scope of changes is minimal | Bypasses the standard verification cycle, requires a quick repeated test pass and subsequent transfer of the fix to the main branch |
| Isolation of the problematic module | The module is not part of the main scenario; the rest of the service works | Part of the functionality is unavailable to users, stubs and dependency control are needed |
| Emergency fix without downtime | The fix is enabled by a configuration toggle | A pre-prepared switch point is needed, otherwise a second execution path appears |
Separately about configuration switching: it delivers a fix in seconds, but only if the switch point was built in before the incident. When it is not there, the team gets two diverging execution paths and root-cause analysis instead of recovery.
We record the choice of method in the policy: who makes the decision, based on what sign, and who confirms service recovery. Without this, the first minutes of an incident go into discussion, not recovery.
Bug workflow: from monitoring signal to production verification
The incident triage policy records not only the steps, but also who makes the decision about an urgent release. Below is the path every bug goes through: from a monitoring signal to confirmation that the fix works in production.
-
Signal intake. The on-call engineer receives a monitoring alert or a support request, checks it against the error log, and assesses the scale: the share of affected sessions and whether the failure blocks a key scenario.
-
Error reproduction. We assemble a minimal scenario — steps, data, environment, release version. We separate out a bug without stable reproduction: a "by eye" fix is not released here.
-
Localization. Using tracing and logs, we narrow the area to a specific request and module, check against recent changes and external dependencies, and rule out false versions.
-
Fix. We prepare a minimal set of changes — only the cause of the failure, without incidental refactoring: this keeps rollback unambiguous and verification fast.
-
Verifying the fix before release. We run the scenario on a production copy with similar data and add a regression run across neighboring paths affected by the change.
-
Release. We assemble a short release with a clear description of changes and the ability to return to the previous build in one step.
-
Post-release observation. We monitor error graphs, response time, and the flow of support requests; if the symptom returns, we roll back and continue analysis.
After the incident is closed, we add an automated test for the reproduced scenario and a knowledge base entry — the same class of failure no longer reaches production.
What remains in the project: configurations, policies, documentation
A one-off emergency fix removes the symptom in minutes, but without artifacts, in a month no one remembers why an exception appeared in the configuration and what scenario it covered.
Therefore, the task is considered closed not when the page responds again, but when the client receives a set of documents according to which the next fix can be made without archaeology.
- Incident documentation: chronology from the first signal to stabilization, symptom, discarded hypotheses, confirmed cause, and resolution method.
- Emergency fix policy: who and under what conditions changes the production environment outside a planned release, how long such a change lives, and when it returns to the normal cycle.
- Access policies for the production environment: roles, minimum required permissions, named accounts instead of shared ones, validity period, and key rotation procedure.
- Change log: a single feed "date — author — what changed — link to the build and to the incident record" that restores the cause of any configuration difference.
- Deployment diagram with failure points: what depends on what, which parameters were changed during the emergency fix, and which values are considered standard.
- Rollback instruction: step-by-step and tested in the test environment, so the decision "we roll back" is made in minutes, not at random.
- Set of checks after the fix: a short list of key scenarios plus added alerts, so a repeated failure is caught by the system, not the customer.
With such a set, a repeated failure is analyzed using documents, not the memory of a departed contractor: the engineer opens the log, finds the cause, and makes the fix in minutes.
At the same time, the project's dependence on a specific person decreases — onboarding a new specialist follows policies, not correspondence.
Case study: how we removed a regression after a major update
After a major update of the catalog and checkout, the share of unsuccessful scenarios grew from 1.8% to 9.4%: users reached the cart and got an error at confirmation.
Support requests came in waves, and the cause was not readable in the logs — the update affected adjacent sections, and the symptom manifested differently in different places.
First we took a snapshot: compared user paths before and after the release, found the step where the chain breaks, and assembled a funnel dashboard in Grafana.
Diagnostics showed a desynchronization in the data access layer — some requests went to an outdated schema and returned an empty result, which the client side interpreted as a failure.
What we did and what we achieved
Rolling back the release entirely was not possible: it contained changes the business needed. Therefore, we released the fix in a targeted way — fixed the access layer, added a schema compatibility check at startup, and retries for unstable requests.
Service recovery time was 47 minutes from cause confirmation to a working release, and the share of unsuccessful scenarios returned to 0.4%, that is, lower than before the update.
To prevent this from happening again, we added a run of key checkout scenarios and an express check of adjacent sections to pre-release checks.
Repeat customer requests for the same problem dropped from 63 to 4 per week — meaning the cause was removed, not the symptom painted over.
What to do if the error is reproducible for customers right now
While the failure is ongoing, the question "why" is subordinate to the question "what to do right now." We set incident priority by three signs: the share of affected customers, failure of a critical scenario (login, payment, checkout) or degradation, and the presence of a workaround. The highest priority is restoring service, not finding the root cause.
We choose a temporary solution in production from three moves, from fast to targeted: rollback of the last release to the previous stable build, disabling the suspicious function with a toggle while preserving the rest of the release, switching traffic to a backup environment, or serving a cached response. We record each step in the log with an exact time — these marks later form the analysis.
What to tell customers about the failure: confirmation of the fact, affected scenarios, known impact, and time of the next update. Without hypotheses about the cause and without promises of an exact deadline — a missed deadline damages trust more than the failure itself. One point of contact and one status publication channel, so support does not contradict itself.
In parallel with repair, diagnostics continue: logs, request tracing, reproduction on a copy with production data, narrowing the range to a specific condition.
After resolution, we add a regression test for the scenario and a monitoring rule so the signal arrives before the customer. This procedure is fixed in the Incident Response Policy, which we hand over together with the escalation procedure to the on-call engineer.
Technical concerns: access to the environment, environments, guarantees
We check access to the repository and servers first — the time to a working release depends on it. We need read access to the code and a separate branch for changes: we branch from the current state of the main line, prepare the fix there, and attach a description of the change. We issue keys personally to an engineer and revoke them on the same day the work is closed.
We raise a copy of the production environment from the same infrastructure description and with the same dependency versions, otherwise verification lies.
We fix responsibility for release and window in writing before the start: who confirms the rollout, what we consider a rollback, and by what sign we stop.
The failure is not reproducible — where will you start?
We collect the trail: time of the event, request identifiers, queue snapshot, and logs at the boundaries. Then we narrow the condition by load, client type, and data set.
If repetition is not achieved, we set up targeted observation in the problematic area and wait for it, rather than fixing at random.
Can we do without a production copy?
For fixes in markup and texts — yes, a test environment is enough. For data, access rights, calculations, and integrations, a copy is mandatory: differences in environment settings produce failures that surface in production at the worst possible moment.
Who is responsible for releasing the fix?
We are — for the quality of the change and its verification on the copy. The rollout is performed by the party with rights in the production environment: usually your side according to our instruction, or we do it under separate written confirmation. We prepare the rollback in advance and describe it in the same message.
Let's discuss the task: what we need from you to get started
To join an urgent fix, a full technical audit is not needed — a minimal set of inputs is enough. The earlier access and a description of the symptom arrive, the shorter the diagnosis: on the first call we are already looking at errors in the logs, rather than collecting context.
What we need from you to discuss the task specifically:
- Symptom: what exactly breaks, on which pages and actions, since when, whether it reproduces consistently, and under what conditions.
- Access: repository, server or hosting panel, build environment, monitoring systems, and logs — with read and deploy permissions.
- Records for the failure window: server errors, browser errors, responses from external services and the payment gateway.
- Release policy: branches, deployment procedure, rollback point, windows when release is allowed.
- Constraints: what must not be touched, what agreements exist with other contractors on incidents and code review.
- On-call contact on your side and the channel where we coordinate actions.
Then we return with an estimate and work plan: presumed cause, scope of fixes, release and rollback procedure, team composition, and timelines for the first result.
Joining the client's project goes through your access — there is no need to transfer the project to us, change infrastructure, or wait for handover. Describe the task in any convenient form: we will clarify some of the inputs ourselves on the call.
