Website performance optimization during degradation

Performance degradation: where site speed is lost

Performance degradation rarely looks like a server crash — usually it arrives as a gradual drop in response time.

At first, growth in response time under load is attributed to a one-off wave of visitors, but during peak hours some requests begin to fail due to exceeded wait time, and pages respond several times more slowly.

The first source is the database. The number of calls to it grows with the number of blocks on a page: catalog, recommendations, reviews and stock levels pull data in separate queries.

While tables are small, this is unnoticeable, but a heavy query without a row limit reads them in full — and one bottleneck of the site slows down all sections at once.

The second is bloated pages. Markup, styles and scripts accumulate release after release, old components remain connected, and response weight grows faster than content. Caching does not save you here: keys depend on the session, and every visitor receives the full build again.

The third is the accumulation of technical debt: temporary solutions, synchronous calls to external services inside a request, a cache that is reset with every release.

Individually, each place costs a fraction of a second; together they produce that same performance degradation that is then difficult to tie to a specific change.

It is made worse by the absence of measurements: people optimize what is noticeable to the eye, not what sustains response. Below we break down how this is addressed — which metrics to capture, what to fix first and how not to return to the previous state a few months later.

What eliminating slowdowns gives: measurable effects

The main effect is that response stops "drifting" under load. If before the audit the response time on a typical page grew 4–6 times during peak hours, then after eliminating bottlenecks the spread between quiet and loaded modes shrinks to 10–15%.

The distribution tail also moves closer: it is rare long responses that the user perceives as "the site is slow," even though the average figure looks decent.

Downtime is measured in minutes, not hours.

Scheduled operations — deploying changes, updating the data schema, switching sites — stop requiring a full stop of the service: traffic is moved to a standby environment, the downtime window is compressed from 40–60 minutes to a few, and some operations pass with no visible interruption at all.

Recovery after a node failure also fits into minutes, because state is no longer tied to a single machine.

A separate result is that an entire class of failures disappears. A cascade, when one slow storage query builds a queue, free workers run out and the entire site goes down, is the most common degradation scenario.

After limiting concurrency, setting explicit timeouts and separating critical and non-critical traffic, this class stops recurring: local spikes remain instead of a total outage.

Under load, behavior becomes predictable: response time grows, but the share of errors does not; failure is limited to one function, and other sections continue to work.

This is the main measurable outcome — the system degrades in a controlled way and according to a clear scenario, rather than breaking down entirely.

Engagement formats: one-time audit, ongoing support, reworking

The three modes differ not in the volume of work, but in the horizon of responsibility for the result.

A one-time technical audit is responsible for diagnosis, ongoing support is responsible for making sure degradation does not return, and reworking bottlenecks is responsible for removing a limitation that has already hit its ceiling.

The formats do not compete; they fit together: the audit provides priorities, reworking removes the main bottleneck, and support reinforces the result with measurements on every release.

An external performance team is support with our participation in planning: we maintain thresholds for response time and transferred data volume, signal slowdown before users notice it, and update diagnostic measurements.

The limitation is honest: the mode requires access to the build, metrics and your list of priorities — without them it degenerates into a set of one-time reports. Describe the task — we will come back with an estimate and a plan for which mode to use to close it.

Work sequence: from measurements to handover of procedures

Performance degradation almost never appears immediately — it accumulates: data grows, integrations are added, the traffic profile changes, and the application begins to "drift" at peaks.

Therefore the work sequence is built not from hypotheses, but from measurements: first we record how the system behaves now, and only then do we change code.

1. Establish the baseline. We take measurements in the production environment under real load: response time for key scenarios, request queues, memory consumption, the heaviest database calls. Measurements on an empty test environment show nothing — degradation manifests precisely with volume and concurrency.
2. Find the bottleneck. We correlate metrics with code and query plans and discard false hypotheses. In practice there is almost always one bottleneck, and the other slowdowns are its consequence.
3. Assemble a solution design. We determine what we change: indexes and query structure, caching, moving heavy operations to background jobs, limiting concurrent calls.
4. Implement in stages. Each change as a separate step with a clear effect and rollback capability, so the contribution of each can be measured.
5. Verify under load. We repeat measurements under the same conditions and compare with the baseline: what matters is not the average time, but behavior at peak and recovery after it.
6. Accept the result. We agree on acceptance criteria before work begins, so acceptance is a comparison of numbers, not an argument about impressions.
7. Hand over documentation. Architecture diagram, measurement procedure, description of growth points and rules under which degradation will not return.

This implementation order gives a predictable outcome: the bottleneck is confirmed by measurements, the effect is verified under load, and the customer's team receives not a one-time fix but a working control procedure.

The set of settings and documents that remains with you

When the work ends, the customer's team is left not with a report of what was done, but with a working set of artifacts they can live with further without us.

All of this lies in the repository next to the code and is versioned with it — not in chat and not in someone else's folder on disk. Below is what exactly is handed over.

• Environment configurations — application parameters, memory and concurrent worker limits, caching and timeout settings, moved into files with a description of what each value is responsible for.
• Service interaction diagram — a map of requests between the application, database, cache and queues, marking critical paths and points where degradation of one node affects the others.
• Access policies — roles, read/write permission separation, rules for service accounts and key rotation procedure.
• Team instructions — how to bring up the environment from scratch, how to deploy and roll back a release, what to check before deployment.
• Response procedure — threshold values of indicators that trigger an alert, the on-call procedure and criteria for deciding to scale or roll back.
• Log of known bottlenecks — a list of places that have already caused slowdowns, with the cause and what was done.
• Monitoring plan — which indicators are collected continuously, where they are stored and who reviews them once a week.

The set is updated during the work, so at the time of handover it describes the system as it is.

Case study: how a growing catalog stopped losing speed

An online store's catalog grew from 12,000 to 58,000 items, traffic almost tripled. Performance degradation followed: the listing responded in 4–6 seconds, filters worked with delay, and after releases slowdowns recurred and persisted until manual intervention.

What we found

• Queries across a growing set of filters ran without suitable indexes — more than a hundred database calls per catalog page.
• The cache was completely invalidated on any card update, so its effect hardly lasted.
• The search index was rebuilt on every request instead of incremental update.
• Media was served at original size, although it was displayed many times smaller in the listing.

What we changed

We rewrote the filter queries and added indexes for them, split the cache into layers with targeted invalidation, and moved search index assembly outside the user request.

Media began to be served at the required size, and monitoring dashboards in Grafana were broken down by page templates — it became visible which exact scenario was pulling the metrics down.

We added a run of key scenarios with a response time threshold to the release pipeline: a release does not proceed further if a page has fallen outside the norm.

Listing response time — 5.4 → 0.9 seconds, share of requests slower than a second — from 34% to 4%, database calls per page — from 120 to 14.

Recurring slowdowns after releases disappeared: ten releases in a row passed without noticeable slowdown, because regression is caught before publication, not from complaints.

How performance degradation differs from peak overload

The symptoms of these two states are similar — the page responds more slowly, the share of errors grows — but the nature is different.

Performance degradation is a slow drift of metrics in one direction: response time adds a percent per week, memory consumption after cleanup does not return to baseline, tables and indexes in the database grow.

Peak overload is an event, not a process. Metrics go up sharply and return just as sharply when traffic subsides: a sale, a newsletter, a bot crawl, a spike from search.

The system remains the same, only the load changes, so it is relieved not by rewriting code but by queues, limiting concurrent requests and power headroom.

The difference also dictates the treatment. Degradation cannot be waited out — it will not roll back on its own; its source is in accumulated issues: a leak, an unoptimized query, an overgrown cache, extra code in another release.

There is nothing to treat in the code for overload — it recurs and is relieved at the infrastructure level.

That is why a single measurement solves nothing: a snapshot at peak will show a bad picture on a healthy system, while a snapshot during a quiet hour will hide degradation.

Dynamics are needed — a trend over weeks, behavior after load is removed, the ratio of typical response and slow tail. Continuous graphs in Grafana provide this picture; a one-time report does not.

Diagnosing the cause is built on comparing windows: before and after a release, under load and without it. This shows exactly where time is going — in the database, in external calls or in page delivery — and which of the two phenomena is in front of us.

Common technical doubts about eliminating slowdowns

A one-time spike after a release and a steady degradation trend are different diagnoses, and they are treated differently.

The first is usually associated with cache warm-up, a one-time migration or a traffic spike and goes away on its own; the second is visible over an observation window of several weeks, when the 95th percentile of response time grows with data volume, not with load.

Until this is distinguished, any edits are guesswork, so we first establish a baseline and look at the distribution, not the average.

We separately address the question of responsibility for performance: it is not a one-time action, but a measurable value with an owner and a verification procedure.

After a release, do we need to wait for it to "warm up"?

No need to wait, you need to measure. We take a control measurement for the same scenarios immediately and again after a few days — this separates cache warm-up from the real effect.

The baseline before the edits remains the comparison point: it shows whether things got better or just "seemed" so.

Is a permanent dedicated specialist needed?

Not required. A working option is a procedure: control measurements after each release, a degradation threshold that triggers an audit, and a clear entry point for optimization questions. We bring in an engineer when there is a deviation, rather than keeping one on staff just in case.

Let's discuss your task and come back with an estimate and plan

So that the estimate and work plan are based on measurements, not guesses, we need to discuss the task concretely: what exactly is degrading, in which scenarios this is noticeable and when it started. The more specific the input data for the audit, the less time will be spent on clarifications.

What to send for the first step of cooperation:

• Before/after metrics: response time graphs for key pages, share of slow requests, time to display main content.
• Load profile: daily and weekly rhythm, peak windows, ratio of authorized visitors, search engine bots and external calls to services.
• Accesses and snapshots: a database copy or anonymized export, slow query log, profiler measurements for the problem period.
• Change history: releases and edits to cache, server and network configuration over recent weeks — degradation is almost always tied to a specific change.
• Constraints: windows when releases cannot be made, availability requirements, what is prohibited from changing in the architecture and on the side of related contractors.
• Point of contact: an engineer or administrator who will confirm hypotheses and provide access to the copy and test environment.

The audit begins with reproducing the problem on a copy: we take a profile, localize the bottleneck, test the hypothesis on the test environment and compare the result with production metrics.

After that, we come back with an estimate and work plan — stages, priorities, readiness criteria and rollback procedure if a change does not produce the expected effect.

The first step takes little: send a description of the task and whatever you have from the list. We will close gaps with clarifying questions, not a meeting for the sake of a meeting.