Azure setup: why infrastructure breaks under peak loads
A failure under peak load rarely looks like a crashed server. More often it is a chain: traffic grew — and the bottleneck of the environment turned out to be not in the application but in the cloud infrastructure around it.
Autoscaling does not bring up new nodes because the subscription quota was exhausted during manual experiments, and no one has revisited the load balancer's connection limits.
The root cause is usually one — an environment assembled by hand through the portal. Settings live in the interface, not in the infrastructure description; quick "for a minute" edits before a release end up neither in the configuration store nor in a backup.
A month later, no one remembers why a firewall rule is open and why the test environment differs from production.
Lack of redundancy adds a second point of failure: a single availability zone, a database without a replica, secrets duplicated in build variables. If a site fails, recovery starts from scratch and stretches for hours.
Blurred permissions complete the picture: contractors and interns have subscription administrator roles, access keys are not rotated, and no one reads activity logs until an incident.
An environment audit almost always uncovers dozens of excess permissions and accounts left over from closed projects.
Azure setup therefore begins not with enabling services but with an environment audit: what actually sustains the load, where the configuration diverges from the description, and what will happen if a site fails. Below we look at how this is addressed.
What order in networking, access, and redundancy provides
We divide the network into isolated segments: applications, data, and the management plane. Default rules deny traffic, and we open permissions selectively — for a specific service and port.
Compromise of one node no longer opens a path to the others, and the failure class "the database was reached through the perimeter" leaves the threat model.
We grant access to groups, not people, with the minimum necessary set of permissions, and services authenticate with managed identities instead of keys in configuration.
There is no key to steal and no one to forget to rotate — a separate class of failures due to divergent settings between environments disappears.
We build redundancy through availability zones and a storage replica in a second region.
Failure of a zone or region no longer means service downtime: traffic is switched to a live site, the application continues to operate in degraded functionality mode rather than returning errors to everyone.
This is predictable degradation instead of collapse: queues smooth the peak, cache sustains reads, and load limiters prevent one consumer from bringing down the rest. The target recovery time is measured in minutes, and we record it in drills, not in a presentation.
On Microsoft Azure, all configuration is described declaratively and stored in a version control system, so the environment is reproduced exactly on any testbed.
Once a quarter we shut down a zone entirely and look at the actual recovery time: after each such run, the figure is either confirmed or becomes a task for the next sprint.
Load placement options: machines, containers, services
The choice of placement mode determines not only the cost of operation but also which tasks are solved without rewriting the application.
Moving a load between options is almost never a simple migration: the failure model, scaling method, and who is responsible for updating the environment change. Therefore, we compare modes by suitability and limitations — for a specific load profile, not out of habit.
| Option | When it fits | Limitations |
|---|---|---|
| Virtual machines | Need full control over the environment, custom system libraries, or licensed software on a specific core | Manual or semi-automatic scaling, OS updates on our side, downtime when recreating |
| Containers in a managed cluster | A service of several components, frequent releases, predictable horizontal scaling | Requires mature image building and observability; stateful workloads are more expensive to maintain |
| Managed services | Queues, cache, databases, load balancing — where the platform itself provides fault tolerance and backups | Less control over settings, platform lock-in, reverse migration is more expensive |
| Hybrid environment | Part of the data must remain on-premises, part is moved to the cloud | The channel between environments becomes a critical path; routing and unified authentication become more complex |
Limitations matter more than advantages. Virtual machines give control, but along with it — routine updates and slow reaction to peaks. Managed services remove operational burden but narrow the configuration space.
A hybrid environment meets data residency requirements, but every service will feel latency in the channel.
We start with measurements: request profile, daily and seasonal peaks, storage and recovery requirements, sensitivity to latency.
Then we distribute components across options — some go to managed services, some remain on machines or in the cluster — and record exactly what will change as load grows.
Work order: from audit to handover to operations
The work order is dictated not by the contractor's convenience but by the sequence in which risks appear: until the inventory is collected, any scheme is a guess, and Azure setup without checks is not considered complete.
Therefore, environments are assembled after the scheme is approved, and handover to operations happens only after recovery testing.
- Environment audit. Inventory of subscriptions, network segments, and services: what is running, who owns it, where manual changes are, and where there are no backups. The output is a dependency map and a prioritized risk list.
- Infrastructure scheme. Target state: boundaries of subscriptions and resource groups, network segmentation, egress points, access roles, and environment separation. We approve it before setup — editing the scheme is cheaper than editing a running environment.
- Environment setup. Development, test, and production are created from declarative descriptions using a single template. Parameters and secrets are stored separately; manual edits in the console are excluded.
- Monitoring and alerts. We collect metrics and logs, set thresholds and notification routes. Operations starts with ready reaction rules, not blindly.
- Recovery testing. We run failure scenarios: node loss, segment unavailability, recovery from a copy into a clean environment. We record the actual recovery time.
- Handover to operations. We provide access, documentation, and procedures, and train the team: typical incidents, escalation order, control points.
What you keep: schemes, policies, procedures
An environment that rests on verbal agreements lasts until its author leaves.
Therefore, by the close of work we assemble a set of artifacts with which your team or a third-party contractor can bring up the environment, troubleshoot an incident, and change configuration without us. Everything is stored in your repository and under your accounts in Microsoft Azure.
- Infrastructure documentation — a diagram of subscriptions, resource groups, and the network with subnet addressing and connection points to the on-premises environment.
- Configuration as code — modules describing compute resources, storage, and load balancing with a list of input values and application order.
- Redundancy schemes — availability zones, recovery targets, backup schedules, and a tested rollback scenario.
- Access policies — roles and their scopes, least privilege, the procedure for granting and reviewing permissions, separate service accounts instead of personal ones.
- Role description — environment owner, on-call engineer, the one who approves changes in production, and the boundaries of support responsibility.
- Support instructions — diagnostics of typical failures, where to look at logs and metrics, actions during degradation, and escalation.
- Observability and decision log — availability checks with alert thresholds, decisions made, and consciously deferred tasks.
Documentation is not written "for a folder": we conduct handover as acceptance — the environment is brought up from it without our hints.
Case study: migrating a service to the cloud without downtime
A service with daily traffic lived on manual releases: builds were launched from an engineer's workstation, access to the environment was granted through a shared account, and rollback was done manually according to an internal instruction.
Recovery after a failure took hours: the procedure was kept in one person's head, and the configuration copy was not updated.
What it was like before migration
A release took 40–60 minutes and required a night window — the service was stopped during that time. Shared access allowed the production environment to be changed unnoticed: after an incident, no one could say who changed what last. A database failure was recovered from in 4–6 hours and lost the last day's data.
How it was migrated and what changed
The environment was deployed in Microsoft Azure using the infrastructure as code approach: environments are brought up from a description, and manual edits in the control panel are prohibited.
Access was distributed by roles — builder, release owner, on-call engineer; every change passes review and leaves a trace in the log.
The migration proceeded in parallel with operations: traffic was switched gradually, shares grew based on metrics, and the old environment was kept ready until stability was confirmed. Backups and the replica were moved to a separate zone, and recovery was practiced in drills.
Results: release — 6 minutes without service downtime, rollback — one command, recovery after failure — 20–25 minutes, data loss — no more than 5 minutes, and the share of changes under automated checks grew from 20% to 95%.
Azure setup from scratch or migration of an existing system
The approach is chosen based on the state of the system, not preference: a clean environment is built when there is no load yet; migration of an existing system — when the product is running and downtime is unacceptable; hybrid mode — when part of the data must remain on-premises. Below — how the scenarios differ at the work level and where each runs into limitations.
In a clean environment, the network and permissions are designed first, not the machines: isolated segments, single sign-on, separate subscriptions for environments so that the development testbed cannot reach production data.
If this layer is defined immediately, resources are added without rebuilding, and access separation remains manageable.
Migration of an existing system begins with inventory and a trial replication in isolation: we compare behavior, then switch traffic.
Migration limitations — binding to specific hardware, site-specific licenses, outdated operating systems, and services that hold state in memory and do not survive a clean shutdown. Such nodes are migrated with rework, not by copying.
Hybrid mode means that part of the load remains on-premises; then the connecting channel is critical — a unified user directory, name resolution, predictable routing, and identical access policies on both sides. Latency between environments determines which services make sense to separate.
Setup in Microsoft Azure in any of the three scenarios follows one principle: boundaries first, then load. The choice of approach determines timelines and which class of failure remains — full downtime or failure of a single segment.
Common technical concerns: access, network, redundancy
An engineer's concerns before starting are usually the same: who gets access and how, how to restrict the network, how environments are separated, and what happens on failure.
If there is no current scheme, we start with inventory — but it is better to establish the basic answers right away.
How is access management organized with dozens of people?
We grant permissions to groups, not people: an employee joins a group — gains a role, leaves — loses it. For services we use managed identities, so keys are not stored in configuration. Revoking access takes minutes, and all actions are logged.
How strictly can the network be restricted?
We place databases and internal services in private subnets — they do not face outward at all. We leave public endpoints only for external traffic and secure them with rules and address lists.
We connect the hybrid environment with a secure tunnel to the office network, without exposure to the open internet.
How are environments separated so that test does not affect production?
The test environment is separated from production: its own resource groups, policies, and limits. Production has no direct manual access — changes go through automated releases with checks. This removes the failure class where an experiment brings down a production service.
How are backup and data recovery tested?
A policy sets the backup frequency and retention period for databases, disks, and configurations.
The main thing is not the presence of backups but confirmed recovery on a separate testbed: we record the target time and recovery point objective and verify them with a training failover.
Let's discuss the task and come back with an estimate and plan
A productive conversation starts with source data, not a forty-page brief. To set up an environment in the cloud, what you already have at hand is enough — our engineers will collect the rest themselves and clarify during discussion of the task.
- Diagram of the current environment: which services are running, where they are hosted, what has already been moved to the cloud, and what remains on own infrastructure.
- Requirements for environments: whether separate environments are needed for development, testing, and production, and whether they must be identical in configuration.
- Load profile: peak values, seasonal spikes, response time requirements, and target availability mode.
- Access rules and role separation: who gets access to resources and under which model, whether there are requirements for action logging and log retention.
- Constraints and boundaries: hard dates, maintenance windows, regulatory requirements, and existing obligations to vendors.
- A responsible engineer on your side who confirms decisions and accepts the results of stages.
In response, you receive an estimate of the scope of work broken down by stages, an implementation plan with sequence and dependencies, the scope of work — what we cover and what remains on your side — and a list of checks confirming the readiness of each stage.
If there is not enough data yet, we will start with inventory: describe the task, and we will come back with an estimate and plan.