GCP Cloud Load Balancer with Terraform and Cloud Armor: turnkey setup

GCP Cloud Load Balancer: where configuration breaks

The most common cause of downtime is not a node failure, but a mismatch between the described configuration and what actually runs. The classic case: a routing rule sends traffic to a group whose health check looks at a different path or port than the application listens on.

The Google Cloud balancer considers the backends healthy, while the user gets a 502—and this continues exactly until someone fixes it manually right in the console.

Manual edits bypassing configuration are the second most frequent breakage. They live until the next state apply: the tool brings the resource back to the described form and silently rolls back the fix, and the incident returns in the next release.

A separate story is health checks that have drifted apart: one service drops out of the balancer on any brief latency increase, the second never drops out and continues receiving traffic when a node degrades.

The cost of such a scheme is broader than minutes of unavailability. Edits outside configuration mean there is nothing to restore the previous working state with: there is nothing to roll back, because the last working version is not recorded anywhere.

We break down how this is closed: a single balancer module with versioning, health checks from one template for all groups, traffic filtering rules at the perimeter—and not a single setting that cannot be applied again.

What managed load balancing provides

A single entry point into the service replaces a dozen disparate points: one external address, one TLS termination, then routing rules by host and path.

A new service connects as a backend, not as another open port, so clients and internal consumers stay on the familiar address while the infrastructure underneath changes.

Traffic distribution between services is described declaratively: backends, health checks, weights between revisions.

An instance that stops responding to the health check is removed from rotation automatically—without manual intervention and without troubleshooting an incident at night.

Hence the reduction in recovery time. A failed release is rolled back by changing the weight to the previous revision, not by rebuilding the environment: traffic goes to a healthy backend in the time it takes to apply the configuration.

Terraform keeps state in code, so the previous working version is always described and reproducible, not restored from memory.

Some failures disappear as a class. Packets are filtered at the perimeter: rules limit request rates, cut off unwanted addresses and signatures, removing load from applications before they notice it.

Application fault tolerance stops depending on whether the developer guessed the resource headroom—backend health is checked continuously, not after a crash.

Managed load balancing remains observable: backend metrics are visible at the route level, so degradation of an individual service is visible before it becomes a general outage.

Balancing modes: comparing approaches

The balancing mode determines not so much performance as the available feature set and routing scheme. An external balancer accepts traffic from the internet, an internal one only from the project network, and these are different resources with different constraints. Below are the options that are actually chosen between.

The limitations of the routing scheme matter more than it seems: rules by paths, headers, and weights exist only at the application layer.

At the transport layer, balancing goes by address and port, without inspecting content, and smooth switching between service versions cannot be built on it.

Choosing an option for the task comes down to three questions: where traffic comes from, whether HTTP inspection is needed, and whether the contour must survive a regional failure.

The answers unambiguously define the mode—and in Terraform one resource appears with clear fields instead of a set of workarounds.

Implementation order: audit, configuration, validation

Work proceeds in iterations: first we capture the current state and traffic profile, then move the balancer into configuration descriptions, and only after that validate behavior under load.

The reverse order—edits directly in the balancer before the audit—almost always produces a scheme that cannot be reproduced and safely rolled back.

1. Audit of current infrastructure. We capture the actual scheme: traffic entry points, target services, routing rules, health checks, protection settings, and anything configured manually and not recorded anywhere.
2. Infrastructure as configuration. The balancer, target services, health checks, and Cloud Armor rules are moved into Terraform modules separated by environment. State is stored remotely with locking, and changes go through approval.
3. Configuring routing and protection. Rules by paths and hosts, response timeout, region binding, request rate limits, and filtering by request attributes.
4. Traffic switching. First a small share of requests goes to the new scheme, metrics are compared with the previous contour, then the share increases. Rollback is returning the distribution with one edit.
5. Load testing. The profile repeats peak scenarios: request burst, single-zone failure, long-lived connections. We look at where health checks trigger and how limits behave.
6. Observability. Latency metrics, error rates, and target service health are consolidated into a dashboard; alerts are tied to failure symptoms, not individual nodes.
7. Configuration handover. We hand over the repository, environment scheme, procedure for typical changes, and rollback procedures, and walk through everything with your team on a live edit.

What we hand over: Terraform modules, policies, procedures

We hand over infrastructure as a repository, not as a set of settings in the console: changes go through review, are applied repeatably, and are rolled back to the previous state. After handover, your engineers edit the configuration themselves, without our involvement.

• Terraform modules — network, balancer, and instance groups are described in code with variables for contours: test and production are deployed from one set with different values. State is stored remotely with locking, so parallel applies do not destroy the infrastructure.
• Cloud Armor policies — filtering rules on the external perimeter: trusted networks, request rate limiting, protection against common attacks. New rules first run in monitoring mode and are switched to blocking only based on accumulated statistics.
• Architecture diagrams — traffic flows, entry points, zone distribution, health checks, and where traffic will switch if a node fails.
• Operating procedures — how to connect a new service, how to change protection rules, in which windows to make changes, and in what order to roll back.
• Infrastructure documentation — resource map, variable descriptions, secret storage locations, and rotation procedure.
• Runbooks for common incidents — rising error rate, latency spike, false positive protection trigger: what to check first and which step restores a working state.

Case: how we reduced load balancer failures

The project is a service with uneven traffic: scheduled peaks, several server pools behind the load balancer.

Configuration was edited manually in the Google Cloud web console: health checks, routing rules, and instance group bindings lived in scattered settings, with no change history.

Every rollout threatened failure: requests could arrive at an instance that was not yet ready to accept them, and rollback was performed according to a text instruction at night.

One such incident consumed several hours of on-call time and recurred regularly—let's break down exactly what was changed.

What changed

The balancer was described declaratively: server pools, health checks, routing rules, and limits are stored in code and validated before apply.

Traffic was distributed across several availability zones, and updates go through smooth draining of instances from under load—a new release first confirms readiness and only then receives a share of requests. Console edits were closed off: any change goes through review and remains in history.

Metrics were taken before and after on comparable traffic. Failures stopped being an event: an unhealthy instance is removed from the pool automatically, and a failed release is rolled back before it receives a share of requests.

The on-call engineer no longer keeps the order of manual actions in their head—the configuration is reproducible, and the same environment can be brought up from scratch.

How does a global load balancer differ from a regional one?

In Google Cloud these are two different products, not two modes of one.

A global application load balancer holds a single address announced from all edge points: a request enters the nearest one, the HTTPS session terminates there, and then traffic goes to a healthy region from the backend list. A regional one is tied to a single region—its address and point of failure are the same.

Hence the difference in failure domain. With a global one, the user does not notice a region failure: health checks remove backends from rotation, and the address continues to be announced.

The regional one fails together with the region—the address does not move; only DNS with geographic routing plus a second load balancer and an external switcher can save it.

Certificates and cache behave differently. A global one needs only a managed certificate—it is issued and renewed at the edge. A regional one requires a certificate in its own region, and with a multi-region scheme they will have to be synchronized. Edge cache works together with a global backend; with a regional one it is limited to that region.

Cloud Armor is connected by a security policy to the backend service: with a global one, rules trigger at the edge, before entering your region; with a regional one, they trigger already inside.

The price of such simplicity is configuration complexity: a regional scheme requires DNS routing and synchronization of certificates and backends, while a single address serves all regions.

A working rule: a public HTTP(S) service with users in different countries—global; a requirement to keep traffic in one region or a non-standard protocol—regional.

Common technical concerns before starting work

Before starting, the same questions almost always come up: how to switch traffic without service downtime, whether protection will start blocking real users, and what the team will have left after handover. Below are short answers to the point, without generalities.

Won't the protection start blocking real users?

Cloud Armor rules first run in preview mode: matches are written to logs, but requests are not rejected. From the logs you can see who falls under the rules—office subnets, search engine robots, and requests from the payment provider are added to exceptions. Blocking is enabled when false positives are zero.

What will the team have after launch?

The entire infrastructure is described in code: we hand over the repository with modules and environments to the customer together with the procedures—how to add a backend, release a rule, roll back a release.

Console edits do not break the scheme: a plan check will show the drift and return the described state.

How difficult is it to maintain this on your own?

Typical tasks—a new backend, certificate, filtering rule—are handled by editing a few lines, and a plan check catches errors before apply. We stay in touch for non-standard changes, but the team has no ongoing dependency on the contractor.

Let's discuss your balancing scheme and protection

We provide a load balancing estimate based on inputs, not on a spoken description: this way timelines and scope do not drift already at the turnkey load balancer implementation stage. Send what you have—further questions will be specific.

• traffic scheme: which services receive requests, from which regions, over which protocols, and whether an external or internal load balancer is needed;
• current state: balancing is assembled manually in the console, partially described in Terraform, or being set up from scratch;
• protection requirements: which Cloud Armor rules are needed—request rate limiting, address lists, filtering by request attributes;
• access and work boundaries: cloud projects, service account permissions, separate environments for testing, who approves changes;
• operating mode: maintenance window, requirements for rollback to the previous configuration, and the procedure for validation after changes;
• reliability expectations: target availability, acceptable latency, what is considered a failure, and how you learn about it.

In response, we will come back with an estimate and work plan: a list of resources that Terraform will describe, the Cloud Armor connection scheme, the order of stages with checkpoints and a rollback plan.

Let's discuss the task—describe it in writing or on a call, and we will tell you what should be clarified before starting.