Setting Up Object Storage Cross-Region Replication (CRR)

What breaks without cross-region replication

A failure in the source region rarely looks like "the region disappeared entirely." More often, a specific segment is unavailable: the bucket returns errors on reads, the share of requests that fail to meet the allotted time grows, and the load balancer gradually takes the node out of rotation.

The service is formally alive, but any operation that needs an object by key queues up — degradation spreads to file uploads, report exports, and queue processing.

The second half of the problem is the quality of copies in the backup region.

Cross-region replication is enabled, but the lag between a write and the object's appearance in another region is measured in hours, some buckets were added later and did not make it into the rules, and lifecycle policies moved unreplicated data to cold archive.

At the moment of failover, it turns out that some objects are missing entirely, others are outdated copies of objects, and it is impossible to distinguish them by metadata. A storage region failure exposes that replication was configured for reporting, not for recovery.

The cost of such an incident is not downtime but service recovery time: while object bodies are being restored from archive, the application runs in read-only mode or serves incomplete data.

Data loss during an incident stops being theoretical — write acknowledgements exist, but the objects themselves remained in one region, and after recovery you have to manually sort out discrepancies between copies. We explain how this is addressed.

What fault-tolerant object storage provides

Fault-tolerant object storage is built not on rare "just in case" copies but on continuous synchronization between regions: a write is acknowledged in the primary region, and its copy reaches the backup region.

A channel failure or site incident no longer means data loss — the backup region holds a current snapshot of the storage, ready to serve.

Failover to the backup region with a rehearsed scenario takes tens of minutes instead of hours of recovery from offline copies.

Read traffic is redirected by changing the endpoint address, while signed links and access policies continue to work because rules and keys are replicated along with the content.

The share of requests that the storage serves without manual intervention at that moment remains at the level of normal load — only write operations during the replica lag window are lost.

Object integrity is ensured not by the act of copying itself but by verification: checksums are compared after transfer, versioning preserves previous revisions, and lifecycle rules keep the replica from lagging indefinitely under a high volume of traffic.

This is fundamental because a silently corrupted object is more dangerous than a missing one — the application reads it as valid and serves it to the user.

The main effect is that the class of failures that stops being critical changes. A failure of a region, availability zone, or backbone link between sites no longer stops the service: failover follows a plan rather than an emergency mode.

In such a design, object storage availability is determined not by whether the primary region is alive but by how quickly you are ready to change the endpoint.

Options for cross-region synchronization

Asynchronous synchronization is the baseline mode: an object is committed in the source region, enters the transfer queue, and appears in the target region within seconds or minutes.

Writes do not depend on the availability of the second site, and the channel does not block data delivery, so this is how the main bulk of data is transferred. The price for this is a window in which the copy does not yet exist.

Synchronous writes to two regions return acknowledgement only after the object is committed at both sites. The copy exists at the moment of response, but every request waits for the second region, and its unavailability stops writes.

That is why this mode is used for a narrow set of critical prefixes rather than the entire container.

Separately consider object lifecycle rules: they are not inherited in the target region automatically. If the policy is not created separately, copies either accumulate or are deleted earlier than at the source — and rollback to a previous state becomes impossible.

Implementation order: from audit to handover

We start with an assessment, not with enabling rules. Most buckets already have versioning, lifecycle policies, and their own write flows — replication on top of them works only when this is taken into account in advance; otherwise, we get duplicates and version divergence.

1. Object storage audit. Inventory of buckets and prefixes: volume, access profile, whether versioning is enabled, which lifecycle rules are already in effect, who reads the data and how.

2. Data flow diagram. We define what is replicated and where: prefixes, direction, whether reverse synchronization is needed, what is excluded — logs, temporary exports, cache.

3. Rule design. Target region, replication role and its permissions, behavior on version conflict and object deletion, what to do with already accumulated volume.

4. Configuration of synchronization rules. We deploy rules per bucket, enable versioning at the source and destination, and restrict the replication role to only the required prefixes.

5. Integrity verification. We compare objects by checksums, track replication lag and status for each rule, and separately test overwrites and deletions — discrepancies most often surface there.

6. Monitoring. Metrics for each rule, alerts for replication stoppage and growing lag — otherwise the failure will be discovered when trying to read from the backup region.

7. Handover to operations. We document the design, roles, and rollback procedure, train the on-call shift, and assign owners for rule changes.

Deliverable configurations, policies, and procedures

After implementation, the team is left not with "a setup on our side" but with a complete set of artifacts that allow the storage to be operated and developed without us.

Each item is tested on a staging environment and tied to a specific action: what to edit when adding a bucket, where to look when cross-region replication lags, and whom to escalate to at night.

• Replication rule configurations — manifests in the repository, not manual settings in the console: filters by prefixes and tags, exceptions for temporary objects, change history.
• Bucket access policies — separation of read and write permissions by role, public access disabled, mandatory versioning, access restricted to service endpoints.
• Roles and permissions for replication — a separate service role with a minimal set: read objects and tags at the source, write to the target bucket, procedure for granting and revoking.
• Data flow diagram — source, target region, copying direction, object states in transit, and the point at which the copy becomes the primary source during an incident.
• Storage maintenance procedure — frequency of full-copy verification, procedure for changing the target region, deletion of outdated versions, metric and alert thresholds for the on-call engineer.
• On-call shift instructions — step by step: status check, moving the load to the copy, stopping and resuming replication, escalation; with notes on which commands are safe and which require confirmation.

How we removed dependence on a single region

The customer is a product service with an audience on two continents; its object storage was located in one region and was S3-compatible.

Both hot data and backups were stored there, so a site failure stopped the entire service, and recovery from archive took hours and was bottlenecked by reading backups.

We implemented cross-region replication: a container-level rule copies every new and modified object to the second site, enabled versioning preserves previous revisions, and a lifecycle policy prevents both sites from growing indefinitely.

Replication covered 46 million objects; copy lag behind the source remains within a few minutes, and this is accepted as the operating mode.

What the failover drills showed

The primary region was shut down twice as part of scheduled drills: traffic was moved to the copy, the application was brought up at the second site, and objects were read and written at the new addresses.

Failover was reduced from forty minutes to six, and the bottleneck turned out to be not the data but manual steps in service deployment — they were moved into an automated scenario.

The reverse transition was also tested: after the load returns, the primary region catches up with changes from the copy, and no checksum discrepancies remain. Now a site failure is not data loss but an interruption for the duration of failover.

How cross-region replication differs from backup

Cross-region replication and backup solve different tasks, although both store data outside the primary region.

Replication maintains a continuous copy: a new object appears in the second region almost immediately after the write, it can be read, and if the region fails, traffic can be switched over.

A backup is a point of return: snapshots at specific moments, from which the state for a required date is restored.

Replication follows changes, including deletion: if an object is deleted or overwritten, the copy will repeat it. Data in the second region is always current, but a logical error is replicated just as quickly as a useful write.

Backup works differently: copies are taken on a schedule or on an event, stored separately, with their own access rights and immutable storage.

This is what provides protection against logical errors — mass deletion, data corruption, application failure, or ransomware actions: the state before the incident remains.

Hence different metrics. With replication, data loss is measured in seconds, but it will not help you return to yesterday. With backup, the point of return is chosen deliberately, but recovery takes longer: you need to deploy the data and switch the load.

In practice, these are not alternatives but two layers of protection. Replication covers regional failure and speeds up access; backup covers data errors and requirements for storing historical states.

The choice starts with a question: do you need to bring up the production environment or roll data back to a specific moment.

Technical concerns before launch

The rule captures only new and modified objects — the array that existed before enablement will not move anywhere on its own.

Therefore, we divide launch into two steps: enable the rule and transfer the accumulated data in a separate pass, verifying the result by lists and checksums.

Below are the things the customer's engineers clarify before the first launch: the fate of existing data, integrity verification, and system behavior when the target region is unavailable.

How is integrity verified after transfer?

We compare size and checksums, and selectively read objects from the target region. Discrepancies go into the report; we transfer only them again — a full pass from scratch is not needed.

What if the target region is unavailable?

Events accumulate in a queue, and retries occur with increasing delay. Writes remain at the source and are available; lag is visible by the number of incomplete replicas; after recovery, the queue drains itself.

Can synchronization be selective?

Yes: the rule can be narrowed by prefix, tags, and size. We exclude temporary files, cache, and logs immediately so that the copy does not balloon with service data.

Is reverse synchronization needed?

A two-way rule creates a loop, so we enable the reverse direction only during traffic failover and protect it with versioning and prefix separation. Otherwise, the object will travel endlessly between regions.

Let's discuss the region synchronization task

To estimate the replication volume substantively, rather than "by feel," we need inputs about the infrastructure — we will clarify some ourselves, but some only you know. Below is a list of what is worth having on hand for the conversation.

• List of buckets and data purpose: what is critical and what is restored from the source.
• Volume and growth pattern: number of objects, average size, share of small files.
• Target regions and latency requirements for accessing the replica from each.
• Access scheme: roles and policies in both regions, whether there is a separate environment for replication.
• Lifecycle rules: what to store, what to delete, whether deletion protection is needed at the destination.
• Failover requirements: manual or automatic switching, acceptable downtime.
• Constraints: encryption with your own key, regulatory requirements for storage location.

We go through the list, mark gaps and the items that influence the architecture most.

In response, we come back not with general words but with a work plan: stages, volume estimate, implementation timeline, and rollback procedure if synchronization does not go according to scenario.

Describe the task in two paragraphs — based on these inputs, we will understand the scale and come back with an estimate and plan.