Integrating VK Cloud Object Storage with CI/CD and Terraform for backups

Configuring VK Cloud Object Storage: where backups break

Backup to object storage breaks not when the copy is created, but during upload. While the archive sits on disk, it is intact; as soon as sending to the bucket begins, connection drops, timeouts, and partially uploaded objects appear.

At the same time, the pipeline often considers the run successful — it sees the return code, not the fact that the written object does not match the local one.

The second failure point is access rights.

A service key with excessive permissions for the entire bucket, an expired signature, or an account granted read-only access all produce the same picture: the job fails on write, and you have to investigate using the timestamp of the last successful object.

Separation by individual service accounts and least-privilege permissions also removes some leak risks.

The third is that the copy registry grows faster than the infrastructure itself changes. Without a lifecycle rule in the bucket, hundreds of objects accumulate, among which it is no longer possible to find the copy of the required release.

A lifecycle rule keeps the last ten versions per tag and deletes the rest: the registry stops growing, while rollback to the previous image remains possible.

The fourth is that no one checks that the copy restores. A checksum, the actual object size, and a trial deployment from the bucket once a quarter catch corrupted archives before they are needed in an incident.

It is precisely configuring VK Cloud Object Storage with these checks that turns upload into a manageable process: integration of storage with the build pipeline becomes part of the overall scheme, and backup upload errors stop recurring from release to release.

Measurable effect: build, recovery, failure class

While copies are taken by a scheduled script to a separate disk, the recovery window is measured in hours, and the recovery point is a day behind the incident.

Incident investigation takes longer than the failure itself: first you need to find an intact archive, then bring up the environment, and only then return traffic. With three or four services, this is a noticeable share of downtime per quarter.

Moving copies to VK Cloud Object Storage changes not the medium, but the operating model: buckets and rules are described in Terraform, upload is launched by the same pipeline as the build, and versioning stores previous object states.

Separately, we enable checksum verification and recovery to a clean environment — otherwise the copy remains unverified until the moment it is needed.

The recovery window is reduced from hours to tens of minutes: the required object is taken directly, without deploying intermediate storage. The recovery point shifts from a day to half an hour to an hour — depth is set by the upload schedule, not by engineer availability.

Verification restores run on a schedule and cover all critical services, not just one selected service.

The failure class changes more than the timings themselves. The scenario “the copy exists, but there is nothing to open it with” goes away: the lifecycle rule keeps the required number of versions, and regular test recovery confirms that the archive is readable.

The risk of losing records between two uploads remains — it is closed by the operations log, which is replayed on top of the latest copy. Downtime after a node failure fits within the window agreed with the business and does not depend on which engineer is on call.

Options for integrating storage with the build pipeline

The integration method determines how predictably the pipeline uploads artifacts and backups and what to do when state diverges.

The difference between approaches shows up not during initial setup, but during growth: separate environments, service accounts, and the requirement to roll state back several steps appear.

What to consider when choosing

First, we look at who owns access to the buckets: if keys are issued by the pipeline, it also owns the rotation policy. The second criterion is reversibility: object versioning must allow returning to the previous state without rebuilding storage from scratch.

The hybrid covers both points with the least rework — the description of buckets and rights remains in code, while upload, rotation, and cleanup are performed by native cloud lifecycle rules.

Implementation stages: from audit to handover of runbooks

We structure the stages so that each step has a verifiable output, and the next one starts only after the previous is accepted.

Below is the sequence followed when configuring VK Cloud Object Storage: from inventorying uploads to the runbooks by which the customer’s team continues to work without us.

1. Inventory of uploads. We analyze which services and jobs write data, at what frequency, in what volume, and where local copies remain. The output is a “source → destination → retention period” table: it immediately shows what is duplicated and what is not copied anywhere.
2. Scheme design. We arrange storages by environments and data types, set versioning, lifecycle rules, and deletion protection. The scheme is agreed with you before work begins, so that environments do not have to be renamed later.
3. Access rights audit. We check who can read, overwrite, and delete what: we revoke excess rights and introduce separate service credentials for each task instead of one universal key. Here we also define what happens to access when an employee leaves.
4. Write configuration. Storages, policies, and access are described in code and versioned — the environment is reproduced from scratch. Build jobs publish artifacts and uploads automatically, with no human involved in the routine.
5. Recovery verification. We deploy copies to a separate environment, measure the time to a working state, and verify data integrity. We repeat the test on a schedule — otherwise recovery remains theoretical.
6. Handover of runbooks. We provide the scheme, policy descriptions, instructions for standard operations, and the procedure for failures, and conduct a walkthrough with your team.

The key step is the fifth: without verified recovery, upload gives a feeling of protection, but not protection itself.

Configurations, schemes, and backup runbooks

Delivery of work is not an oral report, but a set of artifacts by which the backup infrastructure can be redeployed, verified, and handed over to another engineer without correspondence with the author. Below is what remains with the customer in the repository and internal documentation.

• Terraform module for the bucket: storage parameters, object versioning, lifecycle rules for retention depth, access policies, and tags — deployed with a single command in a clean environment.
• Delivered integration configurations for the build pipeline: authorization method, service accounts, steps for publishing artifacts, and upload integrity verification.
• Backup scheme: list of systems, composition of copies (database dumps, file volumes, configurations), frequency, retention depth, ratio of full to incremental sets.
• Recovery runbook: rollback procedure by failure type, recovery point selection, responsible roles, and the criterion by which recovery is considered successful.
• Verification scripts: checksum comparison, test recovery into an isolated bucket, discrepancy report with root-cause analysis.
• Role and access scheme: least-privilege permissions, key rotation, where secrets are stored and who has access to them.
• Backup monitoring: metrics for volume, age of the last upload, and error count, alert thresholds, and notification routing.
• Acceptance checklist: how to deploy from scratch, what to check after handover, and how to make sure the runbook works rather than merely sits in documentation.

Case study: reduced the recovery window and eliminated manual uploads

Before the project, copies were taken by nightly scheduled scripts: upload went to mounted disks, and the completeness of the set was checked manually in the morning.

An incident involving the loss of a volume on one of the environments showed the cost of such a scheme — recovery stretched out, some copies turned out to be incomplete, and data had to be assembled from several sources.

What we changed

We moved copies to VK Cloud Object Storage and described the lifecycle with a policy: fresh versions remain hot, old ones move to an archive class, and interrupted jobs restart on their own.

Integrity verification now runs immediately after the run, not in the morning by the on-call engineer.

The on-call engineer is no longer a point of failure: the recovery runbook is in the documentation and can be reproduced by any engineer. Storage costs do not grow — the lifecycle policy removes outdated versions automatically.

How does configuration via Terraform differ from manual scripts?

The manual scenario looks like this: the bucket, access policy, versioning, lifecycle rule, and cross-region replication are created via the console or a set of CLI scripts.

For one bucket, this is faster than any code — that is exactly why manual storage configuration in VK Cloud Object Storage survives until the first scaling: at the tenth bucket and three environments, policies diverge, and the reason for disabled versioning is sought in the chat history.

Terraform translates the same into a declarative description. You fix the desired state — which buckets are needed, which rules exist, and who has access — while the state file stores the link between the description and the real objects.

Before applying, a plan is visible: what is created, what changes in place, what is deleted. Repeating in another environment gives the same result regardless of who pressed the buttons and in what order.

The difference shows not in the speed of initial setup, but in infrastructure management afterward. Changes go through review in the repository, rollback is a return to the previous version of the description, not restoration from memory.

Drift after a manual edit in the console surfaces in the next plan — a cheap safeguard against environments quietly diverging.

Manual scripts do not disappear entirely: one-time data migration or an emergency operation when the state file is unavailable remain their domain. The working scheme is a declarative base plus targeted scripts, where everything permanent returns to code. Existing buckets are not recreated: they are imported into state and then managed as code.

Technical concerns: access rights, encryption, rollback

Access rights to storage are described by roles, not a shared account key. The build pipeline writes copies using a service account that has rights only to a specific bucket prefix, the application reads its own data branch, and the engineer gets nothing beyond what is needed.

Keys are not stored in the repository: they are injected from a secure secrets store at runtime and rotated on a schedule.

Encryption is covered at two levels: on the VK Cloud Object Storage service side and on the client side, if the dataset requires it.

Versioning is enabled from day one — deleting an object does not erase previous versions, and for critical datasets, overwrite protection is set for the retention period.

Rollback works in two dimensions: data is returned by switching to the previous object version, and settings by returning to the previous infrastructure description.

What if an access key falls into the wrong hands?

The key is limited to one prefix and write permission; it cannot be used to read other data. Scheduled rotation and access logs show who worked with storage and when.

Can the storage settings themselves be rolled back?

Yes. Access policies, lifecycle rules, and encryption are described declaratively, and state is stored remotely. Before applying, we review the change plan; rollback is a return to the previous description.

Let’s discuss the task and come back with a work plan

To come back with a timeline estimate and work plan rather than generalities, we need a minimum of context about your environment.

Below is what is worth preparing for the conversation: with this, we will calculate volumes, break down the stages, and show where the weak points are in the current backup scheme in one pass.

• What we copy: the list of systems, databases, and directories that need backup, and the required backup frequency.
• Volumes and dynamics: how much data there is today and how fast it grows — the storage scheme and object lifecycle policy depend on this.
• Where copies currently reside: disks, tape archive, third-party cloud — to understand what we migrate and what we leave in place.
• How deployment is organized: what roles and access exist in your infrastructure, who is responsible for it on your side, and how changes are coordinated.
• Recovery requirements: how quickly the service must be brought up and how far back in history you expect to roll back.
• Constraints: policies, closed segments, windows when production systems can be touched.

This is enough to discuss the task substantively. We will come back with a timeline estimate, a stage-by-stage work plan, and a division of responsibilities: what we configure, what is required from your team, and how we verify the result. Describe the task — then we will clarify the details and finalize the plan.