Lifecycle policies: why storage grows without control
The active tier of object storage grows not in bursts, but in a steady stream: logs, dumps, intermediate build artifacts, previews, daily copies. Each source adds a small prefix, and until lifecycle policies and data retention periods are set for it, nothing is deleted.
Within a few months, storage hits not a physical capacity limit but a quota and operations speed limit: prefix listing stops fitting into reasonable time, and background jobs scanning keys compete for request limits.
A separate line of the problem is versioning. Enabled once "just in case," it saves every overwrite as a noncurrent version, and without a rule for deleting it, old copies remain forever.
Incomplete multipart uploads go there too: their parts are not visible in a normal listing but take up space and count toward quota. In S3-compatible storage, this looks like a discrepancy: in the console there is one object, in reality there are ten.
The cost of this state is not only space. Scanning a prefix by mask across millions of keys takes minutes, slowing down exports, analytics, and recovery.
Operation rate limits are shared for the bucket, so one forgotten prefix with logs undermines the predictability of neighboring services.
We break down how this is closed: what rules to set right away — what to move to a cold class, what to delete by age, how to handle noncurrent versions and aborted uploads, so that storage overflow does not return after every manual cleanup.
What changes after retention rules are implemented
Lifecycle rules move storage from manual mode to declarative: the policy describes after how many days an object leaves the active tier, when it moves to a cold class, and when it is permanently deleted.
The volume of the active tier drops by multiples — backups and journals that sat next to working data for years move to the archive on schedule. Iterating over objects in a container speeds up because read operations no longer stumble over millions of irrelevant files.
Object archiving removes the main cause of night shifts — volume overflow, after which writes and file uploads fail.
MinIO and other object storages emit lifecycle events and metrics by storage class, so the share of failures due to disk filling drops to zero: the failure class is removed at the rule level, not by an on-call response.
Incident response time shrinks because an alert comes from a fill forecast, not from a service crash. The engineer sees which prefix or tag violated the policy and either corrects the rule or investigates the specific data source.
Rollback remains possible: versioning together with a rule for the last N versions allows returning an object while it has not crossed the deletion boundary.
Policy options: deletion, archiving, transition
A lifecycle rule is a set of actions tied to a prefix, tag, or object age, not a single checkbox in storage settings. They are applied with a delay, usually once a day, so time passes between an object meeting the condition and actual deletion or transition to a cold tier.
The modes are not mutually exclusive: a working configuration combines two or three, and the choice is dictated by the attribute by which data loses value.
| Option | When it fits | Limitations |
|---|---|---|
| Scheduled deletion | Logs, exports, temporary build artifacts with a predictable lifespan | Age is counted, not access: a demanded file may be deleted before it stops being read |
| Transition to cold tier | Backups, audit exports, data with infrequent access | Minimum retention period in the tier: retrieval before the term devalues the transition, and recovery takes hours |
| Transition by tags | Objects whose value changes by events — project closure, client status change | A rule will not see an object without a tag, and mass tagging is a separate task |
| Working with versioning | Storage with versioning enabled where protection against overwrite matters | Deletion without accounting for versions leaves markers and does not free space; separate rules for noncurrent versions are needed |
| Transition by access | Large datasets with unpredictable demand | Built-in rules work by age and tags, not read frequency — metrics collection and an external scheduler are needed |
We separately check rule overlap: if an object falls under several conditions at once, the most specific one triggers, and the result is rarely readable from the policy description.
That is why we run rule sets on a data copy and compare against the access log — this shows how many objects moved to the cold tier and how many were deleted, and whether active paths were affected.
Procedure for configuring lifecycle policies: stages of work
A lifecycle policy applies to the entire bucket and triggers without confirmation: a rule with the wrong prefix or term deletes objects silently, and rollback is possible only from a backup.
Therefore, we do not enable rules immediately in the production circuit — first we record the actual state of storage, build a scheme, test it on a data copy, and only then move it to prod.
- Storage audit. We collect a list of buckets and prefixes, review volume, object age, the share of cold data, and already active rules — we get a map of where there are no policies at all and where they conflict.
- Object inventory. We classify data by lifespan and value: what to delete, what to move to an archive class, what not to touch. We separately mark versions and objects with a mandatory retention period.
- Rule scheme. We describe the policy in a table: prefix, lifespan, transition between classes, fate of incomplete uploads and old versions. We check that rules do not overlap.
- Policy configuration. We transfer the scheme into bucket rules, recording the configuration as code — it can be reproduced in another circuit and rolled back with one edit.
- Testing on a data copy. We copy a sample of the working bucket, run rules with shortened terms, and see what was deleted, what moved to the archive, and whether the needed objects were affected.
- Handover to the team. We provide the bucket map, rule description, and change log, and show how to check that a policy triggered and how to cancel a rule without data loss.
Set of documentation, schemes, and regulations
The set is not a report on work performed, but files an engineer opens on the first day of operation.
Each artifact is tied to a specific decision: an auto-deletion rule, an archiving policy, or an access role, so when requirements change it is clear what exactly to edit rather than ask again. Everything is stored in your version control system together with the revision history.
- Lifecycle rule configuration files — with a comment on which data class each rule applies to and what term it sets.
- Storage scheme: which objects remain in the hot tier, what moves to the archive, and by what attribute the transition triggers.
- Retention regulation — terms by object type, exceptions for backups and legally significant documents, and the procedure for approving changes.
- Access matrix: which roles read, write, and change rules, and where the boundary lies between the application team and the platform team.
- Rollback procedure: how to return an object from the archive, how long it takes, who initiates it, and what is checked before and after recovery.
- Verification report: how many objects underwent transition and deletion in the test circuit, what discrepancies were found, and how they were closed.
- Expansion instructions — how to add a new bucket or rule without affecting existing ones.
Case: how a file registry stopped growing
Initial state
The registry accumulated everything: each release pulled along all image layers, draft builds lived in the same namespaces as release builds, and interrupted uploads remained hanging as invisible fragments.
Volume increased by about 9–10% per month, and it was cleaned manually once a quarter: an engineer compared the tag list with the latest builds and deleted the rest at their discretion.
An error in this list cost a rollback — once they erased an image from which they had not yet migrated to a new release.
Storage grew faster than the load: inventory took hours, the backup window stretched, and the cost of maintaining the registry ceased to be predictable.
What was configured
Lifecycle rules were moved into a declaration: the last 10 images per release tag remain, draft tags are deleted after 14 days, interrupted uploads after a day, logs transition to the cold class after 30 days and are erased after a year.
The MinIO object storage platform itself evaluates object age upon access, so task scheduling and manual runs are no longer needed, and the right to delete remained only with the service account.
The key point: lifecycle rules do not touch tags marked as release — rollback to the previous build remained available.
Metrics before and after
Volume growth dropped from 9–10% per month to less than 1%, and active space shrank threefold. Inventory went from four hours to a few minutes, the backup window is shorter by about a third because only the needed objects are copied.
Time to roll back to the previous release did not increase, but now it is a guaranteed operation, not luck during manual cleanup.
How does a lifecycle rule differ from versioning?
These are mechanisms at different levels, and confusing them is expensive. Versioning stores object history: each overwrite creates a new version, and deletion leaves a delete marker — previous versions continue to take up space.
A lifecycle rule manages their fate over time: moves them to a cold class, deletes noncurrent versions, cleans up incomplete multipart uploads.
A typical mistake is enabling versioning "for reliability" and not creating a rule. Volume grows linearly with the number of overwrites, and the needed version becomes harder to find.
The opposite mistake is a rule without versioning: space is freed, but overwriting is irreversible and there is nothing to roll a release back with.
The third layer is delete protection: a retention period blocks deletion of a version until a set date, and indefinite lock cannot be removed at all. A lifecycle rule will not touch such a version — it does not become a candidate while the lock is active.
| Mechanism | Task | What it protects against | Impact on volume |
|---|---|---|---|
| Versioning | stores all versions and delete markers | data loss on overwrite | grows with every overwrite |
| Lifecycle rule | moves to cold class, deletes by age | uncontrolled growth | reduces and stabilizes |
| Delete protection | blocks deletion until a term | accidental and malicious deletion | sets a minimum |
In one storage they work together: versioning is enabled, the rule keeps a specified number of recent versions and removes the rest, some prefixes move to the cold class, and normatively significant ones are placed under lock. Rollback to the previous build remains possible, and volume stops growing uncontrollably.
Technical questions about retention policies
A lifecycle rule is applied retroactively: the scheduler traverses the bucket and evaluates by filter not only new uploads but also objects that have been there since the previous pass.
Therefore, a rule with a prefix or tag filter is first run in report mode — it shows how many objects and of what age will fall under archiving or deletion — and only then is transition to the cold class or cleanup enabled.
Reversibility depends on bucket versioning. Without it, deletion is final, and an object can be returned only from an external copy.
With versioning, the object turns into a delete marker: recovery is removing the marker or copying the previous version as current, so a cleanup rule is more logically aimed at old versions rather than current ones.
Will a rule affect objects uploaded before it was enabled?
Yes. Age is counted from the upload date, not from the rule creation date, so the scheduler will also pick up the accumulated array. To avoid deleting too much, set an age threshold well above the retention period of fresh data.
How do I return an accidentally deleted object?
With versioning — remove the delete marker or copy the previous version as current. Without it, recovery from a backup or from a replica in another region remains.
How do I test a rule without losing data?
Run it on a test prefix or bucket copy: the report will show exactly what will fall under deletion or archive. A narrow prefix and logging provide a clear trail — you can see what changed object state and when.
Can I change the retention period after the rule is running?
Yes, the rule is editable, and the new term applies on the next scheduler pass. Objects already moved to the cold class will remain there — the reverse transition to the hot class should be described by a separate rule in advance.
Let's discuss the task: storage audit and implementation plan
To come back with an estimate of the scope of work and a rule configuration plan rather than general considerations, we need a snapshot of the current storage state.
The more accurate the initial data, the fewer assumptions in the estimate and the faster the first version of policies goes to the test circuit on a data copy.
What to prepare for the conversation:
- List of storages and prefixes: the purpose of each, which services and teams write to and read from them.
- Volumes: number of objects and total size, distribution by age — share of data older than a month, quarter, year.
- Access profile: logs show what is actually read and what lies dead weight and only takes up space.
- Retention requirements: where there are regulatory retention periods, where data is needed indefinitely, where deletion without return is acceptable.
- Storage classes: which are already connected and which transitions between them are allowed by your provider.
- Constraints: versioning, object locks, replication to another region, dependencies on caching layers.
- Rollback scheme: how to recover an object if a rule deletes too much, and who approves changes to rules.
- Control points: who on the customer side approves policies and by what metrics we confirm that they work.
Describe the task in free form — what is stored, what is concerning, what deadlines are urgent. We will come back with an estimate of the scope of work, an implementation plan, and a list of risks for each storage.