Feature flag deployment: where rollout breaks at scale
Controlled rollout via feature flags — what industry practice calls feature flag deployment — relies on a simple separation: code reaches production regardless of whether the feature is enabled.
The flag becomes the point where release management is separated from delivery, and rolling back changes in production turns into a switch rather than a new build cycle.
While there are a dozen flags and one release, the scheme holds on its own. At scale, release rollout breaks: configuration turns into a shared set of hundreds of keys, edited directly in the dashboard, without review or history.
The default in code, the default in config, and the value in production diverge; the application caches the config and picks up an edit with a delay — some machines run the old logic branch, and this can only be seen in session logs.
The second gap is parallel releases from multiple teams. Flags begin to depend on each other: enabling one changes the meaning of the second, the order of switches is kept in someone's head rather than in a process.
During an incident, the on-call engineer sees five active flags and does not know which one is responsible for the failing branch.
Then comes operations. A renamed or deleted key crashes the service at startup or silently leads it into a branch that no one has tested.
Expired flags are never removed: branching in the code grows, the build only confirms the “enabled” state, and new logic is built on a foundation of old conditions.
We examine how this is closed: an owner for every flag, configuration versioning, a lifecycle from enablement to removal, and verification of both states already at the build stage.
Controlled flag rollout: effects in measurable numbers
Order in rollout is measured not by the number of flags, but by delivery metrics. We record four values before and after introducing order: release frequency, time from commit to production, rollback share, and service recovery time during a failure. Next — exactly what changes and through which mechanism.
Speed of change delivery
A branch lives for hours, not weeks: code reaches the main branch turned off and is enabled by a separate action. Release frequency grows several times over with the same team size — rollout stops being an event with a window and approvals.
A fifth effect is added — predictability: enablement goes in stages, 5% of traffic, then 25%, then 100%, with a pause for metrics between stages. Returning to the previous state also becomes a single operation.
Reduction of the failure class
Severe failures that required rolling back the entire release collapse: a broken feature is turned off with a flag in seconds, rather than rebuilding the artifact and waiting for rollout.
Service recovery time drops from tens of minutes to minutes — this changes not only the number but also the incident review procedure itself.
An error is localized by scope: a broken path affects a share of users, not everyone. On the Unleash platform, such rules are described by configuration, and we keep it in the repository next to the code — flag state does not diverge between environments.
Flag rollout modes: configuration, service, hybrid
The three modes differ in where flag state is stored and who switches it. This determines application delay, rollout effort, and cost of error — so the mode is chosen not by taste, but by load profile and rollback requirements.
| Mode | Where state lives | When it fits | Limitations |
|---|---|---|---|
| Configuration flags | In a configuration file next to the application | Rare switches, value tied to release | A change requires rebuild and redeployment, no instant rollback |
| Server-side control | In a separate service, application requests state | Frequent switches, targeted rollout to groups | Extra network call on the hot path, cache and default behavior required |
| Hybrid scheme | Local cache plus periodic synchronization with the service | Product under load with response-time requirements | Desynchronization between nodes, state version control required |
Selection criteria under load
We look at three things: how often values change, how many targeting rules are needed at the same time, and how quickly a change must reach users. If switches are tied to a release, configuration flags are cheaper to maintain.
If functionality must be disabled for part of users in seconds, server-side control is indispensable. Under high request flow, we take the hybrid: the application reads the cache, and background synchronization pulls in changes.
Limitations of the options
Configuration flags do not allow rolling back functionality without a rollout and map poorly to several environments at once.
Server-side control adds a dependency: if the service is unavailable, the application must work from a saved default value, otherwise availability suffers. The hybrid requires discipline — versioning of states and a clear rule for who wins when cache and server conflict.
Flag implementation step by step: audit, design, handover
Work proceeds sequentially: first we record what has already accumulated in code and configurations, then we design the rollout scheme, and only after that do we connect production.
Automation on top of unaccounted flags entrenches disorder, so we do not change the order of stages.
- Flag audit. We scan repositories and configurations and build a registry: key, declaration location, who reads it, owner, lifetime. Usually half of the entries are dead or always enabled.
- Classification and cleanup. We divide them into release, operational, experimental, and emergency; each type has its own naming and removal rules. We remove excess before starting work.
- Target storage design. We determine where flag state lives, how it is delivered to the application, and what to return if the storage is unavailable. The default value must be safe.
- Rollout design. We specify the share of traffic to the new branch, rollback order, and switching window. We fix the flag removal date in advance, together with the date.
- Pre-rollout verification. We run both states of each flag in staging, and look at metrics and logs for switches. Flags that fail do not go to production.
- Pilot and handover to operations. We enable the system on one service, measure change delivery time and incident count, then scale it. We give the team the registry, access rights, removal process, and on-call instructions.
Readiness criteria
- Every flag has an owner, type, and removal date.
- Switching takes minutes and does not require a release; rollback is one action.
- The removal process and on-call instructions are accepted by the operations team.
What remains in the repository: schemes, policies, processes
Feature flags behave predictably when their state is described in the repository, not in a dashboard or chat. After implementation, the team retains a set of artifacts that let a new engineer bring up a staging environment and ship a change without verbal hints.
- Flag configurations — declarative descriptions: key, owner, default value for each environment, removal condition; an edit goes through review like code.
- Access policies and roles — who proposes a change, who approves it, who enables the flag in the production environment; rights are granted by role, not personally.
- Rollout process — release order: dark launch, enablement for a share of traffic, full rollout, and pre-described stop conditions.
- Environment schemes — composition of environments, the active set of values in each, and the route for promoting state from test to production.
- Rollback procedure — at which step the previous behavior returns, who makes the decision, and which metrics confirm recovery.
- Infrastructure documentation — integration points between flags and services, composition of build steps, set of metrics and alerts.
- Flag inventory — lifetime and responsible person for each switch: temporary ones do not turn into permanent ones.
Configuration change process and roles
Flag configuration is the same kind of managed object as code: an edit is submitted separately, goes through review, and remains in history.
Responsibility is divided by zone — the product owner decides whether to enable behavior, the engineer is responsible for mechanics and reversibility, and the on-call engineer acts according to a ready procedure. A switch whose meaning only one person remembered stops being unmaintainable.
Case study: how flag rollout stopped breaking releases
Initial picture
Rollout flags lived in configuration files in staging environments and were edited manually: by the end of the sprint, the state of test and production environments diverged, and pre-release reconciliation took the on-call engineer 3-4 hours.
A problematic feature could only be disabled by redeployment — from the first user complaint to shutdown took 40-60 minutes.
The release cycle took two weeks; out of 20 releases, 8 were rolled back, and another 5 required urgent fixes on top of the release.
A forgotten switch was more often found through support requests than during checks: there was no separate review of flag changes, and edit history was not stored anywhere.
Result achieved
We moved flag management to GitLab: values became a versioned artifact and go through the same checks as code. A flag change goes through a merge request with mandatory review, and periodic reconciliation prevents environments from diverging.
- Function rollback — 90 seconds instead of 40-60 minutes: switching does not require a build or rollout.
- Share of releases with rollback — from 40% to 6%.
- Release cycle — daily rollouts of 1-2 features instead of a release every two weeks.
- On-call load — 20 minutes per week for reconciliation instead of 4 hours before every release.
No release after the transition required nighttime intervention: the on-call engineer can switch a flag without access to servers, and change history is restored from the repository.
How does a feature flag differ from an environment variable?
An environment variable describes where and how a service runs: dependency addresses, logging mode, pool limits. It is read at process startup and lives as long as the environment itself.
A feature flag is a control signal on top of running code: it changes behavior without a restart and without a new release.
The difference is primarily in the lifecycle. An environment variable is a long-lived entity; it is changed rarely and deliberately. A flag is born for a specific task — gradual rollout, experiment, kill switch — and must die with it. A flag without a removal date and owner becomes an invisible logic branch after a couple of months.
The area of responsibility is also different. An environment variable answers the question “where the code points” — it is an infrastructure setting.
A flag answers the question “what the code does and for whom.” When roles are mixed and rollout is done by editing variables in a specific environment, production gets a configuration that no one reviewed.
And their risks are of a different class: an error in an environment variable breaks startup — noticeably and loudly; an error in a flag does not crash at all, and some users quietly get the wrong branch.
The number of combinations grows as a power of two, and tests cover a small share of states, so flags are kept in a separate registry with an owner and lifetime.
The practical boundary is simple: a value describes the runtime environment — an environment variable; a value determines what the audience will see and when — a flag.
All development environments read one flag registry, but with different targeting rules, and behavior is verified before reaching users.
Engineer concerns: availability, audit, flag cleanup
A flag is a runtime dependency: the application at startup and on every request asks the flag service what to enable for a specific user. This means it is subject to the same requirements as production services — and engineers ask the same five concerns before starting. Below is how each is addressed in practice.
What happens if the flag service is unavailable?
The application starts on a local cache of the last known state, and default values are hardcoded into the service configuration. If the storage is unavailable, traffic does not fail: critical switches are read from the local copy, and only change recording is restored.
The cache is updated by subscription, so divergence between nodes is measured in seconds, not minutes.
Who changes flags and how — is a trail left?
An audit record is written for every switch: author, time, old and new value, link to the task and environment. The journal is read-only and is exported to an external event store, so a switch can be reconstructed in the incident timeline. Changes to segmentation rules are logged separately, not only the flag itself.
How do you separate team permissions?
Team access rights are separated by role and environment: in staging environments, the team manages its own flags; in production, a change goes through confirmation by the service owner.
Flags are tagged with owner and product, so someone else's switch cannot be touched accidentally. There is a read-only mode for on-call engineers and analysts.
Flags will accumulate — who cleans them up?
Every flag has an owner and lifetime; one that is disabled and not requested for a given period goes into a report. After that it is deleted together with the dead code branch — otherwise the configuration turns into a dump. This cleanup cycle for outdated flags is planned from the start, not after the first incident.
What about support after implementation?
Post-implementation support includes on-call duty for the flag service, alerts for subscription delay and error growth, a process for adding new switches, and incident analysis.
We run the migration in parallel: the old mechanism and flags work simultaneously, and we shift traffic service by service.
Let's discuss the task: what to send for an estimate and plan
The implementation estimate is calculated by scope of work, not “by eye,” so we need a snapshot of your infrastructure. A partial set of input data will also work — based on it, we can name stage timelines and the order of work instead of an “from-to” range.
What to send for the estimate and plan:
- List of services and environments where feature flags are needed: how many staging environments, how configurations differ, where parameters are stored.
- How releases are rolled out now: manual steps, who switches system behavior, whether one feature can be rolled back without touching the others.
- Roles and access scheme: who is allowed to change a flag in production, whether an audit log and second-engineer confirmation are needed.
- Current pain points: what breaks most often during rollout, how long rollback takes, where code has to be temporarily patched.
- Constraints: release cycle, windows for changes, requirements for environment isolation and security.
- Target indicators: within what time functionality must be disabled and by which segments changes must be rolled out.
Based on this data, we will return with a plan: separately, discovery and design of the flag model; separately, a pilot on one service; separately, rollout to the rest and handover of documentation to your team.
Stage timelines depend on the number of services and the degree of rollout automation — we fix them in the plan.
Describe the task — we will return with an estimate and work plan that shows what we do at each stage and what you get as a result.