Where DevOps process maturity assessment breaks down
DevOps maturity assessment most often falls apart not at the questionnaire but at the conclusions.
A team completes the survey, gets a consolidated score by levels—and then a debate about preferences begins: some are sure that "everything is fine with us," others recall three night incidents in a month.
Both positions, without measurements, are equally unprovable, and the improvement plan is built on who speaks louder.
Manual operations during release are the most honest indicator. The build is launched from the on-call engineer's workstation, environment settings are edited by hand, and the order of steps is kept in one engineer's head.
While he is around, release proceeds predictably; on vacation or on the night shift, the same release stretches out several times over and ends in a rollback.
Infrastructure bottlenecks rarely lie where people look for them. Test and production environments diverge by versions, some settings live outside the repository, and no one can clearly answer how the environment where an error reproduces differs from the one where it does not.
Diagnostics turns into trying out hypotheses, and the last person who changed something ends up being blamed.
Incidents and service recovery are investigated from memory: logs and metrics are collected manually from different sources, time to detect and time to recover are not recorded, and conclusions remain in chats and leave along with their author.
Therefore, DevOps maturity assessment makes sense only as a measurement—with numbers for release frequency, share of manual steps, recovery time, and monitoring coverage. Let's break down how this is addressed in practice.
What measurable process maturity gives
Process maturity is measured not by a questionnaire but by four axes, for which values are taken before and after: speed of change release, service recovery time, failure rate, and share of manual operations.
The data for measurement already exists in the version control system, monitoring, and incident log.
Release speed matters not by itself but together with batch size and the share of changes that reached production without rollback.
When build, checks, and delivery run as a pipeline, the batch size decreases, and with it the probability of regression falls: a small change is easier to verify and less costly to roll back.
Industry observations DORA show a stable relationship: teams with high release frequency usually also recover service faster. It is not about heroics—rollback to the previous version is practiced, and the change is so small that the cause of the failure is immediately visible.
Failures are analyzed by class: configuration error, code regression, external dependency failure. This shows which check to add so that the class disappears.
Share of manual operations is the portion of release and recovery steps performed by a human. Each such step depends on memory and attention; translated into declarative descriptions and scripts, they stop being a source of variance and recurring incidents.
Together, the four axes give not an abstract maturity level but a work queue: what to close first and what effect this will give after one or two cycles.
Maturity models: levels, scales, and audit formats
A maturity model is needed so that different engineers assess the same environment in the same way: without a scale, "everything is fragile with us" and "seems to work" are a matter of taste.
A workable scale is five levels: manual operations and constant fires, repeatable practices tied to specific people, described processes, measured indicators, and continuous improvement.
The level is assigned separately for each domain—build and release, configuration management, observability, failure recovery—otherwise it is unclear what to fix first.
Audit formats differ in depth and in what remains as an output. You choose not the "best" one but the one suitable for the task and the scope of access to the environment.
| Format | When it fits | Limitations |
|---|---|---|
| Express assessment | A snapshot is needed before deciding to rebuild infrastructure | The level is determined by indirect signs, without validation on test environments |
| Full level-based assessment | Multiple environments, a change plan for a long horizon | Requires access to repositories and customer engineers' time |
| Targeted review of one environment | A problematic area is known—releases, secrets, monitoring | Will not show connections between domains and the overall level |
| Change support | An assessment already exists, a sequence of steps is needed | Without an input assessment, it does not give a starting point and a basis for comparison |
The practical rule is simple: if levels have not been determined for any domain, start with an express assessment and, based on its results, decide whether a full assessment is needed.
If the bottleneck is known, take a targeted review; it faster produces a measurable shift in one place.
An important compatibility detail: the results of all formats are written to the same scale and the same set of domains, so an express assessment can later be expanded to a full assessment without collecting data from scratch.
Work stages: from data collection to a change plan
We work iteratively: first collect facts, not opinions, then test the hypothesis on a narrow area, and only then propose broad changes. This way the plan relies on measured data, not general ideas about the "right process."
- Inventory of environments and pipelines. Record which environments exist, how many there are, how they differ in configuration, where manual steps remain, and how secrets and permissions are passed between teams. Output—a delivery diagram from commit to production with gaps explicitly marked.
- Collection of objective data. Take metrics for a comparable period: release frequency, share of rollbacks and emergency rollbacks, time from commit to production, recovery time after failure, size of the change queue. Build and incident logs are needed here as a source of facts, not as reporting.
- Interviews with teams. Discuss with developers, operations, and release owners where time is actually lost: waiting for review, manual checks, approvals, restoring environments after a failure. Separately note workarounds—they show where the process hinders work.
- Bottleneck map. Connect symptoms to causes: not "long releases" but a specific area because of which the release waits. For each point, indicate the impact on delivery frequency and reliability.
- Hypothesis testing on one environment. Do not change everything at once: choose the area with the greatest effect and configure it fully—from automated checks to rollback. Compare indicators before and after to confirm or reject the assumption.
- Prioritization of changes. Arrange work by the ratio of effect to effort, dependencies between tasks, and risk to current releases. Put quick improvements before structural rework.
- Handover of the change plan. Provide a plan with stages, responsibility areas, readiness criteria, and metrics by which you will see progress without our involvement.
The plan remains a working document: after each stage it can be refined, and proven improvements can be implemented immediately.
Materials that remain with your team
The set of materials is fixed at the start—otherwise the assessment turns into a presentation from which nothing can be done by hand. We give the team a set of documents that live next to the code and are updated as changes occur, rather than gathering dust in a shared folder.
- Assessment report — current maturity level by area, bottlenecks tied to real incidents, priorities, and order of work.
- Environment diagrams — which environments exist, how they differ, where data resides, how code moves between them, and what breaks when they diverge.
- Release regulations — steps from build to production, checkpoints, who decides on rollout and in which windows.
- Rollback procedure — how to return the previous version, in what time, who initiates it, and how it is confirmed that the system is operational.
- Access policies — roles and permissions by environment, issuance and revocation procedure, and a separate scenario for employee departure.
- Infrastructure documentation — composition of services, dependencies, failure points, what is covered by monitoring and by which thresholds.
- List of metrics and incident response regulations — which signals are considered an incident, who receives notifications, and when the on-call engineer joins.
A separate item is the implementation map: a sequence of steps indicating what the team closes on its own and where our engineers are needed. Thus, DevOps maturity assessment ends not with conclusions but with a working plan that can begin being executed the following week.
Case: delivery pipeline and recovery time
The customer is a product service with three environments and about ten teams.
Releases happened once every two weeks: build and deployment were done manually according to instructions in the wiki, some releases ended in rollback, and recovery after a failure took from four hours—the on-call engineer brought up the stand, looked for the required image, and repeated steps from memory.
The maturity assessment showed that the bottleneck was not in the tools but in the lack of reproducibility: no two identical deployments existed.
What was changed
Configuration was described as code, including Kubernetes state and sets of environment variables. The build became unified: the artifact is built once and promoted across environments without rebuilding.
Release was moved to a canary scheme—first traffic to one replica, automatic rollback when key metrics deviate. Each service got a recovery procedure: how to check health, what to look at on the monitoring dashboard, and how to return the previous version.
Metrics before and after
| Indicator | Before | After |
|---|---|---|
| Release frequency | once every two weeks | daily as needed |
| Share of releases without failures | ~60% | 96% |
| Service recovery time | 4-6 hours | 15-20 minutes |
| Manual steps in release | 14 | 2 |
We compared the same services in the same environments, so the difference is the effect of eliminating manual steps, not of changing load. Separately, we counted the share of incidents found by monitoring before user reports: it grew from a third to almost all.
What to do if some processes are already automated?
Partial automation is the most common case during an assessment: build and tests run on their own, images are built, but deployment to production is done manually according to a checklist, and environment settings live in chats.
The bottleneck shifts: code is ready quickly, but release waits for approvals and an on-call engineer who keeps the order of steps in his head. At these joints, time is lost and failures appear that automated tests do not catch.
Therefore, maturity assessment here looks not at whether automation exists but at process connectivity: where it breaks and the result is handed over to a human, which steps are duplicated between repositories, and what confirms the safety of a change.
We walk the change route from commit to production environment and mark every manual seam—it is a candidate for managed release.
The second focus is measurements; without them, island automation is unprovable: one cannot say whether releases have accelerated or not.
We look at release frequency, time from commit to production environment, share of rollbacks, recovery time after failure, and share of infrastructure changes that went through the repository. These numbers show where automation actually helps and where it creates a false sense of control.
Next—transition to managed releases: not writing more scripts, but giving release a single entry point, the same set of steps for all environments, rollback as a standard operation, and configuration under version control.
Then partial automation adds up to a route that is measured and improved by numbers, not by feelings.
Common doubts before starting an assessment
Before an assessment, customers ask roughly the same questions: how much time it will take from the team, whether access to the production environment is needed, and how to later prove that things objectively got better.
We answer them before the start—so that the scope of people's involvement and access boundaries are agreed in advance, rather than surfacing in the middle of the work.
How long will the assessment take and what will the team have to do?
The full cycle is two to three weeks. Of that, interviews take about six hours: one hour-long conversation each with development, operations, and whoever is responsible for releases.
The rest is collection and analysis of artifacts: configurations, Kubernetes manifests, build histories and automated test runs, incident logs. No one fills out questionnaires manually.
Is access to the production environment needed?
No. Exports of configurations and anonymized logs are sufficient; access to live systems is read-only, if it cannot be avoided.
If necessary, we work on a copy or in a customer's isolated environment: credentials are issued for the duration of the assessment and revoked after the report is delivered.
How can we understand that the result is real, not a report for the sake of a report?
We record the baseline snapshot by axes: release frequency, time from code change to deployment, share of automated checks, recovery time after failure, and incident repeatability.
We take the same set of metrics again after two to three months—the difference is visible in dynamics, not in wording.
Let's discuss the task and come back with a work plan
To come back with an assessment and work plan, rather than general words, we need an overview of your situation.
It is enough to describe the environments—cloud, own servers, or hybrid—how many services and teams are in the area of responsibility, how releases are currently done, and where breakdowns occur most often.
Separately, it is useful to know the constraints: change windows, data storage requirements, who approves access and configuration changes.
In response, we send not a presentation but a working document that can be used to make a decision:
- map of the current state: environments, setups, points of manual operations on the path from commit to production;
- list of bottlenecks prioritized by impact on failure frequency and recovery time;
- target model with stages and dependencies between them—what can be done in parallel and what only sequentially;
- work plan with order: what we close first, what we consciously postpone and why;
- metrics by which we will verify the result: release frequency, share of rollbacks, downtime, share of automated checks;
- requirements for access, roles, and environments that will be needed at the start.
You can discuss the task even without a full description: if some data is restricted, the process assessment can start with one environment or one critical service—then the picture is completed based on facts.
Describe the situation in any convenient form, and we will come back with an assessment, work plan, and responsibility boundaries.