Setting Up Managed Kubernetes (DigitalOcean DOKS) for Production

Managed Kubernetes in DigitalOcean: where the production environment breaks

A manually assembled cluster holds up until the first serious peak: while the load is steady, pods start, services respond, and metrics are normal.

Things break exactly where a managed environment would already have taken part of the work off the team — in managed Kubernetes on DigitalOcean, the provider maintains the control plane, but node pools, scheduling, and update order remain with the engineers.

The first failure scenario is a cluster node failure. If there is no spare capacity in the pool, and eviction and readiness probe policies are not configured, pods on the failed node stop receiving traffic, some remain in a terminating state and are not rescheduled.

Then manual service recovery begins: an engineer brings up a node, waits for it to become ready, removes scheduler constraints — and only then are workloads placed again.

The second bottleneck is out-of-sync configurations. Changes made in place diverge from the repository, and the next deployment silently rolls them back. At the moment of incident analysis, no one can say which version of the configuration was actually running.

Third is scaling under load. Without correct resource requests and limits, without metrics for autoscaling and pre-warmed nodes, the cluster hits the pool ceiling.

Downtime under peak load occurs not because of a lack of capacity, but because a new node appears later than the request queue is exhausted.

How these three areas are addressed in a managed environment — we cover further.

What moving a production workload to a managed cluster gives you

In an unmanaged cluster, node failure is a scenario with a manual continuation: an engineer logs into the machine, takes it out of load balancing, and brings up a replacement.

While this is happening, some service replicas are unavailable, and recovery time depends on who noticed the alert first, not on the architecture.

A managed cluster turns this scenario into an automatic one: the node is marked as unavailable, pods are recreated on healthy nodes, and a replacement comes from the pool — without manual commands and without a window where the service waits for an on-call engineer.

The provider maintains the control plane, so updates and fault tolerance of the cluster itself are not on your team — DigitalOcean covers this part, while we build workloads and policies on top.

Automatic scaling works on two levels: the number of application replicas grows according to load metrics, and the number of nodes grows according to actual utilization.

Capacity is not provisioned manually for peak, but appears under load within minutes and goes away during a decline — resource consumption stops following the worst-case scenario.

Recovery time is set by configuration, not by luck: readiness and liveness probes, disruption policies for planned maintenance, and spreading replicas across zones.

This removes a whole class of failures where the cause is not infrastructure but a manual operation — a forgotten node, a change in the console that bypassed the manifests, a configuration that diverged.

Everything is described declaratively and stored in a version control system: rollback is returning to the previous version of the manifests, not recovery from memory.

We measure the effect by three metrics — service recovery time, the share of incidents resolved without human involvement, and capacity headroom at peak.

Kubernetes: managed cluster, own nodes, or hybrid

A managed environment — for example, DigitalOcean DOKS — takes away routine work: the control plane, version updates, fault tolerance of access points, and patches are handled by the provider.

But the choice is rarely framed as “managed cluster or own nodes”: a working configuration is almost always hybrid.

The limitations of a managed environment should be known in advance: there is no access to control plane nodes, privileged modes, system agents, and kernel changes are closed off, local disks are ephemeral, and network policies are limited by the provider's capabilities. This creates a fork — what lives in the managed environment and what is moved to own nodes.

The analysis starts from workload requirements, not convenience: if a service needs kernel access, custom drivers, or node-level isolation — it moves to own nodes, while the rest stays in the managed environment so the team does not spend effort on routine work.

Stages of implementing a managed cluster under load

A production cluster differs from a test one not in the number of nodes, but in that every parameter is derived from measurements: memory consumption profile, traffic pattern, and recovery time requirements.

We work in stages — from load audit to handover to operations, and each stage has a verifiable result.

1. Load audit. We collect metrics over a representative period: peak request intensity, read/write ratio, memory profile, spikes and their cause.
2. Cluster design. We separate node pools for services and system components, configure autoscaling, resource requests and limits, and pod placement rules.
3. Fault tolerance. We spread workloads across availability zones, set the number of replicas, disruption budget, liveness and readiness probes, and prevent replicas of the same service from being co-located.
4. Network, storage, access. Traffic entry point and certificates, private image registry, disks for state, network policies between services, team permissions by role.
5. Failure scenario testing. We forcibly shut down a node and an entire zone, restart pods, disable an external dependency — recording recovery time and failover correctness.
6. Observability. Metrics, logs, and alerts tied to availability objectives so degradation is caught before user complaints.
7. Handover to operations. Procedures for release updates and rollbacks, secret rotation, backup, and instructions for the on-call shift.

This order removes the main risk: when the cluster is assembled, but it is unknown whether it will survive the loss of a zone during peak hours.

What we hand over: designs, policies, procedures

After handover, the client retains not only a working cluster but also a complete set of artifacts the team can use to evolve it without us.

Everything is stored in one repository and one documentation folder, with the structure tied to environments — so the state does not have to be reconstructed from chats and control panels.

• Environment configurations — unified templates for development, test, and production environments, with differences moved into separate layers: environments do not diverge, and any change goes through review and is rolled out predictably.
• Backup designs — a map of placement zones, volume snapshot schedule, recovery targets for each environment, and the procedure for verifying backups.
• Access policies — roles, service accounts, permission restrictions in the production namespace, and the procedure for granting and revoking access.
• Operations procedure — frequency of node pool updates, maintenance windows, rollback procedure, on-call duty, and escalation criteria.
• Cluster documentation — node pool topology, autoscaling, external dependencies, integration points, and service owners.
• Recovery instructions — step-by-step recovery of the cluster and services from snapshots, rehearsed in advance on a staging environment rather than written from memory.
• Observability — a list of metrics and alerts with thresholds, dashboards for nodes, services, and external dependencies.

Case study: moving a service to a managed environment and metrics before and after

Before the move, the service ran on three manually configured nodes: system package updates followed a checklist, certificates were renewed by hand, and balancing rules were edited directly on live traffic. Deployment took an evening — build, copy artifacts, restart, verify.

A nighttime failure meant 40–60 minutes just for diagnostics: the on-call engineer first had to determine the state of the nodes and what exactly had failed.

The workload was moved to a managed environment DigitalOcean: node updates, fault-tolerant pools, and the lifecycle of the cluster itself became the platform's responsibility, while workloads, access policies, and observability remained with us.

The state is described declaratively, so the environment is brought up from the repository rather than assembled from memory.

Metrics before and after

Deployment: before — an evening and manual steps, after — a 12–15 minute automated run. Rollback to the previous version — one run instead of recovery from a backup.

Recovery after node failure: previously 40–60 minutes, now 5–7 — the node is removed from the pool, the load moves over, and a new one comes up on its own.

Manual operations decreased from 30–40 actions per month to 4–5: only version upgrades and policy reviews remained. The share of downtime due to a manual step error went to zero — not because engineers became more careful, but because those steps are no longer in the process.

How does a managed cluster differ from manually assembled nodes?

In a managed cluster, the control plane — API server, state store, and scheduler — lives on the provider side. DigitalOcean keeps it fault-tolerant, applies security updates, and is responsible for availability. Worker nodes, workload configuration, network policies, and capacity remain with us.

If both the control plane and nodes are assembled manually, state store quorum, certificate rotation, backups, and updates are added — a significant portion of nighttime failures lives exactly there.

Control plane updates in the managed option happen without stopping workloads: the provider rolls it out itself, and we choose the available version and time.

The boundary is at the nodes — we update them: we check API compatibility, take nodes out of load balancing one by one, verify, and return them back.

The selection criterion is simple: a managed cluster takes away work that does not create the product.

Own nodes are justified when non-standard extensions over the control plane or strict isolation are needed — then maintaining it becomes a separate permanent area of responsibility.

Common technical questions before cluster deployment

Before deployment to the production environment, the same doubts repeat: what to do with application state, how to update the cluster without downtime, where to roll back to if a release cannot handle the load.

Below are short answers for each point, so decisions are made before the first launch, not at the moment of failure.

How does cluster updating work, and what about downtime?

Nodes are updated sequentially: workloads are evicted to free machines, the application is waited on until ready, and only then is the node taken out of service.

We coordinate the window in advance and keep a reserve of free resources so evicted pods are placed without waiting — the service remains available throughout the work.

How do you roll back a release if it turns out to be problematic?

Each build gets an immutable tag, and revision history is stored in the cluster, so returning to the previous version takes minutes and does not require a rebuild.

The image registry is cleaned by a lifecycle rule: the last ten builds per tag remain available for rollback, and the rest are deleted.

Is default backup and monitoring enough?

We configure volume snapshots and database dumps on a schedule with storage in a separate repository, and verify recovery on a training environment — a copy that has never been restored does not count as a copy.

Cluster monitoring covers nodes, pods, and failure points: alerts arrive before degradation becomes noticeable to users.

Let's discuss the load and come back with a plan and estimate

To come back with an implementation plan and work estimate, we only need inputs about the load and constraints — we will assemble the detailed design on the call.

The engineering minimum is: how many services are moving, what their traffic profile is, and what already runs in managed Kubernetes on DigitalOcean.

Using this data, we calculate node size, zone distribution, quotas, and failure points, after which we send the work order broken down by stages.

We separately discuss the cluster handover procedure — access, configurations, update procedures — so that after launch there are no settings that only the contractor can maintain.

What to send for the estimate:

• list of services and background jobs moving to the cluster, and what remains outside it;
• load profile: peak requests, seasonal spikes, response time requirements;
• current state: how services are brought up now, what is configured manually, and the state of the image registry;
• availability requirements and acceptable downtime window during updates;
• access rules: who gets administrator rights, whether separate environments are needed for load testing;
• what should be restored after a failure and in what order, and where state backups are stored;
• timeline constraints and decisions that remain on your side.

Describe the task in free form — that is enough to discuss the load and come back with a plan and estimate.