Developing CRDs for Kubernetes: Where Automation Breaks
As long as a cluster’s custom resources fit into a single manifest and a single apply command, they look harmless.
The problem shows up later: the reconciliation controller handles an event non-idempotently, the validating handler rejects some updates, and the object hangs in an intermediate state.
Formally, the application works, but the declaration in the repository no longer describes what is actually running in the cluster.
Then manual manifest edits kick in. An engineer goes into the cluster and edits the resource directly—because that is faster than figuring out why the control component is not bringing the state to the target.
After several such iterations, state drift becomes permanent: some objects are created by hand, some from templates, and some by the previous version of the extension.
Deployment automation ceases to be reproducible: a rerun either fails or quietly leaves the cluster in a new, undocumented state.
The cost of such an extension is not a line in the logs but nighttime incident investigations and configuration drift between environments.
Deploying a new environment turns into archaeology: you need to find out which edits were made outside the repository and why the resource schema allowed them through.
We examine how CRD development in Kubernetes turns from a source of manual work into a predictable mechanism: which failure points are closed by the schema and access policy, which by the idempotency of the control component, and which by the order in which objects are applied.
What Changes After Implementing Custom Resources
While configuration lives in scripts and manual edits, every change rollout is an approval, a window, and an on-call engineer who remembers the sequence of steps.
Custom resources in Kubernetes move this logic into a description of the desired state: the operator itself brings the cluster to it and reports any discrepancy.
That is why the effect is measured not by “it became more convenient” but by the metrics we capture on a staging environment before and after implementation.
- Change rollout time. A resource description edit is applied in a single pass—minutes instead of an approved window and a manual sequence across several services.
- Share of manual operations. Routine actions—granting access, rotating secrets, scaling, updating configuration—move to the operator; the engineer is left only with what requires a decision.
- Eliminated failure class. Drift between “what is described” and “what is actually running”: after a node failure or manual intervention, the operator returns the object to the specified state without manual troubleshooting.
- Recovery time after failure. Rollback is reverting to the previous resource description, not reproducing a chain of commands from memory; the result does not depend on who is on call right now.
- State consistency. Related objects read a single source of truth: changes pass validation and are applied in full, not as half a set.
These five items are what we include in the acceptance checklist—each has a before-and-after measurement, so the result is visible in numbers, not in report wording.
Ways to Extend a Cluster: Comparing Approaches
A cluster can be extended in three ways, and the choice between them is not a matter of taste but of whether the object has its own lifecycle logic.
If it does not, a recording format and schema validation are enough; if an external system must be brought to the desired state, you cannot do without your own controller. Here is how these options differ in applicability and maintenance cost.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made operator template | Typical tasks: certificate issuance, service publication, backups, network policies | The set of fields and update order are defined by the vendor; you cannot add your own validation rules, and if it diverges from internal regulations, you are left maintaining a fork |
| Custom resource with a schema description, without a controller | A single recording format for settings for people and the build pipeline, read by an external tool | The server only stores and validates the object: no one recalculates status, and there is no self-healing or reaction to divergence from reality |
| Resource plus a custom controller | Managing external systems and bringing reality to the desired state | More expensive to maintain: retries, access separation, leader election, backward compatibility of the schema during updates |
The options are often combined: first they describe the resource schema and migrate existing configurations to it, then add a controller when manual checks and external scripts outweigh the logic.
Limitations should be discussed before development begins—otherwise the extension will have to be rewritten at the first cluster update.
Stages of Implementing a Custom Resource in a Cluster
Implementing a custom resource changes cluster behavior, so an error in the API schema or controller permissions surfaces already under live workloads.
We run the work so that every decision is validated on the preceding environment: first we fix the task boundaries and cluster state, then we write code, then we deliberately break it on staging. Below is the sequence that implementation follows.
- Cluster audit. Inventory of existing custom resources, controllers, and permission bindings: what is already declared and what the new resource may conflict with by API group and names.
- Resource schema. We agree on spec and status fields, validation rules, API versions, and the strategy for changing them. The result is captured as manifests in the repository.
- Controller implementation. State reconciliation logic, retries, leader election when there are multiple replicas, a separate status subresource—so that status updates do not conflict with the spec.
- Permissions and isolation. Roles, bindings, and service accounts with the minimum necessary set of permissions; we verify that the controller cannot exceed them.
- Load testing. A run on a staging environment close to production: mass object creation, node failure, controller restart, external system unavailability. We look at reconciliation time and the absence of duplicates.
- Rollout and handover to operations. First, some namespaces under metric observation, then the entire cluster. After that—documentation on the schema and permissions, review of typical failures, team training: it will be able to add fields and versions on its own.
What We Hand Over: Configurations, Schemas, Procedures
The delivery is not an archive of manifests with a wish of good luck: it is a set of artifacts your team can use to stand up a staging environment from scratch and investigate an incident without calling us.
We agree on the contents before work begins, so at acceptance you check specific files and documents, not promises.
- Resource schema descriptions. CRD definitions with field validation, default values, and versioning: an invalid object is rejected at the input rather than surfacing an hour later in cluster operation.
- Deployment configuration. Controller manifests, rule sets, resource requests and limits, separation by environment—test, pre-production, production. One source, different values.
- Access policies. Roles and bindings: what each service can read and change, what permissions the on-call engineer has, and who is allowed to edit the resource definitions themselves.
- Rollback procedure. How to revert the previous definition revision and what happens to already created objects in the process—separately for compatible and incompatible changes.
- Schema update procedure. The rule by which a new version is added alongside the old one, and the condition under which the old one is removed without losing objects.
- On-call instructions. Typical signals, what they lead to, what is safe to do on the spot, and when to involve a developer.
- Solution documentation. Component interaction diagram, failure points, known limitations, and example objects for your team’s frequent tasks.
Each item is verified on a demonstration environment: you run the update and rollback with your own hands before the solution goes to production.
Case Study: Eliminating Manual State Edits in a Cluster
Before: State Lived in the Hands of the On-Call Engineer
A Kubernetes cluster served several products: services, databases, queues. Part of the state remained outside manifests—the on-call engineer manually expanded limits, edited labels, and recreated stuck pods.
Over a quarter, such interventions totaled more than forty, and each left drift: the repository described one thing, while the cluster lived by another. Review never saw these edits, rollback did not reproduce the result, and the procedure rested on two people.
What We Moved into the Resource and Controller
The domain was described as a custom resource: the spec contains exactly the fields the on-call engineer used to fill in.
The controller compares the desired state with the actual state and brings the cluster to it—creates missing objects, deletes extra ones, and recreates damaged ones. Recovery became code with tests rather than a sequence of commands. Manual changes to managed fields were closed off, and drift became impossible.
What Changed
| Metric | Before | After |
|---|---|---|
| Manual operations per quarter | 40+ | 3 |
| Incidents due to drift | 8 | 0 |
| Pod recovery after failure | 15-20 minutes | less than a minute |
| Releases without rollbacks | 82% | 97% |
The remaining on-call calls are external outages in adjacent systems, not cluster maintenance. The same approach was extended to two more domains: first we describe the resource and reconciliation rules, then we move the logic out of instructions and into the controller.
How a CRD Differs from an Operator and Where the Boundary Is
A CRD extends the API server with a new schema: resource fields, types, requiredness, default values, validation rules. It is a contract and a record in storage, but not logic.
A CRD by itself does nothing—the created object just sits there, and no one compares reality with what is written in it.
State management appears when a controller runs alongside the schema. It subscribes to events and runs the reconciliation loop: reads the desired state from spec, compares it with what is observed in status, and takes steps to bring them together.
The key requirement is idempotency: a repeated pass produces the same result rather than creating a second entity instead of the first.
An operator is a CRD plus a controller plus supporting scaffolding: access permissions, validating webhooks, finalizers for correct deletion, schema versions, and rules for converting objects between them.
The boundary of responsibility runs along the question: the schema is responsible for the correctness of the data that enters it, while the controller is responsible for bringing the external system to that data.
Hence the choice of approach. If you need a single configuration format, field validation, and clear errors at input, a CRD and a validating webhook are enough.
If the resource describes something live—a database cluster, an external system, certificates—then without a controller the status will remain empty, and failures will have to be tracked manually.
We draw this boundary explicitly: first we fix the schema and status conditions, then we decide which reconciliation steps are actually needed, so that an extra controller does not appear where a resource description is sufficient.
Common Technical Concerns Before Starting Work
Before starting, people most often ask the same things: what will happen to already created objects if the resource description expands, how to update the schema on a running cluster without bringing down operators, and who exactly will get the right to change the new resource type. Below are short answers to these concerns.
We design the schema with expansion in mind: new fields are added as optional and become required only when all sources populate them. This noticeably simplifies manifest migration.
Can the schema be updated without downtime?
Yes. New fields are applied to the API while existing objects continue to be read and served. They become required after all sources populate them, and field removal goes through a deprecation stage—rollback remains possible.
What happens to already created objects?
They continue to work: the operator reads the fields it knows about and does not interfere with new ones until it updates its logic. We migrate existing manifests to the new schema gradually, file by file, without editing the entire repository at once.
How is access separated?
Permissions are granted for a specific resource type and set of actions, not for the entire cluster: a team manages its own objects in its own namespace, while changes to the resource definition itself remain with the platform team. Permission sets are verified by a test before rollout.
What should be done if something breaks after an update?
We keep the previous version of the resource description and the state of objects, so recovery is applying the former schema rather than restoring from a backup. We first reproduce incidents on a test environment with the same manifests.
Let’s Discuss the Task: What Is Needed for an Accurate Estimate
To ensure the estimate and work plan are based on facts rather than guesses, we need project inputs. CRD development for Kubernetes rarely starts from a clean slate: there are almost always running clusters, custom operators, and established rules. Below is the minimum set that removes most unknowns before work begins.
- Cluster composition: how many environments (development, test, production), which Kubernetes release branch is used, and whether there is a separate load-testing environment.
- What already exists: custom controllers, ready-made CRDs, in-house scripts—what we reuse and what we carefully retire.
- Behavior requirements: which objects we create, which fields we validate, how we handle conflicts, and what should happen during rollback.
- Access and boundaries: who grants permissions in the clusters, whether there is an isolated environment with no external access, and how policy changes are approved.
- Checks: which scenarios must pass before rollout, whether manifest autotests and a run on a staging environment close to production are needed.
- Timeline and startup procedure: by what date a working environment is needed, what external dependencies exist, and when changes are allowed.
Agreeing on scope takes one conversation: we fix what is included in the first iteration and what is reasonable to postpone. Then we come back with an estimate and plan—stages, owners, and points where you already see the result. Describe the task, and we will tell you which inputs are missing.