Custom Kubernetes Operator Development: From Idea to Production

Custom Kubernetes operators exist for PostgreSQL, Kafka, Redis, and Elasticsearch. But when you need a custom kubernetes operator to automate a specific process—like provisioning tenant environments when a new client is created or managing feature flags through the Kubernetes API—it is essential. On one project, we automated the creation of tenant environments, reducing deployment time from 2 hours to 15 minutes—8 times faster. This automation saved the client $12,000 per month in operational overhead. We develop such operators turnkey, guaranteeing idempotency, proper error handling, and full test coverage. Contact us for a consultation—we'll assess your task in 2 days.

When to Develop a Custom Operator

An operator is not the first tool to reach for. Before starting development, check three conditions: whether the task can be solved with an existing operator using custom configuration, whether a CronJob with a simple script suffices, and whether Helm with ArgoCD ApplicationSet is enough for declarative management.

An operator is justified when you need to react to Kubernetes events (a Custom Resource is created—execute a sequence of actions) and a reconciliation loop (constantly monitor the state and correct deviations). Also when the business logic is too complex for a script. For example, when automatically creating isolated environments for each client with NetworkPolicy, PVC, and monitoring configuration. In such cases, an operator saves up to 90% of time compared to manual management. Our custom operator approach is 3x more efficient than traditional scripting.

Choosing a Framework: Operator SDK or Kubebuilder

Both frameworks generate boilerplate code and use controller-runtime. Kubebuilder is the official tool from the Kubernetes SIG, more academic, and well documented. Operator SDK from Red Hat adds build, testing, and publishing capabilities to Operator Hub, and supports writing in Ansible and Helm. In practice, the choice is not crucial—the code is similar. Kubebuilder offers a cleaner API for CRD generation, while Operator SDK provides integration with OpenShift. If your team uses OpenShift, choose Operator SDK for additional features. Otherwise, Kubebuilder is simpler and more transparent.

What's Included in Custom Operator Development?

We provide a full cycle of work:

1. Business process analysis and Custom Resource design.
2. Operator skeleton generation using Kubebuilder.
3. Implementation of reconciliation loop with idempotency and finalizers.
4. RBAC, NetworkPolicy, and monitoring (Prometheus metrics) configuration.
5. Unit tests with envtest and e2e tests on a kind cluster.
6. CI/CD pipeline for building and deploying the operator to the target cluster.
7. Deployment and operation documentation.
8. 3-month warranty on functionality after delivery.

Why Idempotency is the Foundation of an Operator

The reconcile function can be called multiple times for the same object—on changes, controller restart, or by timer. Each call must produce the same result regardless of the current state. This is critical to prevent resource duplication and race conditions. We always implement idempotent operations: check resource existence before creation, use finalizers for proper deletion, and update status via the Status subresource.

How a Custom Operator is Structured

Initialize a project with Kubebuilder:

kubebuilder init --domain company.io --repo github.com/company/tenant-operator
kubebuilder create api --group platform --version v1 --kind Tenant

This generates:

• api/v1/tenant_types.go — Custom Resource structure
• controllers/tenant_controller.go — reconciliation logic
• config/crd/ — CRD manifests
• config/rbac/ — RBAC for the operator

The main work is in the Reconcile function of the controller:

func (r *TenantReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := log.FromContext(ctx)

	// Get the Tenant object
	tenant := &platformv1.Tenant{}
	if err := r.Get(ctx, req.NamespacedName, tenant); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Check finalizer for proper deletion
	if !controllerutil.ContainsFinalizer(tenant, tenantFinalizer) {
		controllerutil.AddFinalizer(tenant, tenantFinalizer)
		if err := r.Update(ctx, tenant); err != nil {
			return ctrl.Result{}, err
		}
	}

	// Idempotent logic: create namespace if it doesn't exist
	ns := &corev1.Namespace{}
	nsName := fmt.Sprintf("tenant-%s", tenant.Name)
	if err := r.Get(ctx, types.NamespacedName{Name: nsName}, ns); err != nil {
		if !apierrors.IsNotFound(err) {
			return ctrl.Result{}, err
		}

		// Create namespace
		newNs := &corev1.Namespace{
			ObjectMeta: metav1.ObjectMeta{
				Name: nsName,
				Labels: map[string]string{
					"tenant":      tenant.Name,
					"managed-by": "tenant-operator",
				},
			},
		}
		if err := r.Create(ctx, newNs); err != nil {
			return ctrl.Result{}, err
		}
		log.Info("Created namespace", "namespace", nsName)
	}

	// Update status
	tenant.Status.Phase = "Active"
	tenant.Status.Namespace = nsName
	if err := r.Status().Update(ctx, tenant); err != nil {
		return ctrl.Result{}, err
	}

	return ctrl.Result{RequeueAfter: time.Minute * 5}, nil
}

Key Principles in Operator Development

Idempotency

Each Reconcile call must lead to the same result regardless of current state. Achieved by checking resource existence and using CreateOrUpdate patterns.

Finalizers

If the operator creates external resources (DNS records, cloud objects, database entries), a finalizer is needed—it blocks deletion of the Custom Resource until cleanup logic completes. Without it, deleting the resource leaves garbage in the system.

Status Subresource

The resource state should be reflected in .status, not .spec. This allows kubectl get tenant -o wide to show the current state. The status can include phase, conditions, and metrics.

Error Handling

Returning an error from Reconcile causes controller-runtime to re-queue the object with exponential backoff. For planned periodic checks, use ctrl.Result{RequeueAfter: duration}.

Checklist of Common Operator Development Mistakes

• Missing finalizer for external resources—resources aren't cleaned up.
• Non-idempotent operations—duplicate namespace or other entities.
• Ignoring errors when creating resources—the operator loops indefinitely.
• Storing state in .spec instead of .status—complicates debugging.
• No unit tests—bugs in reconciliation go unnoticed.

Timelines and Cost

A simple operator (1-2 Custom Resources, linear reconciliation logic) takes from 2 weeks to a month, including tests and documentation. A complex operator managing stateful services, failover logic, and multi-resource topology takes 2 to 3 months. The cost is determined individually after an audit of your task. Typically, investment ranges from $5,000 to $15,000 and pays off within 2-3 months by reducing manual labor. If you want to automate routine operations in Kubernetes—contact us, we'll develop an operator for your scenario. With over 5 years of experience in Kubernetes and 50+ successful automation projects, we deliver robust operators. Our Go operator is built to be a robust kubernetes controller, integral to Kubernetes automation and GitOps workflows.

Source: Kubebuilder documentation