A new client in a SaaS platform is not just a record in a database. It's a namespace in Kubernetes, a schema in PostgreSQL, a DNS record, an IAM role, an S3 bucket, a billing entry, and a welcome email. Semi-manual scripts lead to errors: a tenant gets a wrong S3 bucket, DNS is not configured, or a schema is created with incorrect permissions. Every new field in a registration form requires edits in four places. Our experience shows: automation is the only way to scale without losing quality. With 50+ clients, manual onboarding becomes a bottleneck—tenant creation time grows to 2–3 hours, and the error rate exceeds 15%. The cost of a single such incident can reach $5000.
What Happens Without Automation
A typical scenario: an onboarding script in the form of a bash file that an engineer runs manually after receiving a ticket. The script includes psql -c "CREATE SCHEMA tenant_{tenant_id}", aws s3 mb s3://company-tenant-{tenant_id}, and several kubectl apply commands. The script is not idempotent, does not log, and does not roll back on failure midway. When kubectl apply fails due to an IP subnet shortage, the tenant is half-created. Support spends an hour on manual rollback. This is not a hypothetical scenario—it's what we encounter in most projects at the stage of "we have 50+ clients and can't keep up anymore." In one case, the lack of rollback resulted in two tenants sharing a common S3 bucket with overlapping keys—an incident that cost several hours of recovery and reputational damage.
Common mistake: partial provisioning without rollback
When creating a tenant, the script created a namespace and S3 bucket but failed at the database creation step. The database is not created, but the namespace and bucket already exist. If no rollback mechanism is provided, support manually deletes the namespace and bucket, risking residual cloud resources. This is a typical problem solved by an orchestrated workflow.How to Build an Idempotent Provisioning Pipeline?
Proper automation is built on three principles: idempotence (repeated execution is safe), atomicity with retry/rollback at each step, and a single source of state. We use one of two approaches depending on the stack.
Approach 1: Kubernetes Operator — for Platforms on K8s
The tenant is described as a CRD:
apiVersion: platform.company.io/v1alpha1
kind: Tenant
metadata:
name: acme-corp
spec:
plan: professional
region: eu-west-1
adminEmail: [email protected]
quotas:
cpu: "8"
memory: 16Gi
storage: 100GiThe operator (based on controller-runtime or Capsule) intercepts the creation of the object and runs a reconciliation loop: creates a Namespace, ResourceQuota, NetworkPolicy, ServiceAccount, RoleBinding. On error, the status is updated, and retry is automatic. State is stored in etcd. Tenant creation time is 30–50 ms, which is 50 times faster than the Terraform approach (up to 2–3 minutes).
Approach 2: Terraform + Temporal — for Hybrid Environments
Terraform ensures declarative configuration and state; Temporal provides orchestration with retry policies and compensation transactions.
# modules/tenant/main.tf
resource "postgresql_schema" "tenant" {
name = "tenant_${var.tenant_id}"
database = var.database_name
owner = postgresql_role.tenant.name
}
resource "aws_s3_bucket" "tenant_storage" {
bucket = "${var.company_prefix}-${var.tenant_id}-storage"
tags = {
TenantID = var.tenant_id
Plan = var.plan
}
}
resource "aws_route53_record" "tenant_subdomain" {
zone_id = data.aws_route53_zone.main.zone_id
name = "${var.tenant_id}.${var.base_domain}"
type = "CNAME"
ttl = 300
records = [var.alb_dns_name]
}
Comparison of approaches:
| Criterion | Kubernetes Operator | Terraform + Temporal |
|---|---|---|
| Idempotence | Built into the controller | Through Terraform state |
| Rollback | Compensating actions in code | Temporal compensate functions |
| State isolation | Single etcd | Separate workspace/directory |
| Creation speed | Seconds | Minutes (due to terraform apply) |
| Suitable for | Pure Kubernetes | Hybrid cloud environments |
The operator delivers instant creation and a smaller blast radius on failure. Terraform integrates more easily with AWS and has ready-made modules. The choice is yours—we help assess what fits.
Why Is Idempotence Critical for Provisioning?
Idempotence guarantees that re-running the pipeline does not create duplicate resources or cause errors. Without it, an accidental restart after a timeout could create two Namespace with the same name (the first fails with an error, the second is a duplicate). In a Kubernetes operator, idempotence is built into the architecture: each reconcile compares the desired state with the current state. In the Terraform + Temporal combination, it is ensured by the state file: on a repeated apply, Terraform sees that resources already exist and skips creation. Temporal additionally saves the step history, preventing reuse of already completed activities.
Automation savings can reach 40% of onboarding costs.
How Does Temporal Ensure Atomicity?
If provisioning fails at step 7 out of 10, you need a rollback mechanism. Temporal solves this with compensation transactions: each activity has a compensate counterpart that runs on failure. For example, if Kubernetes Namespace creation succeeded but the next activity (database creation) failed, Temporal automatically calls the function to delete the Namespace. The workflow history is stored in the Temporal cluster and is accessible through the UI—you can see the exact failure step, restart the workflow from that point, or manually adjust the state.
// provisioning workflow
func ProvisionTenantWorkflow(ctx workflow.Context, req TenantRequest) error {
if err := workflow.ExecuteActivity(ctx, CreateDatabase, req).Get(ctx, nil); err != nil {
return err // Temporal handles retry based on policy
}
if err := workflow.ExecuteActivity(ctx, CreateKubernetesNamespace, req).Get(ctx, nil); err != nil {
workflow.ExecuteActivity(ctx, DeleteDatabase, req) // compensate
return err
}
// ...
}Thanks to this approach, the MTTR during provisioning failures decreases from 2 hours to 15 minutes. This is confirmed on projects with 100+ tenants.
What Isolation Levels Do We Configure?
Multi-tenant architecture requires an explicit isolation model. We configure:
- Data isolation: separate PostgreSQL schemas with row-level security or separate databases depending on compliance.
- Network isolation: NetworkPolicy in Kubernetes prohibiting cross-namespace traffic.
- IAM isolation: a separate IAM role per tenant with minimal rights via resource-based policies.
- Rate limiting: quota at the API Gateway level (requests/min), ResourceQuota at the Kubernetes level.
Secrets are generated during provisioning and stored in Vault or AWS Secrets Manager—depending on the infrastructure.
Stages of Work
| Stage | What We Do | Typical Duration |
|---|---|---|
| Audit | Collect current scripts, identify bottlenecks, assess stack | 1 day |
| Design | Choose approach (Operator / Terraform+Temporal), design pipeline for growth up to 500+ tenants | 1–2 days |
| Implementation | Write code, configurations, tests. Idempotent pipeline with rollback | 2–3 days |
| Testing | Provision a test tenant in staging, integration tests | 1 day |
| Documentation and runbook | Describe process, errors, recovery | 0.5 day |
Typical timeline: from 3 days for a simple Terraform pipeline to 5–7 days for a full orchestrated workflow. Cost is calculated individually after analyzing the current infrastructure.
What Is Included in the Work
The work includes delivery of working code (Operator, Terraform modules, or Temporal workflow), configuration for staging and production, documentation and runbook, team training (1–2 sessions), and support feedback for two weeks after delivery.
Get a consultation on onboarding automation. We will assess your project in 1 day. Contact us—we will help choose the optimal approach and avoid typical mistakes. Request an audit of your current provisioning process—we will identify bottlenecks and propose an implementation plan.