Developing a Terraform provider: where automation breaks
As long as internal services are described manually — in a wiki, a spreadsheet, or a note next to the code — infrastructure lives separately from that description.
An engineer makes a change through the control panel, and it appears in the document a week later or does not appear at all. Within two or three releases, no one can confidently say what configuration is currently in the production environment and why it is exactly that way.
The discrepancy between configuration and reality surfaces at the most inconvenient moment: during rollback, when migrating to another environment, or when analyzing an incident.
A script that “always worked” relies on a manual edit from six months ago that even its author does not remember. Recovery turns into archaeology, and bringing up the production environment from scratch turns out to take longer than agreeing on a maintenance window.
Developing a Terraform provider addresses this differently: internal services are described declaratively, in the same way as cloud resources, and become part of the single source of truth for infrastructure.
Any deviation of the plan from the actual state is visible before apply, not after. A manual edit stops being anonymous — it either ends up in the description or is rolled back by the next apply.
Next, we break down how this works: which entities should be moved into resources and data sources, how to build infrastructure management automation without manual edits, and what to do with already accumulated drift.
What a unified management layer for internal services provides
A custom provider turns internal services into the same kind of declarative resource as cloud resources: a unified management layer where the service description lives in the repository, is applied through Terraform, and goes through code review. What follows are measurable consequences visible in the very first week of work.
A single description instead of manual edits
There is one source of truth: the service's environments, permissions, secrets, and monitoring points are described in one file. Every change goes through an apply plan, and the discrepancy between the description and the real state is visible before rollout, not after a failure.
A new service is provisioned from a template: environment preparation takes one apply instead of days of manual operations, and an entire class of failures — “edited manually and forgot to commit” — disappears.
Separation of permissions and environments by default
Roles, environment boundaries, and access levels are baked into the provider itself, not into a wiki instruction. A developer describes a service but cannot grant themselves extra permissions or affect someone else's environment — the attempt is rejected at the plan stage.
Recovery after a failure becomes the apply of a known description: the environment is brought up from configuration in minutes rather than assembled from the on-call engineer's memory, and the result matches the original.
Predictable environment changes, fewer manual operations, faster rollout of a new service, and recovery — these are the effects of one layer, not a set of disparate scripts.
Comparing approaches: scripts, templates, a wrapper, or a custom layer
What to choose is determined not by taste but by the number of internal services that need to be described declaratively, and by who will maintain those descriptions six months from now. Below are the real alternatives teams choose between, with their limitations.
| Option | When it fits | Limitations |
|---|---|---|
| Manual scripts and instructions | A one-off environment, one or two services | State is not tracked, environments drift, rollback is manual |
| Configuration templates | Typical services with rare differences | Copies are edited manually, the change plan is not reviewed |
| A wrapper over an existing tool | A single entry point on top of an existing core | Does not manage the resource lifecycle, some parameters are unavailable |
| A custom automation layer from scratch | The resource model is unique, ready-made tools do not fit | State, import, and rollback are done in-house |
| Extending an open-source solution | The model is close to ready-made, but some capabilities are missing | Dependence on external code, edits must be carried over during updates |
When templates and scripts are enough
While there are few services, two environments, and configuration changes once a quarter, the “template plus instruction” combination is cheaper and reads faster.
The limit comes when there are more copies than people: discrepancies between environments are searched for manually, and restoring the environment after a failure becomes a separate task.
When you cannot do without a custom layer
If a service has its own lifecycle, dependencies between objects, and rules that are not present in ready-made tools, neither a template nor a wrapper will hold them.
That is when Terraform provider development begins: we describe resources, pre-apply checks, and import of already existing objects so that changes proceed predictably and state does not drift from reality.
Stages of work: from infrastructure audit to code handover
We work in short iterations: first we record what is currently done manually, then move it into the provider and test it in a test environment. This order allows infrastructure to be changed without stopping services and does not leave the team alone with incomprehensible code.
What we capture at the start
We review the list of services, environments, and access roles: what is created manually, what is already automated, and where descriptions diverge from the actual state. Separately, we collect constraints — security rules, the change approval procedure, and audit requirements.
The output is a resource map and a list of access gaps, not a general description of “how everything is set up.”
- Infrastructure and access audit: inventory of services and permissions, reconciliation of the real state with descriptions.
- Target description schema: which entities become resources and data sources, and how they relate to each other.
- Provider development and configuration: implementation of resources, import of existing objects, correct handling of changes and deletion.
- Testing in a test environment: creation, modification, and rollback on a copy of the environment, reconciliation of the plan with the expected result.
- Pilot operation: the team works through the new tool in a limited area, while we support and fix issues based on feedback.
- Handover and team training: code walkthrough, deployment and update procedure, and incident analysis regulations.
How we verify the result before handover
We hand over not a “working script” but a reproducible cycle: the change plan matches the expected one, a repeated run does not produce extra operations, and rollback returns the configuration to its previous state.
A separate readiness criterion is that the customer's team performs a typical change on their own without our prompting.
Artifacts in the repository: code, diagrams, regulations, access
We hand over not only working code but the entire surrounding framework that allows your team to live with it without us: from service descriptions to instructions for the on-call shift. Below is what exactly remains in the repository after the work is completed.
Code and infrastructure description
- Terraform provider source code with a description of each resource and data source: which parameters are required, which are computed, and how conflicts are handled during changes.
- Environment configurations — separate parameter sets and isolated states so that testing cannot affect the production environment.
- Ready-made modules for typical services: connecting a new service comes down to a few lines instead of copying someone else's file in full.
- A set of checks and usage examples — a sandbox where you can see how a resource behaves during creation, modification, and deletion.
Diagrams and regulations for the team
- Diagrams of relationships between services, environments, and external systems: on one page, you can see what depends on what and what a change will affect.
- Access policies and roles — who reads state, who changes it, who applies changes in the production environment; separation is baked into code and checked automatically.
- Change regulations: the procedure for releasing a new version, backward compatibility rules, and the rollback procedure.
- Instructions for the on-call shift: typical failures, how to distinguish them from infrastructure failure, and what to do step by step.
Case study: 40 services managed by Terraform instead of manual edits
A team came to us with 40 internal services living in several environments, and every environment was assembled manually.
We moved everything to a unified infrastructure description in Terraform and developed a provider for internal services — for those entities for which no ready-made resources existed.
What it was like before
Rolling out a new service took two to three days: an engineer set up the network, load balancer, databases, queues, and permissions according to a 30-step instruction.
A release required 15–20 manual edits, and some of them diverged from what was already running in production — environments gradually drifted apart.
Recovery after a failure was hindered by the fact that no one knew the current configuration for certain: it was reconstructed from logs and service polling. About half of the incidents were caused not by a code error but by configuration drift.
What it became after
A service is described by a file in a shared repository, and the provider brings infrastructure to the desired state. Rolling out a new service was reduced to 40 minutes, manual edits per release went down to a few instead of 15–20, and all of them go through review as code.
Environment recovery and rollback come from the description, not from the engineer's memory: we restore a faulty configuration in minutes instead of hours.
The class of failures “configuration drifted between environments” is gone — discrepancies are visible at the plan review stage before apply, not in production.
What to do if infrastructure is already described manually?
There is no need to rewrite everything at once — that is exactly where such projects stall.
Environments already configured manually continue to work, and the description is introduced alongside them: at first, it only reads the current state and builds a change plan without applying anything. As long as the plan shows discrepancies, control remains with people.
The first step is importing existing resources: each object is bound to the description and enters the state file, but the configuration itself does not change.
The goal of this stage is an empty plan: it means the description exactly matches reality and the next run will not destroy the working environment. Discrepancies are resolved one by one — usually they are attributes set manually and forgotten.
The migration order is governed by blast radius. First come stateless services without external traffic — queues, caches, and test environments: a mistake there costs a restart, not downtime. Then networks, load balancing, and access permissions, followed by data stores.
What carries traffic is migrated last and one at a time — then rollback is limited to a single component.
Step-by-step onboarding looks like this: the service is described, imported, and the plan is empty — only after that is control handed over to the description, and manual edits through the interface are closed off.
Between steps, we record the result: how many objects are under management, which discrepancies have been resolved, and how many services have passed reconciliation with an empty plan.
This way, the transition happens without downtime and without one big switchover, after which accidents usually occur.
Technical concerns: state, permissions, rollback, support
The concerns that arise before developing a Terraform provider are usually the same: where state lives, who is authorized to apply changes, how to roll back, and what happens after code handover. Below are brief answers; we discuss the details on a call about your task.
Where is state stored, and who locks changes?
State resides in a remote storage backend with locking during apply: only one engineer changes one environment at a time. Environments are separated by distinct parameter sets and access via service accounts, and state version history is preserved.
Who is authorized to change infrastructure?
Any engineer on the team can build a plan; apply is available only to a role with confirmation, and the production environment is separated from the test one. Every apply leaves a record: who, when, and what was changed.
How is rollback of an environment change handled?
Before apply, the plan shows all changes, including resource deletion. Rollback relies on a pinned provider version and a state snapshot; resources with data are not recreated in the process.
What happens when service interfaces change?
The new version is released alongside the old one, so existing configurations continue to work. Deprecated fields are marked with a warning when building the plan, and migration is done by importing resources without deletion and re-creation.
Who maintains the code after handover?
We hand over the repository, tests for schema and behavior, resource documentation, and the release procedure. The customer's engineers practice the process on their own tasks; support and enhancements are covered by a separate agreement.
Let's discuss the task: we'll review the infrastructure and come back with a plan
An accurate estimate of timelines and scope for a custom provider is possible only after reviewing the current state of the infrastructure.
Describe the task — which internal services need to be moved to declarative management, what boundaries the environments live in, and what timeline constraints are already fixed.
On our side, we will review the configurations, state separation, and access roles, and then come back with a work plan and a distribution of stages by timeline.
What helps us build an accurate estimate:
- list of internal services and their interfaces — what already exposes control calls and what will have to be wrapped;
- environment boundaries: how many environments there are, how parameters differ, and where the separation between the test and production environments lies;
- current state storage: where the state file is located, who has access to it, and whether there are locks for concurrent changes;
- access requirements: which accounts and roles are granted to the provider, who is responsible for permission rotation, and which actions require confirmation;
- constraints and timelines: hard dates, change freezes, maintenance windows, and rollback requirements;
- list of approvers: who on the customer's side accepts the module schema and the readiness verdict.
After the review, we will come back with a plan: the sequence of stages, team composition, points at which you can verify the intermediate result in your own environment, and the procedure for handing over the repository and documentation to your engineers.
If some services are not yet ready for code-based management, we will say so directly and suggest where to start so as not to block the rest.
Ready to start with a short, focused conversation: describe your automation task — we will come back with an estimate and a plan.