Nomad container orchestration: where manual deployment breaks
When there are three services and two machines, manual service deployment seems manageable: connect to the server, start the process, adjust environment variables on the spot.
It breaks not at launch, but a couple of weeks later — the configuration on one node already differs from another, and there is nothing to restore the original state with: edits live in chat threads and in the on-call engineer's memory.
Environment drift accumulates quietly and surfaces at the worst possible moment.
The most costly part is the lack of service self-healing: a crashed process does not come back up on its own, liveness checks are not linked to traffic routing, and some requests continue going to a dead node until the on-call engineer notices the rise in errors.
A nighttime failure turns into a half-hour manual operation — find the node, recall the command, start the container, make sure the version is the same.
Nomad container orchestration takes this work out of the on-call engineer's hands: the service description is stored as code, the scheduler places tasks across nodes itself, monitors state, and restarts failed ones without human involvement — including on another node if the current one is unavailable.
The same artifact goes to test and production from a single description, so environment drift stops being the norm. We break down how this is addressed.
What changes after migration: rollout speed and class of failures
Rollout stops being a manual sequence: the task description in the configuration defines how many instances to update at once, how long to wait for readiness checks, and what to do if they fail.
The HashiCorp Nomad scheduler starts new instances alongside the old ones, shifts traffic to them, and only then shuts down the previous ones; if the check fails, it returns the previous version with a single command.
The rollout speed of one service drops from half an hour of manual actions to a few minutes, and a release of a dozen services stops being a nighttime operation.
Service recovery time is determined not by the on-call engineer's reaction speed, but by restart and rescheduling policies.
A failed instance comes back up on its own — a configured number of attempts with increasing delay between them — and if a node fails, its tasks move to free machines with placement across availability zones taken into account.
As a result, a typical incident is closed in minutes rather than tens of minutes of waiting for the failure to be noticed and handled manually.
A whole class of failures caused by environment drift also goes away: while servers were configured manually, production and test environments inevitably diverged, and the cause of downtime was found only after an incident.
Now the task description lives in the repository and is applied identically across all environments, and rollout state is visible from instance statuses and version history.
The share of downtime caused by infrastructure drops severalfold — not because there are fewer failures, but because they are no longer unnoticed and stretched out over time.
Bottom line: predictable rollout speed, recovery in minutes, and rollback to the previous version without manual assembly. Instead of an estimate of 'about a week per release,' metrics appear that show the bottleneck — build, readiness check, or lack of resources under load.
Deployment options: single node, cluster, hybrid setup
Nomad container orchestration can be installed on one machine or several — the difference is not in capabilities, but in what happens if a node fails.
The mode is chosen before installation: a single node can be converted to a cluster later, but state and task restart rules will have to be rebuilt.
A separate case is the hybrid setup: some services remain on the previous machines with manual management, while the new part already goes into the orchestrator.
This is how migration is done without stopping processes, but the price is different levels of observability — old services have neither health checks nor automatic restart.
| Option | When it fits | Constraints |
|---|---|---|
| Single node | Internal service, trial run, predictable load | Node failure stops everything, updates happen with downtime, no load headroom |
| Cluster, three nodes or more | Production workloads that need continuity | Requires quorum, reliable network between nodes, and external state storage |
| Hybrid setup | Some services cannot yet be moved | Different observability levels, different access policies, more complex rollback |
The common constraint across all three options is state: databases and files are moved to external volumes, otherwise restarting a task on another node will lose them. Beyond that, the modes differ in fault tolerance and the amount of manual operations during updates.
Work sequence: audit, design, configuration, validation, handover
Orchestration implementation is not just cluster installation, but a sequence where each step produces a verifiable result and can be rolled back.
We go from inventory to a placement design, then to workload migration and failure testing, and only after that hand the cluster over to the team with operating procedures.
- Infrastructure audit and service inventory. We gather a complete list of processes, their dependencies, availability requirements, load profiles, and sensitivity to latency. The output is a service map that shows what can be run in the orchestrator at all and what requires a separate mode.
- Service placement design. We determine which tasks go to the shared node pool and which are separated — stateful services, tasks with local disk, nodes for heavy background processing. We fix resource constraints, distribution rules, and prohibitions on co-locating conflicting services.
- Cluster design. We calculate the number of nodes for peak load and reserve for the failure of one or two, determine the composition of the control layer and state storage. We separately work through network zones and access between segments.
- Workload migration. We move services one by one, starting with non-critical ones: describe tasks in the Nomad configuration, set health checks, restart policies, and resource limits. Each migration is accompanied by a before-and-after metrics comparison.
- Failure testing. We deliberately shut down nodes, stop the leader control node, and simulate loss of connectivity between zones. We observe how quickly tasks move and whether the service remains available to the user.
- Update and rollback validation. We run staged version updates, verify that rollout stops when errors rise, and test return to the previous release.
- Cluster handover to the team. We provide designs, on-call procedures, the process for adding services, and standard recovery scenarios, and conduct a walkthrough with the customer's engineers.
By the end of the work, the team is left not with a 'configured cluster' but with a reproducible placement design and clear operating procedures.
Container access policies, designs, and procedures
After implementation, the customer is left not only with a working cluster but also with a set of artifacts that can be used to develop it without a contractor. We hand them over during acceptance: we verify each item on your staging environment rather than showing it on slides.
- Service descriptions — declarative manifests in the repository: resources, health checks, restart and placement strategy. Rollout and rollback are reproduced with a single command.
- Access policies — roles and tokens with least-privilege permissions: a developer does not change the production environment, an on-call engineer restarts tasks but does not edit the Nomad cluster configuration.
- Environment separation — test, pre-production, and production environments are isolated by namespaces and quotas: an experiment does not affect the production environment or consume its resources.
- Cluster designs — current topology: control nodes, worker pools for different workload types, reserve zones, and explicitly marked points of failure.
- Update procedure — maintenance windows, canary run on a subset of nodes, and the procedure and criteria for rollback to the previous version.
- Recovery procedure — frequency of state and secret backups, target recovery times, and the procedure for a lost node or zone.
- On-call instructions — common failures, diagnostic commands, and the sequence of actions so that a shift handover does not require calling the contractor.
Documents live in your repository and documentation system, not in chat threads. During handover, we run release rollback, permission revocation, and restore from backup together with your engineers — procedures are considered accepted only after manual verification.
Case study: three environments and unified rollout control
Before the project, the three environments — development, pre-production, and production — each lived their own life. Rollout was done manually: an engineer logged into machines and started containers with scripts, adjusting configurations locally and recording them nowhere.
Drift between environments surfaced in production, and rollback turned into a nighttime log investigation and another manual run.
Before: manual launches in three environments
Measurements over a month painted a grim picture. The average rollout took 40–60 minutes and depended on who was on call, the share of failed releases reached a quarter, and recovery after a failure took 25–40 minutes.
The same release was assembled differently in the three environments, so 'works in test' guaranteed nothing.
After: unified rollout control
Orchestration was built on Nomad: the description of each service became a declarative file in a shared repository, identical for all environments — only the set of environment variables differs.
Rollout is started with a single command, proceeds in steps with availability checks, and if a check fails, the system returns the previous version itself, without an on-call engineer or manual commands.
Rollout time dropped to 5–8 minutes, failed releases to less than 2%, and recovery to 2–3 minutes via automation.
A separate effect is predictability for the team: releases are deployed on any business day, not in a window with minimal traffic.
The on-call engineer is no longer a mandatory part of rollout, and the change history for each environment is in one place: you can see what changed and when.
What to do if services are already running manually
Manual launch is rarely total: usually some services are started by a script according to instructions on a separate machine, some live in old environments, and nothing is in the orchestrator yet.
A full shutdown for a 'one-day' migration is almost always more expensive than gradual migration: while one service is being switched, the others continue serving users.
We start with inventory: for each service, we record entry points, dependencies, configuration files, state, and launch method. This produces a list of dozens of items, divided into three groups — migrate immediately, migrate with modifications, leave in old environments.
Next, the Nomad cluster is deployed alongside the current environment and connected to the same load balancer.
Manual launch and the orchestrator work simultaneously: switching is done one service at a time by changing the route, the first gets 5–10% of traffic, and if metrics deviate, the rule is returned back — rollback takes minutes, not hours.
Old environments remain a separate pool for services that cannot be moved without rewriting — binding to local data, licenses, or hardware.
New services are born in the orchestrator right away, and manual launches decrease over time without a separate project to stop the business.
Common technical doubts before starting
The customer's engineers ask not about the tool, but about operations: what happens if a node fails, where secrets are stored, whether services will have to be rewritten, and what will remain with the team after handover.
We work through this during design — changing the architecture here is cheaper than redoing the cluster in production.
What happens if a node fails entirely?
The control layer is an odd number of servers; the quorum withstands the loss of one of them. Tasks are rescheduled onto live nodes, and before removal a node is put into graceful drain mode.
Where are secrets stored?
In a separate secret store, not in the repository and not in open environment variables. Access is granted by policies for a specific service, and rotation happens without rebuilding the image.
Will applications have to be rewritten?
Usually not. It is enough to read configuration from environment variables, expose a liveness check, write logs to a stream, and not keep state on the local disk. Packaging and launch description change.
What will remain with us after handover?
The cluster design, update and recovery procedures, on-call instructions, and a walkthrough of typical failures. We train the team and remain available according to the agreed support level.
We cover compatibility with the current environment and migration without downtime in the next block — along with application requirements.Let's discuss the task and come back with a migration plan
To estimate, we do not need access to the production environment — a description of what is already running is enough. The more specific the input data, the fewer assumptions in the migration plan and the shorter the path to starting work.
- list of services and connections between them: what communicates with what, where shared state is stored;
- how services are launched now and which steps are still done manually;
- availability requirements and the allowed maintenance window;
- infrastructure constraints: where nodes are located, which external systems cannot be bypassed;
- how many environments are needed, how often releases go out, who initiates deployment;
- who on the customer side is responsible for access and stage acceptance.
We come back with a migration plan: stages, scope of work for each, what is required from your team, rollback points, and the criteria by which a stage is considered complete.
The plan is based on your environment, not a generic template, so weak points are visible in advance — the network between nodes, secret storage, and the update procedure.
Work starts with a pilot: we move one non-critical service to Nomad orchestration, test behavior under load and the rollback scenario, and only then scale the design to the remaining services.
This way the first step produces a measurable result before anything in the production environment is affected.
Describe the task as it is — we will come back with an estimate, a migration plan, and a clear entry point.