Supervisord: why processes crash and don't come back up
The picture most often encountered: a queue worker is running in a terminal session, at night the session breaks, the process receives SIGHUP and dies.
No one notices—by morning thousands of unprocessed requests accumulate in the queue, and the on-call engineer brings the service up manually and clears the backlog.
The cause is that a crashed service does not come back up on its own: the kernel only terminates the process and returns the exit code to the parent.
If the parent was a shell or a temporary script, there is no one to collect the return—the process becomes orphaned, and no one initiates a restart.
The same mechanism produces zombie processes: the parent does not collect the status, the entry remains in the process table and accumulates until new descendants can no longer be created.
Supervisord solves this at the operating system level: the daemon starts as a system service, acts as the parent for each program, and therefore collects the exit code—zombies do not accumulate.
Autostart of processes on the server is described declaratively: the configuration for a program defines the command, working directory, user, autostart flag, autorestart, pause between attempts, and their maximum number.
Process logs are not lost: the output and error stream is written to separate files with rotation by size and number of copies, so investigating a night failure does not require searching for traces across the whole machine.
State is visible with one command, and manual restart of an application is performed pointwise without affecting neighboring services. Below we discuss how this is handled in practice.
What changes after implementing process control
Before implementing process control, the lion's share of downtime came from an unnoticed process crash: the service died at night, and it was learned about from morning requests.
Service recovery time consisted of detection time plus manual start—tens of minutes and mandatory on-call involvement.
Supervisord turns this into a managed mode.
Processes are described declaratively: automatic restart triggers on a failure exit code, startretries and startsecs cut off "flapping" services, exitcodes sets the list of normal terminations—a planned stop is not masked as a failure and does not cause a restart loop.
Observability of service state is built on supervisorctl status output and the event listener: a transition to a failure state goes as an alert to monitoring along with the tail of the process log.
The on-call shift sees not "server unavailable" but a specific service, cause, and number of attempts—diagnosis starts from the fact, not from a search.
The result is measurable: recovery fits in seconds and does not require a human, the share of downtime due to unnoticed crashes falls to single-digit percent, and the number of manual interventions is reduced severalfold. The load on the on-call shift stops growing with the number of services.
Supervisord and alternatives: what to choose for a service
The choice of startup method rarely comes down to functionality: almost any process manager can restart a crashed service and write a log.
The difference shows in details—who supervises the process, how startup dependencies are described, what happens after a node reboot, and how to change configuration without downtime.
Below are the options actually chosen on projects. They can be combined: inside a container, there is often its own supervisor for multiple application processes.
| Option | When suitable | Limitations |
|---|---|---|
| Supervisord as process manager | Multiple services on one machine: groups, startup priorities, unified status | Is not PID 1 — needs external autostart, does not manage OS-level services |
| System service manager | One or two services, autostart on boot, resource limits and journal | Dependencies only via targets, no flexible process groups |
| Application as PID 1 in a container | Microservice with one process and declarative configuration | Does not restart child processes; for multiple, a supervisor is needed |
| Orchestrator | Many replicas, self-healing, rolling updates without downtime | Overkill for one machine, requires manifests and separate infrastructure |
Practical rule: one process in a container—minimal external management; multiple processes on a node—Supervisord with priorities and groups; system services—to a dedicated system manager.
Startup dependencies are specified explicitly: otherwise after a reboot the service comes up before its database and cache are ready, and falls into a restart loop.
Implementation stages: from audit to handover to operations
Inventory is half the work.
Before writing configuration, our engineers collect everything that actually runs on the servers: services started manually in someone else's session, scheduler tasks, scripts left by previous contractors, "temporary" solutions that have lived for years.
Supervisord takes them under management—but only if it is known exactly what it must hold and in what order.
Order of work
- Audit of service startup. Record what runs now, who owns the process, where logs are written, what restarts on failure, and what dies silently.
- Process dependency diagram. Determine who waits for whom: the application does not start before the database, background tasks before the queue. The diagram shows the startup order and shutdown order.
- Configuration of management. Move services under Supervisord: autostart, restart rules, limits, log rotation, separate system users.
- Failure scenario testing. Kill processes, drop dependencies, reboot the machine—see whether everything comes up in the right order and whether "half-alive" states remain.
- Acceptance checks. Run the agreed checklist on test and production environments together with your team; resolve discrepancies before handover.
- Handover to operations. Provide configurations, diagram, instructions for typical operations, and incident response regulations.
This order removes the main risk—a system that works "as long as no one touches it." After handover, your team manages services with standard commands, not by searching for someone who remembers how it was started.
Configurations, restart policies, and regulations
We hand over documentation not "for show": from it, an on-call engineer who did not participate in implementation can bring up a crashed service and understand why it stopped.
Below is what physically remains with the customer in the config repository and internal knowledge base.
-
Process configuration files — one description per service in a separate directory, with paths, environment variables, attempt limits, and startup order; configurations are versioned, changes pass review.
-
Restart policy — which exit codes are considered normal, how many attempts are given for recovery, what to do on repeated crash: leave stopped, bring up with delay, or escalate.
-
Process access permissions — which user each service runs as, who may stop and restart it through the control interface, how access to logs and the socket is restricted.
-
Group and dependency diagram — what starts first, which process waits for a neighbor, which cannot be stopped alone without risk to others.
-
On-call shift regulations — "symptom → action" table, threshold at which the on-call engineer calls an engineer, escalation template, and incident logging procedure.
-
Maintenance instructions — how to add a new process, how to smoothly remove a service from rotation, where to view logs, and how to roll back configuration to the previous revision.
-
Metrics collection and alerts — process states, memory consumption growth, and frequent restarts are visible before the service finally crashes, not from a morning call.
Each item passes acceptance on the test environment: we deliberately kill processes and check whether the policy works as recorded in the regulation.
Case: service stopped crashing unnoticed by the team
Before implementation
The service relied on five processes—queue workers, task scheduler, external events handler—and was started manually from the console.
When a worker crashed, the queue kept growing, and the team learned about it from a client complaint: from failure to first signal about three hours, another same amount for investigation and manual restart. Over a month, 11 such outages were recorded, of which 8 were discovered by users, not monitoring.
What was changed and what became
Each process was described declaratively: launch command, dependencies, working directory, stop behavior, and restart limits.
Supervisor took over state control, autostart after node reboot, and automatic recovery of crashed processes—with delay and attempt ceiling so a crash loop is not masked as stable operation.
Output streams were consolidated into one log with rotation, and alerts to the on-call engineer were attached to recovery and limit-exceeded events.
The share of outages closed without a human rose from zero to 92%—recovery takes seconds instead of hours. Reaction time to a real incident reduced from three hours to several minutes: the signal now comes from the system, not the client.
Manual interventions remained 1-2 per month instead of 11, and they are related to code defects, not infrastructure. The results report included a process map, configuration, alerting scheme, and on-call regulations—service state is clear without reading source code.
How Supervisord differs from the standard service manager
The standard service manager (in most distributions—systemd) is designed for host level: one unit—one service, dependencies between units, autostart on boot, limits via cgroups, and a unified binary journal.
For system daemons this is exactly what is needed—it starts before application code and survives reboot with the host.
Supervisord covers another layer—a group of application processes within one environment and without superuser privileges.
Configuration is flat: INI sections [program:name] with number of instances, process name template, environment variables, retry policy, exit code for normal stop, and priority that sets start order; [group:...] collects a dozen processes into one manageable unit.
Flexibility of launch configuration is the key difference.
The standard manager expects the application not to fork and to behave correctly with the socket; Supervisord also works with those that daemonize, wait for a neighbor on a port, or fail during warmup—set delay before retry, attempt limit, stop signal.
Group management goes through supervisorctl: restart all processes with one command, status per instance, and unified event log—separate file per process or common stream via event listener.
Where the standard manager is more reliable: autostart after reboot, isolation and limits, reaction to crash via cgroup, not outdated pid file.
Therefore the approaches do not compete but combine: Supervisord lives under the standard service and inside holds a group of applications—the host gets predictable start, the application gets flexible retry rules and startup order.
Technical doubts and common configuration mistakes
After a reboot, the server returns not "the same state" but what is described in configuration: autostart is enabled by the autostart flag, order is set by priority—the database starts before the application that connects to it.
If the Supervisord process was started manually from a session, after reboot it won't be there: a systemd unit or init script is needed, otherwise configs are in place but processes aren't.
A new program is added on the fly: reread rereads files, update starts only new and changed groups, other processes are not restarted. The config is kept in a separate file in the included directory—then editing one service doesn't affect neighbors.
What happens after server reboot?
Only programs with autostart=true will come up, and the daemon itself via a systemd unit with autostart. Manual start from the terminal is not preserved.
Can a service be connected without downtime for others?
Yes, reread and update affect only changed groups. If there is a syntax error in the file, the daemon will reject entirely and leave the previous version running.
How does the system behave when memory is low?
OOM-killer chooses the victim by oom_score_adj. The daemon brings the crashed process up again, and without a startretries limit this creates a restart loop that finishes off neighbors.
What to do if the application spawns child processes?
One signal to the main process is not enough: stopasgroup and killasgroup are needed, otherwise workers remain hanging and hold the port or a database lock.
What mistakes are most common?
Overly high startsecs hides crashes on startup, missing environment gives a service without needed variables, logs without rotation fill the disk. A separate item—running as root: the daemon allows it until privilege isolation is required.
Let's discuss your task: what to send for evaluation
To come back with a solution diagram and work plan, not general wording, we need a snapshot of the current configuration. The more accurate the inputs, the fewer clarification iterations: evaluation relies on real processes, not assumptions about them.
Send what you have—this is enough for the first version of the plan:
- list of services: program names, launch commands, working directories, environment variables;
- startup order and dependencies: what starts first, what waits for database readiness, which processes must run as a single instance;
- current configurations, including breakdown by separate files and directories;
- how restart is currently handled after crash and after node reboot: systemd unit, script, or manual start;
- where logs are written and how long stored: Supervisord output or application's own files;
- target environment: one node or several, containers or machines, service user permissions;
- what hurts now: processes not coming up, restart loops, lost tasks, log growth.
From this we assemble a process map, propose a management scheme, and provide a work plan with stages and rollout order. Describe the task—we'll come back with evaluation and plan.