Runtime security for containers: what breaks in production
In a typical cluster, workloads run with excess privileges: the base layer is root by default, a service account is mounted into every pod, and privileged mode is left on "just to be sure it works." At the same time, no one observes behavior inside the container — shell launch, writing to a system directory, or an outbound connection to an external network you see not in an alert but in an incident postmortem.
Image scanners check the artifact before launch and do not answer the question of what the process does after startup: which files it reads, which ports it listens on, where connections go.
That is why incidents in the cluster are uncovered after a leak or anomalous traffic, and the entry point is reconstructed from application logs rather than from behavior.
Container runtime security on tools like Sysdig moves checks to runtime.
Container behavior control is built on kernel events: reading a sensitive file, launching an unexpected process, or accessing an address outside the usual profile are compared against a baseline and a rule — anomalous process behavior is visible immediately, not a day later.
Threat detection in production here relies not on signatures but on deviation from a profile, so it also triggers on previously unknown techniques.
We capture the normal operating profile of each workload, describe allowed operations declaratively, enable response from warning to blocking, and only then touch production.
We explain how this is covered: where to collect telemetry, how not to bring down the cluster with the first rules, and who is on duty for alerts.
What workload behavior control provides
Behavior control is not about prohibitions but about a baseline: the team knows how the service behaves normally and sees any deviation immediately, not a day later from user complaints.
This baseline is what gets built when configuring Sysdig, and its effect is measurable in three areas.
Incident investigation — minutes instead of hours
The on-duty engineer no longer needs to assemble the picture manually from application logs, system logs, and traces. When an alert triggers, it is immediately clear which process launched what: the command, file, network call, and the chain of processes that spawned it.
The average time from signal to cause drops from hours to minutes, and the debate "is it the application or the environment" is settled with facts, not versions.
A class of failures that never reaches the environment
Behavior rules describe what a workload is allowed to do at startup: launching a shell in the container, writing to the application directory, accessing an unexpected address. Such events are cut off at startup — before the service accepts traffic.
A whole class of failures and compromises that previously surfaced under load ceases to exist as an incident.
Predictability during updates
A new version of the service is compared against the behavior baseline automatically: if an uncharacteristic process appears after rollout, rollback starts before degradation.
Updates stop being a lottery — the share of releases that pass without deviations grows, and service behavior remains consistent across environments.
Stability under load
When it is known which file and network accesses are normal, noise in on-call shifts drops: the on-duty engineer responds to deviations, not every event.
Workloads stop competing for resources with extraneous processes, and service response time stays within previous bounds after each release.
Summing up: incident investigation is measured in minutes, some failures never reach the environment, and updates proceed predictably. This is the substance of workload behavior control work.
Three ways to control: node agent or sidecar container
Container behavior control is built in one of three ways: an agent on each node, a sidecar container next to the application, or built-in cluster tools. The choice determines which events even make it into the log and how deeply process behavior inside the container is visible.
A specialized tool like Sysdig provides visibility at the system call level, but even for it depth depends on the mode: a node agent sees the whole node, a sidecar container sees only its pod. Below is a comparison of options with the honest downsides of each.
| Option | When it fits | Limitations |
|---|---|---|
| Agent on each node | Clusters that need a unified policy and coverage of all containers, including short-lived ones | Requires privileges on the host, adds extra load on the node and a separate update channel; the agent itself becomes something to monitor |
| Sidecar container in a pod | Control is needed for individual services, settings are configured together with the application | Shares resources with the application, visibility depth is lower: host-level events remain outside control, manifests must be edited for each service |
| Built-in cluster tools | Basic control: change audit, policy checks, response to obvious violations | Process behavior inside the container is almost invisible, there is no system call history — detection is built on indirect signs |
None of the modes eliminates blind spots: an event not intercepted at the right layer cannot be reconstructed after the fact. That is why the mode is chosen for the task, and it is decided in advance who responds to the event stream and what to do with it next.
Stages of implementing behavior control in a cluster
Implementing behavior control begins not with installing an agent but with inventorying what is already happening in the cluster.
While the real system calls and network connections of services are unknown, any detection rule turns into guesswork: first a stream of alerts, then disabling policies.
-
Audit of container privileges and profiles. We collect actual capabilities, privileged modes, and access to the runtime socket — we get a list of services with excessive privileges.
-
Coverage scheme. We select namespaces and workloads for the first stage and enable Sysdig in alert-only mode to collect a behavior baseline.
-
Setting up detection rules. We describe launching a shell in the container, reading files with credentials, writing to foreign directories, accessing the socket. We tie rules to service groups rather than the entire cluster.
-
Noise suppression. Based on accumulated statistics, we remove alerts from platform service processes and scheduled tasks — the queue remains one where an engineer is needed for individual assessmentCheck on the production environment. We run controlled scenarios and make sure the signal arrives and regular releases do not break.
-
Response regulations. We fix what counts as an incident, who resolves the signal and within what timeframes, and when a rule is moved from alerting to blocking.
-
Handover to the on-call shift. We go through real alerts, hand over the coverage scheme and escalation procedure — from then on, the rules are maintained by their hands.
What we hand over after implementation
Implementing runtime security does not end with launching an agent on nodes — after that, the system lives in your environment. We configure Sysdig so that the support team and on-call shift can work without us: edit rules, investigate events, connect new clusters.
That is why what is handed over is not a contractor's dashboard but a set of artifacts in your accounts and repositories.
- Rule configurations — declarative policy manifests: launching privileged processes, writing to system directories, accessing metadata, network connections. Changes go through review and are rolled back with a single change.
- Node coverage scheme — a map of clusters: where the agent is installed, which namespaces are excluded and why, which nodes require a separate profile.
- Access policies — a role matrix: who sees the event stream, who edits rules and enables blocking, who works read-only; separately, service accounts and their scopes.
- Implementation documentation — architecture diagram, dependencies, integration points with the event collection system, the procedure for updating the agent and checking after it.
- Response regulations — a table "event → priority → on-call shift action → deadline": when it is enough to record an incident and when container isolation is needed.
- Configuration handover — export of agent rules and profiles plus the procedure for deploying from scratch in a new environment: test, backup, or another region.
- Session with the on-call shift — review of real events on your stand so that the regulations do not remain a document no one opened.
Case study: eliminated a class of failures at startup
Initial state
The customer is a platform with several dozen containerized services, where runtime protection was formal: agents were deployed but ran on default rules in alert mode.
Any alert went to a common channel, and the on-duty engineer opened it manually and decided whether it was real activity or the service simply invoked a shell during initialization.
Threat detection time was measured in hours from an event to the moment someone looked at it, and the queue of unprocessed alerts grew faster than it could be cleared.
What changed
We collected a behavior profile for each workload and moved Sysdig into blocking mode for a narrow set of techniques: launching a shell inside a workload container, writing to sensitive paths, uncharacteristic outbound connections.
Legitimate patterns — migrations, cache warm-up, artifact builds — were put into explicit exceptions tied to a specific image and label, rather than disabling rules entirely.
We separately configured priorities: critical events go to the on-call channel, the rest to a store for review.
Metrics before and after
Detection time dropped from tens of minutes to 2–4 minutes, and the share of false positives from seven out of ten cases to one.
The number of events requiring manual review fell by about an order of magnitude: instead of a hundred cards a week, the on-duty engineer gets a few, and each is labeled with a specific container, image, and process.
Incident investigation by the team stopped being a needle hunt: a rule is either refined based on the fact or remains and is recorded in the knowledge base.
How does behavior control differ from image scanning?
Image scanning works with static content: layers, manifest, library versions, secrets and files left in layers.
The scanner compares the content against a database of known vulnerabilities and configuration rules — this happens before launch, at build time or in the registry, so findings can be fixed without affecting production.
But an image describes only what the container is built from, not what it becomes in the runtime environment.
Behavior control turns on after the process starts and looks at actual actions: system calls, file accesses, network connections, launching child processes, mount attempts.
The difference between the approaches is in the moment and subject of observation — scanning answers the question "what is inside," behavior control answers "what is happening now." An image can be thought of as a passport at the entrance, behavior as a log of actions after the door has closed.
One without the other is not enough for two reasons. A scanner is powerless against what is not in its database: exploitation of a known library in a new way, code loaded at runtime, compromised credentials, or actions of a legitimate process that was led astray. Such activity does not change a single image layer, and image scanning simply will not see it.
The reverse situation — behavioral rules without preliminary cleanup produce a stream of noise: a base image with hundreds of vulnerabilities generates false events and devalues the signal.
In practice, the approaches complement each other: scanning cuts off what is known and narrows the attack surface, while behavioral rules in Sysdig describe the allowed application profile — which binaries run, where traffic goes — and anything outside the framework becomes an event.
Detection at startup gives a window between container start and the first suspicious action, sufficient not only to record the incident but also to stop the process or isolate the container before the attack develops.
Limitations and technical doubts
Behavior control in container runtime security is not a signature antivirus but a model of what is allowed: which processes run, what they access in the file system and network.
The method requires training on real traffic, and non-standard builds and custom agents have to be explicitly added to exceptions.
Doubts are resolved in a pilot: the agent is installed in observation mode, rules only write to the log and block nothing, and services are moved to blocking mode one by one. Below are the questions asked most often.
How will this affect node load?
The agent sees system calls, but only the stream filtered by rules goes into processing, not everything. Application nodes do not require additional resources: analytics and log storage are moved to separate nodes. For latency-sensitive services, rules are configured more leniently.
How many false positives will there be and who investigates them?
During training there are alerts: we investigate them and refine rules, and typical behavior patterns of known images are already described. After the profile stabilizes, only isolated events remain, and they come with context — process, image, triggered rule.
Is it compatible with the current environment if some services are already deployed?
The agent is installed alongside the application and does not require rebuilding images: some services connect immediately, the rest as releases happen.
We separately check integration with existing security tools and log collection so that events are not duplicated, and the move to blocking mode can be rolled back by changing the profile.
Let's discuss the task and come back with an implementation plan
The first conversation is about the structure of your environment, not about a sale.
We need inputs: where containers run — in a cluster managed by an orchestrator or on separate nodes, how images are built and stored, whether there are restrictions on installing an agent on nodes, what requirements for access and log retention periods apply in the company. This is enough to estimate the scope of work and not invent architecture on the fly.
Based on the discussion, you receive:
- an environment map: which nodes and workloads fall under observation, and which we consciously exclude and why;
- a set of detection rules for your services — from launching a shell inside a container to accessing sensitive files and privilege escalation attempts;
- operating mode: first alert-only to filter out false positives, then blocking according to agreed rules;
- a scheme for outputting events to your logging and alerting system with separation by severity levels;
- a stage-by-stage implementation plan with checkpoints at each: what we deploy, what we check, what we record as a result;
- project boundaries — what work is included, what remains on your team's side, and where our area of responsibility ends;
- an estimate of work broken down by stages and conditions for starting cooperation.
If the environment is heterogeneous, we suggest starting with one environment: we will show how the rules behave on a real workload, and then roll them out to the rest. Describe the task — we will come back with an assessment and implementation plan.