Red Team / Blue Team exercises for DevSecOps: validation and training

Red Team: where defenses break in the DevSecOps environment

An attacker rarely breaks what is intentionally closed.

More often, they follow paths that no one considered a security boundary: the difference between an image that passed build checks and what actually runs in production; a service account with administrator rights “so it doesn’t get in the way”; a monitoring dashboard exposed on the internal network without authentication.

A separate category is secrets inherited by all branches and living longer than the task for which they were created.

A Red Team exercise differs from scanning in that it checks not the presence of a security control, but the ability to pass the chain to the target.

Engineers act according to an agreed scenario, within a specified window, and without destructive actions: they record entry points, persistence steps, collected evidence, and which logs missed this path.

This reveals environment vulnerabilities invisible to automated scanners — they look for known patterns, not the logic of your deployment.

Practice yields three persistent types of findings. Divergence between build and production: debug modes, manual configuration edits, images not from the expected repository state.

Excessive access: shared keys, roles without scope restrictions, accounts not revoked when an employee leaves. And auxiliary services left accessible “for debugging.”

The value of the exercise is not in the list of vulnerabilities, but in understanding exactly where the environment lets an attacker through.

The hidden attack paths found are turned into access and verification rules, and the team gains experience that cannot be learned from documentation.

What regular security testing delivers: measurable effects

Security testing makes sense when its result can be compared with the previous run. That is why before the exercise we record baseline security metrics, and after it we calculate the delta for each.

We build scenarios according to tactics from MITRE ATT&CK, so that checks reach accounts, trust relationships, and the build pipeline, rather than being limited to port scanning.

What we measure before and after

• Attack detection time — from the first suspicious action to the alert. A typical starting point is hours; after setting up event correlation, minutes.
• Share of successful passes — what portion of scenarios reaches the target resource. It decreases from run to run, and this is a measurable indicator, not a feeling.
• False positive rate — how many alerts the on-call engineer closes without action. If it is half, notifications stop being read.
• Log completeness — whether events exist along the entire path. A gap in the logs means the attack will not be reconstructed chronologically.

Which classes of failures are closed

• Lack of segmentation: one service can see a neighbor’s databases.
• Long-lived keys and excessive privileges of service accounts.
• Secrets in the repository and in build variables that end up in logs.
• Manual edits in the production environment without a trace or rollback capability.

Closure of an item is confirmed by a repeat run: if the scenario no longer passes, the failure has been eliminated, not masked.

Such a checklist becomes a working document — it is used to plan the next cycle and assess how much environment downtime was avoided through early detection.

Exercise formats: from a one-off assault to a continuous cycle

Red Team exercise formats differ not in depth, but in horizon and in what exactly they measure.

A one-off security test provides a snapshot as of a date; an exercise with the defensive shift tests people’s reaction and procedures; a continuous testing cycle keeps detection and response in working order.

The choice is determined by the question: prove that defenses do not hold, or build a process that holds them continuously.

Exercise boundaries are recorded in writing before the first step: segments and services, the work window, the contact for stopping, and the escalation procedure if training activity coincides with a real incident.

Without a stop condition, a training attack on a live environment is indistinguishable from a real one, and the on-call shift starts investigating the wrong thing.

How the exercise proceeds: stages of the engineers’ work

A Red Team exercise is not a one-off attack, but a managed sequence: first we record boundaries and failure points, then we check whether defenses will see them. Below are the stages we follow so that the result is reproducible and does not depend on the luck of a single shift.

1. Environment audit. We assemble a map of segments, entry points, and critical services, and mark what is included in the exercise and what remains outside its boundaries. Here we also agree on interaction rules and a communication channel during the run.
2. Environment preparation. We deploy an isolated environment — a copy of the working scheme on synthetic data, with no access to production. Separate networks and accounts prevent actions from intersecting with real traffic.
3. Scenario design. Each risk gets its own scenario: credential compromise, lateral movement, privilege escalation, data exfiltration. Each scenario predefines the indicator to which defenses must respond.
4. Run. We follow the scenarios and record telemetry: what made it into the logs, what went unnoticed, how much time passed from the first action to the response. We do not disrupt attempts — we document them.
5. Detection validation. We compare the rules that fired with those that should have fired, and calculate the share of detected steps and the delay until notification.
6. Results review. We consolidate findings by criticality and separate gaps in visibility, in rules, and in organizational processes — each has its own remediation method.
7. Handover of detection rules. New and adjusted rules go into monitoring, and disputed scenarios remain in the set for regular runs.

Deliverables: reports, detection rules, procedures

The exercise is closed not with a presentation, but with a set of artifacts that remain in your environment and work after the team leaves. Below is the minimum set we deliver at the end: without any of these items, the test is considered incomplete.

• Attack path map. A diagram of the Red Team’s movement from the entry point to the target resource: nodes, accounts, and techniques used in MITRE ATT&CK notation. It immediately shows which transitions leave no traces in the logs.
• Detection rules. Not a description of “what to detect,” but ready-to-use rules for your system: trigger condition, threshold, correlation window, and mapping to a technique. Each has been run on a test environment — there are no false positives on background traffic.
• Exercise report. A timeline with metrics: time to first signal, share of logged steps, number of fired and missed detections, and the points where the attacker was stopped.
• Response procedure. Who makes the decision, to whom they escalate, what they do with an isolated system, and within what timeframes. Roles are assigned to specific people and shifts, not to departments.
• Access policies. Closure of excessive privileges, separation of privileged accounts, and adjustment of segmentation — as ready-made configuration changes, not a wish list.
• Remediation plan. Weak points with priority, owner, and closure criterion. What cannot be fixed by configuration receives a compensating control and a review deadline.
• Scenario set for repeat runs. Exercise steps in a reproducible form — to verify that detections have not fallen off after a release or infrastructure change.

Case study: a hidden attack path and what changed after the review

The attack was conducted without warning the on-call shift: entry from the external environment, the goal being to exfiltrate data from the production environment.

The starting point was a service account token left in build history; from there, the scenario developed through steps described in the MITRE ATT&CK matrix as legitimate access followed by privilege escalation.

The hidden attack path looked routine: requests came from a real service, with valid permissions and familiar frequency. The first signal appeared only on the third day, when data was already being copied to an intermediate storage.

Of the fourteen scenario steps, automation flagged five — the rest dissolved in the general flow because events from the cluster, secret storage, and network were not linked into a single chain. The logs existed, but there was no picture.

What changed after the review

The review produced six findings, and all were closed with fixes, not a report.

We built event correlation by a single account and added a rule for anomalous secret access, limited the lifetime of service tokens and made rotation mandatory, and narrowed outbound connections from workloads to known destinations.

A playbook with ready-made steps landed on the on-call shift: what to check, whom to wake, which workload to isolate.

A repeat exercise a quarter later showed a shift in metrics. Detection time dropped from three days to forty minutes, thirteen of fourteen steps now produce a signal, and the average on-call response time came within ten minutes.

Detection no longer depends on the attentiveness of a specific engineer on a specific shift — a combination of rules works, not luck.

How Red Team differs from a one-off penetration test

A one-off penetration test answers the question “what holes are there in the perimeter.” Specialists work from an agreed list, find vulnerabilities, and hand over a prioritized report — then internal remediation work begins.

The value is finite: once the report is closed, the knowledge remains in the document, not in the shift’s skills.

A Red Team exercise answers a different question — “will defenses manage to notice and respond.” The attacking group imitates a real adversary: chooses a target, bypasses security controls, establishes persistence in the infrastructure, and tests the response.

What matters here is not only the fact of penetration, but also how much time passed before detection, whether escalation worked, and whether the signal reached the on-call engineer.

The key difference is the participation of the defensive shift. A penetration test does not involve it: it learns about findings from the report. In the exercise, the shift plays in real time, and the review is based on its logs and decisions, not only on the attackers’ actions.

The output is not a list of vulnerabilities, but changes in processes: correlation rules, alert thresholds, escalation procedures, and on-call checklists. The shift gains a skill that remains with the team after the exercise.

What to decide before the start: boundaries, risks, access

We record the exercise boundaries in writing before the first command: the list of segments, accounts, and services that may be touched, and a separate list of prohibited actions — attacks on industrial systems, data encryption, corruption of backups.

There, too, are the criteria for an immediate stop and the person who has the right to order it. Without this document, the exercise turns into uncontrolled impact, and its results cannot be defended before the service owner.

We agree on engineer access as strictly as the boundaries. The attacking group receives temporary accounts with a limited lifetime; the defensive group receives extended access to logs and detection systems, but without the right to change production settings.

All actions of both groups are written to a shared log, so the results show at which step defenses worked and where monitoring would have worked.

Who participates on the customer’s side and how much time will it take?

Three roles are needed: the service owner, who confirms the boundaries; the on-call engineer, who sees system behavior in real time; and the person responsible for stopping, with the right to interrupt the work. Other employees are not distracted.

The customer needs to allocate several hours for coordination and ensure the on-call shift is present on the day of the exercise.

What if the attackers find something bigger than expected?

Such findings are the goal of the exercise, but we handle them according to procedure: a critical vulnerability is not developed further, but recorded and passed to the service owner immediately, on the day of discovery.

We split the report into two parts — a closed one for the support team and an overview one for management, without ready-to-use exploitation instructions.

Let’s discuss your task and come back with a plan

For the exercise to deliver a result, and not a report for the sake of a report, we need the context of your infrastructure at the start. From you — five inputs:

• a diagram of key services and what you consider critical: where an hour of downtime is costlier than a false positive;
• log sources and detection tools that already collect events — we test real visibility rather than building it anew;
• boundaries of permitted actions: which nodes and data we do not touch, what is allowed during working hours, and which changes require approval;
• windows for the active phase and contacts of on-call staff who will confirm the legitimacy of actions when an alert fires;
• success criteria: time to detection, share of recorded steps, and quality of the review.

In response, we send a package you can use to make a decision:

• the exercise scheme: scenario, stages, roles of the attacking and defensive teams, and control points;
• scope of work divided into preparation, active phase, and review;
• timeline estimates by stage and resource requirements on your side;
• the procedure for handing over results: event timeline, log exports, and a list of blind spots;
• a remediation plan with priorities and re-testing of key paths.

Describe the task — we will come back with a timeline estimate and a work plan, not general promises.