End-to-end traffic encryption between services: TLS/mTLS

Traffic encryption between services: where protection breaks

The perimeter is usually closed: TLS termination at the load balancer, strict inbound traffic rules, regular external checks. Inside the cluster the picture is different — calls between services travel over the cluster network in plaintext.

HTTP, gRPC, requests to the cache and queues go unencrypted, and a process that has gained rights in its namespace can listen to someone else's traffic.

The problem is that observability tools do not show this. Metrics show response codes and latencies, logs show request bodies when debugging is enabled, and traces show the duration of segments.

None of these signals answers the question of whether the channel was encrypted and whether the caller's certificate was verified.

Service calls — readiness probes, metrics collection, requests to the broker — are rarely accounted for at all, so unprotected service interfaces live for years outside anyone's area of responsibility.

Hence the cost of a mistake: intercepting data inside the network hands over access tokens, session keys, and personal data in the clear, and there is almost nothing to notice this after the fact.

Network policy does not close the issue either — it limits who can talk to whom, but not what exactly they transmit.

During a security requirements review, such a segment becomes the first finding, and fixing it becomes an urgent task involving reworking the configurations of all exchange participants.

We break down how this is closed: mTLS at every interface, automatic certificate rotation, default deny for unencrypted connections — with control over whether internal traffic protection actually works, and not just declared in configuration.

What changes in reliability and timelines after implementation

Measurable effects here are easier to look at across four quantities: the share of manual operations in certificate issuance and rotation, recovery time after a service failure, the number of incidents at service interfaces, and the share of connections that undergo mutual authentication at all.

Before implementation, usually two of the four are not measured at all — and this is the first thing that changes.

The class of failures that disappears entirely is implicit trust within the perimeter. When a service accepts a request simply because it came from the internal network, any spoofing or routing error looks like "strange behavior" and is investigated for hours.

Mutual certificate validation turns this into an explicit failure at entry: the connection is not established, and the log shows who failed verification and why.

Recovery time is reduced through rotation. Previously, certificate renewal was a manual operation with a downtime window and the risk that it would be forgotten — now the lifespan is short, and replacement happens automatically and without stopping traffic.

Revoking a compromised certificate also becomes a routine operation rather than an emergency nighttime event.

A practical sign that the change has taken hold: the share of services whose connections are verified on both sides approaches one, and the number of incidents at service interfaces falls so much that they stop being classified as a separate category. Then reliability stops depending on whether a particular engineer remembers the next deadline.

Traffic encryption modes: what is appropriate where

The channel protection mode is chosen not on the principle of "stricter is better" but by the trust boundary and by who manages keys on each side.

With one-way authentication, the server presents a certificate and the client does not: the channel is protected from eavesdropping, but the caller remains anonymous.

Mutual authentication (mTLS) adds a reciprocal requirement — a service without a valid certificate will not get a connection.

Three things influence the choice of protection scheme: where encryption is terminated, whether you are ready to issue and rotate a certificate for each participant, and what happens if one key is compromised.

The transition between modes is almost never done all at once.

The most compatible option is an intermediate one, where the entry point accepts both verified and unverified connections, and the share of mTLS connections serves as the criterion for completing the migration: once it reaches zero, strict mode is enabled with a single setting.

Built-in service mesh management tools, such as Istio, support this transition out of the box, so rollback amounts to restoring the previous policy rather than rebuilding the configuration.

Inventory, trust model, and certificate issuance

Traffic encryption between services begins not with configuration but with an honest picture: who connects to whom, who is the client and who is the server in a pair, where the channel is already protected, and where data travels in plaintext.

Without such a registry, any configuration turns into guesswork, and the first forgotten service breaks the exchange.

Workflow

1. Service inventory. We collect the participants in the exchange: name, environment, who initiates the connection, which port it listens on, and what is already protected. The registry becomes the source of truth that we check against at each next step.

2. Trust model. We decide who issues certificates, how long the intermediate CA and the certificate itself live, where private keys are stored, and who has signing authority. We fix naming rules immediately so that the certificate is tied to the service, not to the machine.

3. Certificate issuance. We put issuance and renewal on automation: an agent on the node obtains the certificate, renews it well before expiration, and reports if renewal did not succeed. Only revocation and troubleshooting remain manual.

4. Pilot environment. We move the first services into observation mode: the connection is verified, but if there is a mismatch it is not dropped — the discrepancy is written to the log. This surfaces pairs of "who talks to whom" that were not in the registry.

5. Failure testing. We break things intentionally: the issuing CA is unavailable at renewal time, the node clock has drifted, the certificate is revoked, the client presented an old one. Each case must have a predictable outcome and a log entry, not a silent drop.

6. Mandatory verification by groups. We move to strict mode in pairs, from non-critical services to core ones, monitoring the failure rate and with the ability to restore the previous behavior with a single setting.

Key rotation procedures and access policies

Encryption between services configured once lasts only until the first infrastructure change: a pod was added, a service was moved to another cluster, the owner changed — and trust between components rests on one engineer's memory.

We break down which artifacts we document in writing so that the process can be reproduced without us.

• Key rotation procedure: who owns the material, what the certificate lifespan is, what is issued automatically and what manually, and by what procedure a scheduled change takes place without downtime.
• Access policies: which service trusts whom and on what basis, who is authorized to issue credentials, which scopes are allowed, and what limits each participant.
• Trust map: a diagram of connections between services and environments, encryption termination points, and the boundaries beyond which traffic is already unprotected.
• New service onboarding instructions: a step-by-step procedure — from credential request to verification that traffic is actually encrypted and the parties see each other.
• Emergency revocation procedure: who makes the decision, within what window the credential is revoked, how the compromised node is isolated, and how connectivity is restored.
• Inventory and expiry control: a registry of all credentials with owners and expiration dates, regular reconciliation, and alerts in advance of expiration.
• Private key storage rules: where it is stored, who has access, how access is granted, and how it is revoked when a role changes.
• Infrastructure documentation: up-to-date diagrams, configuration parameters, and the procedure for reproducing the environment from scratch at a backup site.

The result is predictable operations: a new service joins the trust domain in hours, key revocation takes minutes, and the security review is based on documents, not stories.

Implementation case study: measurements before and after migration

The customer is a payment service with forty internal components. Exchange between them went over an open channel, and the services identified each other by network address: access to the cluster was enough to read someone else's traffic and impersonate any component.

The certificates used at the perimeter were issued manually — from request to finished file took up to two days, and scheduled rotation became a separate event with a night window.

We started with inventory: mapped traffic flows, marked already protected paths, and recorded baseline metrics — the share of services with mutual authentication, certificate issuance time, and the number of failures during rotation.

What we changed

Inside the environment, we stood up our own certificate authority and handed its management to the Istio service mesh in mutual authentication mode.

Certificate lifespan was reduced to several hours with automatic rotation without restarting services, and encryption was first enabled in observation mode — telemetry showed which flows were still in plaintext. Strict mode was enabled only after every path was confirmed to have no unencrypted traffic.

Reversibility was tested separately: returning the policy to observation mode opens the channel in minutes, so a configuration error does not leave services without connectivity.

The number of incidents related to access between components dropped to zero, and manual certificate operations stopped consuming engineering time.

What to do if certificates are issued manually?

Manual issuance is when a certificate is requested through the certificate authority console, and the renewal date is kept in a calendar and in one engineer's head.

While there are a dozen services, the scheme is tolerable; at a hundred nodes it produces a predictable class of failures: renewal was missed, the trust chain was not picked up, and the connection between services fails with a certificate validation error.

The error is visible not where the cause is — diagnostics drift toward the network and load balancer.

An already deployed scheme does not need to be rewritten entirely, and that is the main thing.

We start with an inventory: which certificates have been issued, where private keys are stored, which names are covered, which consumers validate the chain, and which instances have the nearest expiration date.

This map shows what goes into automation first and what has to remain in the manual loop with explicit reminders.

The transition to automation is done service by service and without downtime: new certificates are issued by an automated CA, old ones live out their term, and both trust chains are added to the stores on clients.

We change one service at a time; after replacement we check not the fact of a "green lock" but the full path — mutual authentication, the name in the certificate, validity period, and response to revocation.

Rollback remains trivial: the previous certificate and key are kept in the store until the end of the replacement window.

A managed process is when renewal is based on remaining lifespan, keys are rotated together with the certificate, and lifespans are shortened without an increase in manual operations.

Metrics that show the result: share of automatically renewed certificates, time from request to finished certificate, number of manual actions per month, and number of failures due to expiration — the latter drops to zero.

Common concerns: compatibility, failures, rollback

Before starting, the same concerns almost always come up: will encryption work on services that no one has touched for years, what happens during a failure, and can everything be rolled back. These are the points where the fate of the implementation is decided — we address each.

What happens if a certificate has expired or a node is unavailable?

We configure the internal environment on a fail-closed basis: a connection without successful verification is not allowed through in plaintext, and the incident goes to metrics and notification. We make exceptions selectively — for liveness probes. We automate certificate rotation so that expiration does not become a cause of failure.

Can changes be rolled back if something goes wrong?

Yes. Policies are described declaratively and versioned; rollback is returning to the previous description, not manual editing. The first stage runs in observation mode: traffic flows as before, violations are only logged, so we enable strict mode based on facts.

How can you tell which service is breaking the connection?

From proxy logs and handshake metrics, you can see who initiated the connection, at which step it was interrupted, and with which certificate chain. The failure is localized to a specific service, rather than by blindly trying configurations.

Let's discuss your protection scheme and come back with an assessment

For the assessment to rely on facts rather than assumptions, we need a snapshot of the current state of traffic exchange.

Without it, the traffic protection implementation stage runs into unaccounted connections: a service that everyone considered internal goes through a shared node, and its traffic remains in plaintext.

The following set is enough for the review:

• list of services and traffic flows — who communicates with whom, over which ports and protocols;
• where traffic is currently terminated (load balancer, proxy, gateway) and which nodes see it in plaintext;
• how certificates are issued and stored, who owns the internal certificate authority;
• requirements by environment: server-only verification or mutual client verification at each segment;
• which environments are included in the first stage and whether there is a test environment that replicates production;
• permissible work windows and the degradation limit at which traffic exchange cannot be stopped.

We will come back with an assessment and work plan: list of connections to migrate, order of enablement by environment, certificate lifespans with a rotation procedure, and a rollback scheme.

We introduce strict mode one flow at a time: first the test environment, then production, monitoring the share of successful handshakes and connections that failed verification.

We set up automation of certificate issuance and renewal before migrating production connections; otherwise manual rotation becomes a separate point of failure.

Describe the task as it is for you — a draft network diagram and a list of services are already enough to start.