Configuring SSM for secure server management without SSH

Secure server management via AWS Systems Manager

While there are only five servers, access to them rests on static SSH keys and a list of trusted addresses.

Then familiar degradation begins: keys are copied between engineers, settle in chats and backups, are not revoked after an employee leaves, and the list of allowed subnets grows to the point where no one remembers why a particular entry is there. Direct access to the server turns into an unmanaged channel.

The cost of such a scheme manifests during incident investigation.

Logs retain the fact of key login, but it is not visible who the key belongs to, where the connection came from, or what happened inside the session; revoking access is a manual walk through all machines, not a single action.

One shared key per team solves the speed issue, but completely nullifies audit of administrator actions.

AWS Systems Manager removes the root cause: the connection does not go to an open port but through a managed session, where permissions are granted by roles, revocation happens by removing a role, and every command is written to a log tied to a specific person. Server management without SSH stops being an exception and becomes the standard mode of operation.

Below we break down how this is closed: what permissions engineers need, what to do with the overgrown list of keys, and how to configure logging so that a trace remains for every action.

What changes after abandoning direct SSH connections

Abandoning direct SSH connections is a change in the access model, not a habit: the connection is initiated by an agent on the host, and permissions are checked on the AWS Systems Manager side. Here is exactly what changes and by what signs this is visible in the first week.

• Closed inbound ports. The rule for the administration port leaves the perimeter: external scanning no longer finds an open service, and key brute-forcing disappears from login attempt reports.
• Single entry point. Instead of a scattering of addresses and keys — one interface with roles: connection to any host follows one scheme, and a new machine does not need to be separately "opened outward."
• Full command log. Every session is recorded in full — who, when, and what they executed. Incident investigation takes minutes instead of reconstructing events from memory.
• Access rotation without stopping work. Sessions live for a limited time and are renewed transparently for the engineer, so a lost device stops being a problem: there are no permanent keys on it.
• No long-lived secrets on hosts. Keys and passwords do not spread across directories and environments — access is granted by role and revoked centrally.
• Reduced incident class. Entire categories disappear: password brute-forcing, a forgotten open port after debugging, "shadow" keys with contractors.

The measurable transition metric is the share of hosts without a publicly accessible administration port and the completeness of session recording.

Server access options: comparison of modes

Access modes differ not in convenience but in attack surface. The shorter-lived the account and the fewer ports exposed outward, the fewer compromise scenarios — that is why choosing a mode is primarily a matter of restrictions the team is willing to accept for support.

In practice, modes are combined: managed channel as the primary path, just-in-time access for routine work, permanent key only as an emergency entry inside an isolated environment.

Mode limitations are fixed before implementation, otherwise the emergency path quickly becomes the primary one.

Implementing Systems Manager: from audit to handover

We work so that not a single server falls out of management and access does not become broader than needed. First we record the current state, then change the access scheme, and only after checks disable direct SSH login. Below is the order of AWS Systems Manager implementation.

1. Server inventory. We collect a list of nodes with environments, tags, and owners. Separately mark what lives outside the common perimeter: that is usually where key-based login remains.
2. Roles and access policies. We describe who is allowed what: server role, engineer permissions, logs, commands. We tie permissions to groups, not people — when someone transfers, access is rebuilt automatically.
3. Deployment of the service component. We install the agent with templates and automation — new nodes register themselves. Network access to service addresses goes via private routes.
4. Access verification. We go through the checklist: does the session open, do commands work, are logs and metrics visible, are there bypass paths around the new scheme.
5. Fault tolerance testing. We look at what happens when the service is unavailable and permissions are revoked: the session closes, the engineer gets a clear error. Separately we test recovery of access to a node.
6. Handover to operations. We deliver procedures: how to grant access to a new engineer, how to connect a server, how to act on failure — plus role diagrams and control checks.

Each stage ends with a working scheme, so cutover happens without a downtime window — there is no fleet where half the nodes are already under control and half are on old keys.

Set of documents and configurations you receive

Access documentation is not an appendix to delivered infrastructure but a working tool: it is used to investigate an incident at three in the morning without waking the author of the latest changes.

Below is the set that remains with you after migrating a server fleet to management via AWS Systems Manager.

• Access scheme — who, from which networks and through which entry points reaches servers, which routes are fully closed, where the only allowed paths remain.
• Role matrix — mapping "team or role → server group → allowed actions." A new engineer gets the needed scope of permissions the same day, not after chat correspondence.
• Configurations and policies — files from which access is raised on new instances automatically, without manual completion and "forgotten" servers outside the perimeter.
• Response procedure — sequence of actions when communication with a node is lost, access is revoked from a departed employee, credential compromise is suspected, and the management channel itself fails.
• On-call shift instructions — a step-by-step scenario on one page: how to open access, how to terminate someone else's stuck session, whom to contact on failure, what to record in the log.
• Infrastructure documentation — server registry with tags, environments, and owners: it shows what falls under the policy and what remains aside.
• Audit log — where commands and exports are written, how long they are stored, how to export history for a specific node for the required period.

We keep the set in your repository next to the code and update it together with infrastructure changes — so that in six months the document describes the actual state, not the state on handover day.

Case study: moving away from direct connections on a server fleet

The customer's fleet is about 180 servers: some in the cloud, some on their own site. Each of 24 engineers had their own SSH key with indefinite access; copies of these keys lived on laptops, in build images, and in work chats.

A new person was granted access in one to two days: while the account is created, the key is distributed to all hosts, and none is forgotten.

Management was moved to AWS Systems Manager. An agent was installed on each instance, permissions were granted to the instance role, and the connection goes not directly but via a session according to access policy — with all commands recorded in the log.

Direct connections were closed at the network group level, and keys were removed from hosts and from the deployment configuration.

Metrics before and after

Permissions ceased to be indefinite: access is granted for a task, with a validity window and automatic revocation — disabling a person now means removing them from a group, and this applies to all hosts at once.

Planned credential rotation is no longer a separate week-long campaign but a regular background operation.

What to do if SSH keys have already spread across the team

The situation where permanent keys have already spread across the team cannot be closed in one evening.

Inherited keys lie in engineers' personal directories, in build node configurations, and on laptops of external contractors, so revoking them all at once means cutting off access to those who are currently maintaining production servers.

We start with inventory: which keys exist, where they are installed, who uses them, and what work depends on them.

Then comes a gradual transition, not flipping a switch. On servers we deploy the AWS Systems Manager agent and describe access by roles: the engineer opens a session via the managed channel, not by a file lying on disk, and permissions are granted for a task.

Old keys remain working during the transition period — otherwise it would be not a migration but an outage in the middle of the workday.

External contractors are a separate story. For them, access is granted for a term and tied to an account, not to someone else's public key: when work is done, revoking access is one action in permissions, not walking around a dozen machines looking for a copy.

Each session is recorded in the log with the name of who opened it and the command that was executed.

We close the transition period by a measurable sign: logs contain not a single login by an inherited key. Then we block direct access and purge old keys from trusted lists — but not earlier.

Until the login counter is zero, part of the team has a safety net, and the transition does not turn into a race with deadlines.

Technical concerns before abandoning direct access

Before abandoning direct access, engineers ask the same questions: what happens if the management service fails, how to update the service component on a fleet of machines, how to separate permissions between teams and contractors, and whether emergency access remains. Below are answers to each: they determine whether the scheme turns out workable or only on paper.

How is the service component updated on a fleet of machines?

In waves through target groups: first one machine, then a canary group, then the rest — with automatic stop when metrics deviate. The component version is fixed centrally; there is no need to go around servers manually.

How to separate teams and contractors?

We grant permissions by roles and resource tags: the application team sees only its servers, the contractor sees a specific environment and an allowed set of commands.

Multiple cloud accounts are brought together through single sign-on, so the access list lives in one place rather than in a spreadsheet for ten people.

Does emergency access remain?

Yes — a separate bastion node with short-lived keys and mandatory session recording. It is enabled according to procedure and closed after the incident: permanent keys in administrators' hands are no longer needed.

Let's discuss the task and come back with an estimate and plan

To come back with an estimate and plan, not general wording, we need a snapshot of the infrastructure. Then work proceeds in a fixed order — from solution design to support, and at each step it is clear what is required from you and what we do.

• Task analysis. We clarify what exactly we are closing: manual key issuance, shared access under one user, lack of an action log on servers, or requirements of security service checks.
• Infrastructure input data. List of servers and accounts in which they live, current access methods, who grants and revokes permissions, network restrictions, whether the AWS Systems Manager agent is already on the machines.
• Solution design and scope of work. We show how access will go through managed sessions, which roles and policies are needed, where logs are stored; separately we record what we close, what remains on your side, and in what form we hand over the result.
• Implementation plan. We break it into stages with checkpoints: pilot on a server group, migration of the rest, disabling previous access methods — with the possibility of rollback at any step.
• Handover procedure. We deliver configurations, roles, policy templates, and documentation, and show the on-call shift how to grant and revoke access, and how to view session history.
• Post-implementation support. We stay in touch on incidents and changes: new servers are connected using a ready template, and access rules are edited without rebuilding the scheme.

It is enough to describe the task and current access scheme — we will assemble the rest during discussion and come back with an estimate by stages.