Securing SSH Keys with HSMs

Secure Shell (SSH) is one of the most trusted protocols in modern infrastructure. It protects remote access, automates deployments, and secures communication between systems. Authentication in SSH environments is typically based on key pairs, where private keys grant direct, passwordless access.

In many organizations, these SSH keys are widely distributed across servers, user machines, and automation platforms, often without centralized visibility or governance. Over time, they become long-lived credentials that are rarely rotated, monitored, or tightly controlled.

This creates a critical risk. If a private key is exposed through a compromised host, backup, or code leak, it can be reused as a valid credential, granting an attacker the same level of access as the authorized user, with no reliable means of distinguishing between legitimate and unauthorized use.

This highlights a key limitation: despite its cryptographic strength, SSH is only as secure as the protection of its private keys. In most environments today, those keys are stored as ordinary files on disk, copied across servers, embedded in scripts, and unintentionally duplicated into backups and container images, often without centralized tracking.

This is where Hardware Security Modules (HSMs) come in. They generate and store private keys within secure, tamper-resistant hardware, such as FIPS 140-3 Level 3 validated HSMs. This approach prevents key exposure and eliminates many of the weaknesses that make traditional SSH key management an attractive target for attackers.

In this blog, we will examine why traditionally managed SSH keys create systemic security risks and how HSMs fundamentally change that by securing private keys within dedicated security hardware and enforcing strong controls.

Traditional Ways of Storing SSH Keys

While SSH itself is a secure protocol, the risks emerge from how SSH private keys are typically handled in real-world environments. To understand these risks, it is necessary to look beyond cryptography and examine how SSH keys are stored, distributed, and used operationally.

Most organizations have no idea how many SSH keys exist across their environment. Rather than being managed as centralized security assets, keys are distributed across systems in ways that prioritize convenience over control. They are often stored as files on disk, embedded in automation scripts, saved within configuration management platforms, and unintentionally included in backups, virtual machine images, and container snapshots, frequently without anyone realizing it. Once created, these keys often remain in use for years without formal rotation, centralized visibility, or consistent governance controls.

Moreover, organizations rarely maintain a complete inventory of where keys exist, who owns them, or what systems they can access, and this is where the risk begins to compound.

Why Are Traditionally Managed SSH Keys a Security Risk?

The storage patterns described above create a fragile security model. In each case, the private key ultimately lives in an environment that the operating system can access, and so can any sufficiently privileged attacker.

The risks outlined below are not edge cases or misconfigurations. They are the natural outcome of treating SSH keys as ordinary files and shared secrets rather than high-value cryptographic assets that require the same level of protection as other privileged credentials. This approach leads directly to issues such as:

• Private Keys Can Be Copied Without Detection

  When SSH private keys are stored as files on disk, or loaded into memory by SSH clients, agents, and automation processes during authentication , any user or process with sufficient privileges can copy them. Whether a key is stored as a file, embedded in a script, or extracted from a backup, there is no native mechanism to detect or prevent duplication.

  If an attacker compromises a server, workstation, or automation platform, extracting SSH keys is trivial and leaves no audit trail. Once copied, the key can be reused from anywhere, making it almost impossible to distinguish legitimate access from malicious activity. SSH authentication validates only possession of the key, not the identity or context of the system using it.

• Malware and Compromised Systems Expose Keys

  Malware targets SSH keys to gain persistent access. It can scan file systems for known SSH key locations, extract keys from configuration tools, or capture keys when they are loaded into memory.

  Although SSH private keys may be encrypted at rest with a passphrase, they must be decrypted and available to the SSH client or agent during authentication. At that point, the security of the key depends entirely on the integrity of the operating system and the processes handling it. If the host is compromised, software-based controls such as file permissions cannot reliably prevent key misuse or unauthorized signing operations.

• Key Sprawl Creates Hidden and Persistent Access

  SSH keys are often copied across servers, shared between users, or embedded in automation scripts. Over time, organizations can lose track of where keys exist and who has access to them. This situation is further complicated by the way SSH access is structured: a private key on a user’s machine corresponds to a public key entry in the authorized_keys file on every server it can reach. These two components are managed separately, with no built-in mechanism to keep them in sync automatically.

  As a result, even when a private key is deleted from a user’s machine, access may not be automatically revoked. The corresponding public key can remain in authorized_keys files across multiple servers, continuing to grant access to any entity that still possesses the private key, including attackers who may have copied it earlier.

  Over time, this leads to “shadow access” that persists even after employees leave or systems are decommissioned, creating persistent, unmanaged access paths that are difficult to detect and eliminate.

• Long-Lived Keys Magnify Risk

  SSH keys rarely expire and are often left in use for years. Without rotation or lifecycle management, a single stolen key can provide attackers with long-term access, enabling persistence and lateral movement across the environment.

  The risk compounds when those long-lived keys also rely on aging cryptographic algorithms. DSA-1024 keys are deprecated and cryptographically broken. RSA-1024 is considered weak and is no longer recommended. It is worth noting that not all RSA-based SSH keys are affected equally. OpenSSH 8.8 deprecated the ssh-rsa algorithm specifically because it relies on SHA-1, which is vulnerable to collision attacks.

  When a long-lived key also uses a weak algorithm, the organization faces not just the risk of theft, but also exposure to cryptographic attacks.

• Limited Auditing and Weak Attribution

  Shared SSH keys make auditing and accountability difficult. Authentication logs show that a key was used, but not necessarily who used it or why, complicating incident response, forensics, and compliance reporting.

All of these risks share a common root. SSH private keys live in places where a sufficiently privileged attacker can reach them. Eliminating this exposure requires a fundamentally different storage model, one in which private keys never exist as plaintext outside the secure hardware boundary. This is precisely the security gap that HSMs are designed to address.

What Is an HSM?

An HSM is a dedicated, tamper-resistant device designed to generate, store, and use cryptographic keys within a secure hardware boundary. Unlike software-based key storage, an HSM is purpose-built to protect private keys from extraction and to enforce strong controls over how those keys may be used.

HSMs apply strong physical and logical protection. Keys generated within an HSM are typically non-exportable, meaning the raw key material cannot be retrieved in plaintext form through standard interfaces. Cryptographic functions such as key generation, digital signing, and key agreement are executed inside the device, and only the results of those operations are returned to external systems.

This approach shifts trust away from the host operating system. Rather than relying on file permissions or memory protection, key security is enforced by the HSM itself through dedicated hardware and policy controls. As a result, attacks that rely on reading key files, scraping process memory, or abusing software access paths are significantly constrained. By keeping SSH private keys within a hardware boundary where they cannot be extracted or copied, HSMs significantly reduce the risk of SSH key theft.

How do HSMs Secure SSH Keys?

HSMs provide a tamper-resistant, hardware-enforced environment for generating, storing, and using SSH keys. The following points explain how HSMs secure SSH keys and address common risks associated with traditional SSH key management:

1. Private Keys Never Leave the HSM

   SSH keys generated inside an HSM remain non-exportable in plaintext. The private key cannot be copied, read, or extracted through standard interfaces by administrators, applications, or attackers. Where required, wrapped exports may be permitted under strict key custodian controls, but the key never exists in usable plaintext outside the hardware boundary.

   During authentication, the HSM performs signing operations internally. The client submits a request, and only the resulting signature is returned. The private key itself is never exposed to the operating system, application processes, or system memory.

   Even if an attacker gains root access to the host, the key cannot be extracted. This significantly reduces SSH key theft scenarios such as file exfiltration, backup leakage, and memory scraping.

   Alternatively, organizations can use the HSM as a key encryption store, where the SSH private key is encrypted by a key that never leaves the HSM and stored as an encrypted file on disk.

2. Centralized Key Control and Policy Enforcement

   HSM-backed SSH keys enable centralized enforcement of access policies that are very difficult to achieve with file-based key management. Access controls can define who is allowed to use a specific SSH key, for which systems, and under what conditions.

   Key usage can be restricted based on identity, role, time window, or originating system. Rather than relying solely on endpoint security, the HSM enforces policy at the point where the cryptographic operation occurs.

   This replaces unmanaged key files with enforceable, auditable security controls that significantly reduce misuse and overprivileged access.

3. Strong Hardware-Based Protection

   HSMs provide a secure environment isolated from the host operating system. Because the SSH private key never exists in plaintext outside the HSM boundary, common attack techniques such as malware-based scanning, credential dumping, and memory scraping cannot reach the key material.

   Even on a fully compromised host, an attacker cannot steal the key for reuse elsewhere. They can only attempt to use it through the HSM’s signing interface.

   The standard integration mechanism that makes this possible is PKCS#11, a widely supported cryptographic API that allows SSH clients to interact with HSM-resident keys without ever exposing the private key material. OpenSSH supports PKCS#11 natively; users can add an HSM-backed key to the SSH agent using ssh-add -s, or instruct the SSH client to use a PKCS#11 provider directly by specifying the library path with the -I flag. This enables hardware-enforced authentication within standard SSH workflows, keeping private keys secure on the device.

   One residual risk worth noting is SSH agent forwarding. When a user connects to a target server through a jump (intermediate) server, agent forwarding lets the jump host use the SSH agent running on the user’s local machine. This eliminates the need to store private keys on the jump host itself. However, if agent forwarding is enabled and the intermediate host is compromised, an attacker can access the forwarded agent socket and request cryptographic operations on the user’s behalf. While the private key itself never leaves the HSM, the attacker can still leverage the agent to authenticate to other systems.

   For this reason, agent forwarding should be disabled wherever possible in HSM-based deployments, and strong endpoint security controls remain essential alongside the use of hardware-backed keys.

4. Strong Audit Logging and Accountability

   HSMs provide detailed, tamper-resistant audit logs for cryptographic operations and administrative actions. HSM-integrated systems can record each cryptographic operation and correlate it with identity and policy decisions.

   Unlike traditional SSH logs that only indicate that a key was used, HSM-backed logging enables precise attribution and centralized visibility. This significantly strengthens incident response, forensic investigations, and compliance reporting by providing a clear record of how and when SSH keys were used.

5. Secure Key Rotation, Revocation, and Automation

   Centralized key control enables secure and predictable SSH key lifecycle management when integrated with a key management system. HSM-backed keys can be rotated by generating new key pairs within the HSM, disabled immediately to prevent further use, or revoked centrally at the cryptographic level.

   While the HSM enforces key protection and usage policies, lifecycle operations such as distributing new public keys and removing old ones across servers and systems are handled by integrated automation and orchestration layers. If a key is suspected to be compromised, access can be cut off instantly by disabling usage at the HSM level. This is especially critical in large environments with thousands of servers and automated workflows.

   For CI/CD pipelines and automation tools, HSM-backed SSH keys prevent reusable credentials from being exposed. Even if a pipeline or build system is compromised, attackers cannot extract SSH keys for use outside the authorized environment.

6. Compliance Readiness and Cryptographic Agility

   From a compliance standpoint, HSMs provide centralized control, enforceable access policies, and tamper-resistant audit logs. Every key usage and administrative action is recorded and attributable, making it easier to meet regulatory standards, including PCI DSS v4.0 Requirement 8.6, NIST SP 800-57 key management guidelines, and SOC 2 CC6.1 logical access controls.

   For organizations operating under stricter regulatory or government frameworks, FIPS 140-3 Level 3 is a federal cryptographic module validation standard that requires physical tamper resistance, identity-based authentication for module access, and zeroization of critical security parameters upon tamper detection, making it the appropriate baseline for environments handling sensitive or regulated workloads. Many compliance frameworks either explicitly require or strongly favor the use of FIPS-validated hardware for cryptographic key protection.

   Beyond compliance, HSMs support long-term cryptographic agility. As algorithms evolve and security requirements change, keys can be regenerated and managed securely without redesigning the entire access model, ensuring a future-proof and resilient SSH key infrastructure.

Together, these controls shift SSH key security from a model based on trust and file permissions to one based on hardware enforcement, policy, and auditability.

SSH with HSMs in Practice

In a practical HSM-backed SSH deployment, the workflow changes subtly behind the scenes to secure private keys without altering the user experience.

1. Key Generation: SSH private keys are generated directly inside the HSM. The private key never leaves the secure hardware boundary.
2. Public Key Distribution: Only the public key is distributed to target servers, just as in traditional SSH setups.
3. Internal Signing: During authentication, the SSH client requests the HSM to sign the server challenge. The private key remains inside the HSM at all times. The signature is sent to the server, which verifies it against the stored public key. If the verification succeeds, access is granted.
4. Enforced Policies: Access policies stored in the HSM determine which users, systems, or roles are allowed to use a given key.
5. Centralized Logging: Every cryptographic operation, including authentication attempts and administrative actions, is logged securely for auditing and compliance.

From the user’s perspective, SSH behaves just like a normal connection. They authenticate as usual, and all security enhancements, including key protection, policy enforcement, and audit logging, occur transparently within the HSM.

Benefits of Securing SSH Keys with HSMs

Moving SSH keys into HSMs transforms how they are protected, eliminating risks associated with software-based storage. This approach not only strengthens security but also simplifies management, auditing, and compliance.

The benefits of securing SSH keys with HSMs include:

• Significantly Reduced Attack Surface: SSH private keys are never stored on disk, in backups, or in system images, eliminating a major attack vector and preventing silent exfiltration. This removes one of the most common persistence mechanisms used by attackers.

• Protection Against Insider Threats: Even privileged users cannot extract or reuse SSH keys outside approved systems, reducing both accidental and malicious misuse.

• Improved Compliance and Governance: Centralized control, auditable key usage, and enforced policies help meet NIST SP 800-57, SOC 2 CC6.1, and PCI DSS v4.0 Requirements. HSMs validated under FIPS 140-3 Level 3 further strengthen this posture by providing formally verified cryptographic module security, giving organizations a hardware-backed foundation that satisfies the assurance requirements of the most stringent regulatory and government frameworks.

• Safer Automation and CI/CD Pipelines: SSH keys cannot leak from build systems, and access can be scoped to specific workflows, protecting automated processes.

• Long-Term Cryptographic Control: HSMs enable secure key generation, algorithm agility, and migration to stronger cryptography for long-term security.

By implementing HSM-backed SSH key management, organizations not only strengthen security but also establish a scalable, auditable, and future-ready foundation for privileged access.

Managing SSH Keys at Enterprise Scale with Key Management Systems

Understanding why HSMs matter for SSH key security is one thing. Implementing that model across a large, complex environment is where the real complexity begins.

At enterprise scale, the challenge goes far beyond securing individual keys. Organizations need to account for every SSH key already in use across thousands of servers and user machines, many created without formal oversight and never rotated. They need consistent policies applied across environments that were never built with centralized key management in mind, and they need ongoing visibility into how keys are being used without slowing down the teams that depend on them.

HSMs solve the cryptographic problem. They keep private keys inside a hardware boundary where they cannot be extracted or copied. But they do not tell you how many keys exist, who owns them, or whether any of them should have been revoked months ago. That is the operational problem, and this is where a Key Management System, or KMS, becomes essential.

A KMS is a centralized platform designed to manage the full lifecycle of cryptographic keys across an organization. In the context of SSH key security, a KMS sits above the HSM layer and handles everything the HSM cannot do on its own.

Key discovery is where most organizations realize the scale of the problem. A KMS scans servers, user machines, and automation systems to build a complete inventory of every SSH key in the environment, including who owns it, what systems it can access, and when it was last used. Without this visibility, rotation and revocation are guesswork.

Lifecycle orchestration automates what would otherwise be manual and error-prone. A KMS can rotate keys on a schedule, issue ephemeral session-bound keys that expire automatically, and revoke access instantly across thousands of servers when a key is suspected to be compromised. When rotation happens, the KMS ensures the old public key is removed from authorized_keys files across all servers at the same time, preventing the stale access that manual rotation consistently leaves behind.

Policy enforcement gives security teams control over how keys are used, not just where they are stored. A KMS can restrict which users or roles can request a signing operation, enforce time-based access windows, require approval workflows for sensitive key operations, and flag anomalous usage patterns in real time.

Audit and compliance reporting consolidates key usage logs from across the environment into a single, searchable record. Every key generation, rotation, revocation, and authentication event is captured and attributable, giving security teams the evidence chain they need for incident response and regulatory audits.

For organizations that also deploy HSMs, a KMS can extend its governance capabilities to HSM-resident keys, managing their lifecycle and visibility alongside software-based keys within the same platform. This gives organizations a unified view across their entire SSH key estate regardless of where individual keys are stored.

Rather than assembling and maintaining custom integrations between key stores, directory services, and deployment pipelines, organizations need a unified platform that brings full lifecycle management, policy enforcement, and audit capability together in one place.

How Can Encryption Consulting Help?

At Encryption Consulting, we understand the challenges enterprises face in managing SSH keys at scale. Our solution, SSH Secure, is built to deliver end-to-end key lifecycle security, centralized visibility, and HSM-backed protection, ensuring that organizations can manage keys confidently without added complexity.

Some of the key features of SSH Secure include:

1. Centralized Visibility and Ownership Mapping

   Through a combination of agent-based and agentless discovery, SSH Secure locates every SSH key across servers and user machines. All keys are stored in a unified inventory with ownership and usage details, eliminating orphaned keys and ensuring full accountability across the environment.

2. Automated Key Lifecycle Orchestration

   SSH Secure automates the complete key lifecycle, covering secure generation, policy-driven rotation, and revocation. Keys can be rotated or revoked on demand or in accordance with organizational policies. For sensitive operations, SSH Secure can issue ephemeral session-bound keys that expire automatically. This centralized lifecycle management enforces least-privilege access, reduces the risk of compromise, and ensures keys do not remain valid beyond their intended use.

3. HSM-Integrated Protection

   All private keys are generated and stored within HSMs. Keys are generated using strong cryptographic algorithms such as RSA-4096, ECDSA, and Ed25519, providing strong cryptographic protection, resistance against cryptanalytic attacks, and efficient performance.

4. Policy-Driven Control for Key Operations

   All key operations, such as generation, approval workflows, rotation, and revocation, are enforced through policy-based controls. This ensures consistency across the environment, reduces manual errors, and maintains organization-wide security standards. Policies can be adapted to fit regulatory requirements or customized to support internal governance models.

5. Continuous Monitoring, Auditing, and Compliance Readiness

   SSH Secure provides real-time monitoring of key activities with detailed event logging and built-in anomaly detection. Logs can be integrated with Splunk or Grafana Loki dashboards for advanced visualization, correlation, and alerting. Flexible audit capabilities include downloadable logs and detailed reports, giving security teams clear insights into key usage and overall posture. Centralized auditing with policy-based alerts enables proactive security management, rapid anomaly detection, and faster incident response.

Implementing HSM-backed SSH key management at enterprise scale involves more than choosing the right hardware. It requires discovery, lifecycle orchestration, policy enforcement, and ongoing visibility across a complex environment. At Encryption Consulting, we built SSH Secure to handle exactly that, delivering end-to-end key lifecycle security and HSM-backed protection without adding operational complexity.

Conclusion

SSH keys are a cornerstone of modern IT security, but when stored in software or scattered across systems, they create significant risks for organizations. HSMs address these risks by ensuring private keys never leave a secure, tamper-resistant boundary, making theft, misuse, and exfiltration significantly harder to achieve.

Treating SSH keys as high-value cryptographic assets rather than simple files gives organizations centralized control, compliance readiness, and resilient protection for one of their most critical authentication mechanisms. For organizations managing privileged access at scale, HSM-backed SSH key management is not a future consideration. It is an operational necessity today.

If you are unsure where to begin, whether that is discovering how many SSH keys your organization currently has, understanding your exposure, or evaluating HSM integration options, Encryption Consulting can help. Reach out to us at info@encryptionconsulting.com to start a conversation.