Best SSH Key Management Solution for Distributed Remote Engineering Teams Guide

Implement SSH agent forwarding for small teams as a starting point, use dedicated tools like Teleport or HashiCorp Vault for enterprise-scale teams needing audit trails and access controls, or combine OIDC authentication with cloud provider-native solutions for minimal friction. The key is reducing manual key rotation while maintaining visibility into who accesses production infrastructure.

The SSH Key Management Problem

Remote engineering teams typically face several key management challenges. Developers need access to production servers, staging environments, and various internal services. Each developer might have multiple keys for different purposes—a personal key, a work key, and keys for specific projects. When team members leave or roles change, revoking access quickly becomes critical.

The traditional approach of manually distributing and tracking SSH keys doesn’t scale. Without centralized management, you lose visibility into who has access to what, key rotation becomes infrequent, and compromised keys create security vulnerabilities that go undetected.

SSH Agent Forwarding and Key Chaining

For smaller teams or those starting with minimal infrastructure, SSH agent forwarding provides a straightforward starting point. This method allows developers to use their local SSH agent when connecting through intermediate servers.

# Add your key to the SSH agent
ssh-add -k ~/.ssh/id_ed25519

# Connect with agent forwarding
ssh -A user@jump-server.example.com

The -A flag enables agent forwarding, allowing the connection to use your local keys through the jump server. While convenient, this approach has limitations for larger teams. Agent forwarding requires trust in intermediate servers, and tracking which keys have access to which systems becomes difficult.

Implementing a Centralized SSH Key Directory

A more structured approach involves maintaining a centralized directory of authorized keys. This works particularly well for teams with their own infrastructure.

# Directory structure for centralized key management
# /opt/ssh-keys/
# ├── users/
# │   ├── alice.pub
# │   ├── bob.pub
# │   └── charlie.pub
# ├── servers/
# │   ├── production/
# │   │   ├── app-server-1.authorized_keys
# │   │   └── app-server-2.authorized_keys
# │   ├── staging/
# │   └── development/
# └── groups/
#     ├── devops.pub
#     └── backend-team.pub

This structure separates keys by user, server environment, and team. Administrators can manage access by adding or removing keys from specific files, and version control of this directory provides an audit trail.

Using Ansible for SSH Key Distribution

Ansible excels at managing SSH keys across multiple servers. This approach combines automation with infrastructure-as-code principles.

# ansible/playbooks/ssh-key-management.yml
---
- name: Manage SSH keys across servers
  hosts: all
  become: yes
  tasks:
    - name: Ensure SSH directory exists
      file:
        path: "/home/{{ ansible_user }}/.ssh"
        state: directory
        owner: "{{ ansible_user }}"
        mode: '0700'

    - name: Deploy authorized_keys file
      copy:
        src: "files/authorized_keys/{{ ansible_hostname }}/"
        dest: "/home/{{ ansible_user }}/.ssh/authorized_keys"
        owner: "{{ ansible_user }}"
        mode: '0600'

With this playbook, you maintain authorized_keys files in version control, and Ansible distributes them to servers. Adding a new developer involves adding their key to the appropriate file and running the playbook.

GitOps-Based SSH Key Management

For teams already using GitOps workflows, storing SSH key configurations alongside infrastructure code makes sense. This approach treats SSH key management as part of your codebase.

# Example structure in your infrastructure repository
# infrastructure/
# ├── ssh/
# │   ├── keys/
#   │   │   └── users/
#   │   │       └── developer-keys/
#   │   │           ├── alice_ed25519.pub
#   │   │           └── bob_ed25519.pub
#   │   ├── templates/
#   │   │   └── authorized_keys.j2
#   │   └── scripts/
#   │       ├── add_key.sh
#   │       ├── remove_key.sh
#   │       └── rotate_keys.sh

A key rotation script might look like:

#!/bin/bash
# scripts/rotate_keys.sh

set -euo pipefail

KEY_DIR="keys/users"
TARGET_SERVER="$1"
NEW_KEY_NAME="$2"

# Generate new key pair
ssh-keygen -t ed25519 -f "${KEY_DIR}/${NEW_KEY_NAME}" -N "" -C "${NEW_KEY_NAME}@$(hostname)"

# Add new public key to authorized_keys template
cat "${KEY_DIR}/${NEW_KEY_NAME}.pub" >> "templates/authorized_keys.j2"

# Notify team about new key
echo "New key ${NEW_KEY_NAME} generated and added to template"
echo "Run deployment to apply changes"

Short-Lived SSH Certificates

For high-security environments, SSH certificates provide superior access control compared to traditional public keys. Certificates eliminate the need for per-server key distribution and enable time-limited access.

# Generate CA key pair
ssh-keygen -t ed25519 -f ssh_ca -C "team-ca"

# Sign a user certificate (valid for 24 hours)
ssh-keygen -s ssh_ca -I "developer-alice" \
  -V "+24h" \
  -z "20240315" \
  id_alice.pub

# The signed certificate (id_alice-cert.pub) can now authenticate
# without being added to individual server authorized_keys files

Servers trust the CA key rather than individual user keys. When access needs revocation, you add the principal to a revocation list rather than removing keys from every server. This scales significantly better than traditional key management.

Configure servers to trust your CA:

# On each server, add to /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/trusted_ca.pub

Managed SSH Key Solutions

Several commercial and open-source tools provide full-featured SSH key management without building custom infrastructure.

Teleport offers zero-trust access with SSH certificate-based authentication. It integrates with identity providers and provides session recording.

Smallstep focuses on certificate-based SSH access with automated rotation and fine-grained access policies.

HashiCorp Vault can manage SSH keys and provide dynamic SSH credentials, useful for teams already using Vault for secrets management.

For most distributed remote engineering teams, starting with a structured file-based approach using Ansible or similar tools provides good balance of complexity and capability. As teams grow and security requirements increase, migrating to certificate-based solutions becomes worthwhile.

Practical Recommendations

Start with these steps regardless of which solution you choose:

Audit existing keys: Identify all current SSH keys and their access levels. Remove unused keys.
Establish a key policy: Define requirements for key types (Ed25519 preferred over RSA), key rotation frequency, and access review cadence.
Implement access groups: Organize access by team and environment rather than individual keys.
Automate provisioning: Every new developer should receive access through automation, not manual server configuration.
Plan for offboarding: Ensure clear processes for removing access when team members transition.

The right solution depends on your team size, infrastructure maturity, and security requirements. Small teams benefit from simple Ansible-based approaches, while larger organizations should invest in certificate-based systems or managed solutions that provide audit trails and automatic rotation.

Built by theluckystrike — More at zovo.one