[$ ] doc: ssh-key-migration

Scope and assumptions

This is the procedure for the operator-side keypair rotation across a fleet of XMRHost VPSes. The trigger is one of: (a) the operator’s workstation was compromised and the in-use private key may have leaked; (b) the keypair is older than the brand’s rotation cadence (the brand defaults to annual); (c) the operator is migrating from a legacy RSA-2048 keypair to Ed25519. The procedure is identical in all three cases.

This is NOT a procedure for rotating the SERVER-SIDE host keys (those live in /etc/ssh/ssh_host_* and are rotated via the procedure in harden-sshd step 6). Server-side host-key rotation is a separate operation with a different failure mode.

Baseline assumptions:

• A fleet of 3-30 XMRHost VPSes, each with the operator’s existing public key in /root/.ssh/authorized_keys (or a non-root operator account’s ~/.ssh/authorized_keys, depending on the brand baseline you’re using).
• All boxes are reachable over SSH from the operator’s workstation now; if any are not, fix that first — you cannot rotate keys on a box you cannot reach.
• The workstation has GNU parallel installed (or you’re comfortable looping the procedure shell-by-shell).
• An out-of-band channel to the upstream’s serial / VNC console for at least one box, in case something unexpected happens during the migration.

The procedure deliberately treats the new and old keys as BOTH-active for the duration of the rollout. This is the safe ordering: every host accepts both the new and the old key simultaneously, the operator confirms the new key works on every host, and only then does the old key get removed. The unsafe ordering (“remove old, install new”) creates a window where a fat-finger locks you out of every box at once.

Step 1 — generate the new Ed25519 keypair on the workstation

On the workstation. Use ~/.ssh/id_ed25519_xmrhost_2026 (date-suffixed for clarity in the next rotation):

ssh-keygen -t ed25519 -a 100 \
  -C "operator@$(hostname)-2026" \
  -f ~/.ssh/id_ed25519_xmrhost_2026

The -a 100 parameter sets the bcrypt KDF rounds for the passphrase; resists GPU brute force. [RFC 8709]

Verify the file permissions:

-rw-------  1 you you  444 May 06 22:14 /home/you/.ssh/id_ed25519_xmrhost_2026
-rw-r--r--  1 you you  104 May 06 22:14 /home/you/.ssh/id_ed25519_xmrhost_2026.pub

Add the new key to the SSH agent — keep the old key in the agent too for the duration of the rollout:

ssh-add ~/.ssh/id_ed25519_xmrhost_2026
ssh-add -l   # confirm both keys are loaded

Step 2 — write the host inventory

Maintain a flat-file inventory of the fleet — one host per line, with operator-facing nickname + connect string. The brand convention: ~/.ssh/xmrhost-fleet.txt.

# ~/.ssh/xmrhost-fleet.txt — XMRHost operator fleet inventory.
# Format: <nickname>  <user>@<host>  [comment]

is01-relay      root@198.51.100.1     # Tor relay, Iceland
is01-onion      root@198.51.100.2     # Hidden services, Iceland
ro01-i2p        root@203.0.113.1      # I2P floodfill, Romania
ro01-matrix     root@203.0.113.2      # Synapse, Romania
ro02-bill       root@203.0.113.5      # Monero accounting, Romania
ch01-edge       root@192.0.2.1        # Edge, Switzerland

(The IP addresses above are RFC5737 documentation prefixes; replace with real IPs.)

If you have an Ansible / Terraform fleet inventory already, export hostlist into the same flat format. The procedure below loops over this file.

Step 3 — append the new key to each host (old key still active)

For each host, append the new public key to ~/.ssh/authorized_keys. The OLD key is still in the file; both work simultaneously.

NEW_PUB=$(cat ~/.ssh/id_ed25519_xmrhost_2026.pub)

# Loop over the inventory, skipping comment lines.
grep -v '^#' ~/.ssh/xmrhost-fleet.txt | grep -v '^$' | while read -r nick conn _; do
  echo "==> $nick ($conn)"
  ssh -o StrictHostKeyChecking=accept-new "$conn" \
    "echo '$NEW_PUB' >> ~/.ssh/authorized_keys && \
     chmod 0600 ~/.ssh/authorized_keys && \
     wc -l < ~/.ssh/authorized_keys"
done

Expected output per host:

==> is01-relay (root@198.51.100.1)
2
==> is01-onion (root@198.51.100.2)
2
==> ro01-i2p (root@203.0.113.1)
2
...

The 2 is the line count of authorized_keys (1 = old key only, 2 = old + new). Every host should print 2 if it had only the old key going in. Hosts that print 3 may already have a stray key — investigate before the deauthorisation in step 6.

If any host fails to connect, do NOT proceed. Fix that one host’s connectivity, retry, then continue with the rollout.

Step 4 — verify the new key works on each host

For each host, attempt a connection USING THE NEW KEY ONLY. The cleanest way is to disable the old key in the agent transiently and re-add after:

# Save which keys are loaded.
ssh-add -L > ~/.ssh/.agent-snapshot

# Remove the old key from the agent — keep ONLY the new key for this test.
ssh-add -d ~/.ssh/id_ed25519_xmrhost_OLD

# Loop and verify.
grep -v '^#' ~/.ssh/xmrhost-fleet.txt | grep -v '^$' | while read -r nick conn _; do
  result=$(ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519_xmrhost_2026 \
                "$conn" 'echo ok-from-$(hostname)' 2>&1)
  echo "[$nick] $result"
done

# Re-add the old key for safety.
ssh-add ~/.ssh/id_ed25519_xmrhost_OLD

Expected output:

[is01-relay] ok-from-is01-relay
[is01-onion] ok-from-is01-onion
[ro01-i2p] ok-from-ro01-i2p
...

Every host must print ok-from-<nickname>. Any host printing Permission denied did not actually receive the new key in step 3 — re-run step 3 against that host alone, then re-verify.

Step 5 — distribute the new key to ssh-agent on a SECOND workstation (optional but recommended)

If the operator has a backup workstation (a second laptop, a Tails-booted USB), copy the new private key there too. The threat model: if the primary workstation is destroyed (hardware failure, theft, fire) BETWEEN the deauthorisation of the old key and the next opportunity to generate yet another keypair, the entire fleet is locked out.

# On the primary workstation:
scp ~/.ssh/id_ed25519_xmrhost_2026* you@backup-workstation:~/.ssh/

# On the backup workstation, set permissions:
chmod 0600 ~/.ssh/id_ed25519_xmrhost_2026
chmod 0644 ~/.ssh/id_ed25519_xmrhost_2026.pub

The encrypted private key file is safe in transit over SCP — the bcrypt KDF passphrase from step 1 protects it on the backup machine. Keep the passphrase paper-only, separate physical storage from the key file.

If you’re using a hardware-backed SSH agent (YubiKey with PIV, or ssh-add -K on macOS), the backup is the hardware device itself; skip this step.

Step 6 — deauthorise the old key

Now, and only now, remove the old key from each host’s authorized_keys. The operation is grep -v against the old public key string:

OLD_PUB_LINE=$(cat ~/.ssh/id_ed25519_xmrhost_OLD.pub)

grep -v '^#' ~/.ssh/xmrhost-fleet.txt | grep -v '^$' | while read -r nick conn _; do
  echo "==> $nick ($conn) — removing old key"
  ssh "$conn" "
    grep -v -F '$OLD_PUB_LINE' ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.new && \
    chmod 0600 ~/.ssh/authorized_keys.new && \
    mv ~/.ssh/authorized_keys.new ~/.ssh/authorized_keys && \
    wc -l < ~/.ssh/authorized_keys
  "
done

Expected line count after the removal: 1 per host (only the new key remaining). If any host prints 0, the grep matched the entire file and the host now has NO authorized keys — restore from ~/.ssh/authorized_keys.bak (the SSH session is still alive; you have one chance to fix this before the session times out and the box is permanently locked out).

A safer pattern that backs up first:

grep -v '^#' ~/.ssh/xmrhost-fleet.txt | grep -v '^$' | while read -r nick conn _; do
  echo "==> $nick ($conn) — backup + remove old key"
  ssh "$conn" "
    cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.before-rotation-$(date +%F) && \
    grep -v -F '$OLD_PUB_LINE' ~/.ssh/authorized_keys > ~/.ssh/authorized_keys.new && \
    chmod 0600 ~/.ssh/authorized_keys.new && \
    mv ~/.ssh/authorized_keys.new ~/.ssh/authorized_keys && \
    wc -l < ~/.ssh/authorized_keys
  "
done

The ~/.ssh/authorized_keys.before-rotation-YYYY-MM-DD files can be cleaned up after a couple of weeks once the new key is well-established.

Step 7 — final audit: no host still trusts the old key

The final pass confirms that no host accepts the old key any more. Re-add the old key to the agent transiently, attempt a connection, and EXPECT a Permission denied:

ssh-add ~/.ssh/id_ed25519_xmrhost_OLD

grep -v '^#' ~/.ssh/xmrhost-fleet.txt | grep -v '^$' | while read -r nick conn _; do
  result=$(ssh -o IdentitiesOnly=yes -o BatchMode=yes \
                -i ~/.ssh/id_ed25519_xmrhost_OLD \
                "$conn" 'echo OOPS-OLD-KEY-STILL-WORKS' 2>&1)
  case "$result" in
    *"OOPS-OLD-KEY-STILL-WORKS"*)
      echo "[$nick] FAIL — old key still authorized"
      ;;
    *"Permission denied"*)
      echo "[$nick] OK — old key correctly rejected"
      ;;
    *)
      echo "[$nick] UNKNOWN — $result"
      ;;
  esac
done

ssh-add -d ~/.ssh/id_ed25519_xmrhost_OLD

Expected: every host prints OK — old key correctly rejected. Any host printing FAIL still has the old key in authorized_keys — re-run step 6 against that host.

Step 8 — destroy the old key file

Once every host correctly rejects the old key, destroy the old private key file:

shred -uvz ~/.ssh/id_ed25519_xmrhost_OLD
shred -uvz ~/.ssh/id_ed25519_xmrhost_OLD.pub  # public key too — no leftovers

Update the operator’s bookkeeping: log the rotation date, log the new key fingerprint, store both alongside the offline backup of the new keypair’s passphrase.

Optional — add the new key to authorized_keys via cloud-init for new boxes

If the operator provisions new boxes via cloud-init or via the upstream provider’s web UI, update the cloud-init template / panel default to include the NEW public key. Otherwise the next box you provision will start out with the old key in authorized_keys, defeating the rotation.

# cloud-init user-data for a XMRHost VPS — uses the new operator key.
#cloud-config
users:
  - name: root
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINEW2026... operator@workstation-2026

Closing — what to do next

The fleet is now on the new key. Order of next steps:

1. Set the next rotation reminder (calendar, ticketing system) for the brand’s annual cadence.
2. If the workstation that holds the new key has a configured SSH config (~/.ssh/config), update the per-host IdentityFile blocks to point at the new private key (otherwise OpenSSH will try every key in the agent on each connect — works, but produces noise in auth.log and counts against MaxAuthTries).
3. Run the audit script from step 7 quarterly as a continuous-verification check — it should pass cleanly until the next rotation.
4. Consider migrating to a hardware-backed key (YubiKey + ssh-keygen FIDO support, or a YubiKey with PIV via Resident Keys). The procedure is identical to the one above except the key generation in step 1 is ssh-keygen -t ed25519-sk instead of ssh-keygen -t ed25519. Hardware-backed keys eliminate the “private key file leaks from disk” failure mode entirely.

The single most important property of this procedure is that the rollback is trivial up until step 6: as long as the old key is still in authorized_keys somewhere, you have a path back. After step 6, the rollback requires the upstream’s console — which is exactly why steps 1-5 spend so much time confirming the new key works on every host before anything is removed.