How Far Did It Reach? Blast-Radius Audits and Credential Rotation

Part 3 of a three-part series on investigating and recovering from a software supply-chain compromise. Parts 1 and 2 covered the dependency and the host. This part covers everything that the host could reach.

TL;DR

• A compromised dev machine matters to the extent it had access to other things. Enumerate that access concretely before you rotate anything.
• ssh-keygen -y -f <key> tells you in one second whether a stolen private key is usable without a passphrase. That single fact sets your urgency.
• A second, unfamiliar key fingerprint in the SSH logs is the most direct evidence this audit can produce.
• Rotate in one order only: add the new key, verify it from a second session, then remove the old one.
• A clean audit is evidence the key was not misused. It is not proof the key was never copied. Retire it anyway.

Start with real exposure, not theoretical exposure

Parts 1 and 2 covered the dependency and the machine that ran it. This part covers everything that machine could reach.

The most direct form of that access usually sits in plain sight. An SSH private key. A cloud CLI session. A browser full of saved logins. A set of long-lived API tokens. Enumerate concretely rather than reasoning about credentials in general:

• SSH keys. Which keys exist, and which hosts does each client config point at? The Host, HostName, and IdentityFile entries in ~/.ssh/config define your literal audit scope. Does each key have a passphrase?
• Cloud CLI sessions. Was aws, az, gcloud, or a similar tool authenticated and usable from this machine during the exposure window?
• Platform sessions. Code hosting, package registries, CI/CD. Anything with a browser session or a locally stored token.
• Plaintext config. .env files, CI variable dumps, config files with embedded credentials.

Is the key usable if stolen

ssh-keygen -y -f ~/.ssh/<key-name>

If this prints the public key with no passphrase prompt, the private key has no passphrase. A bare copy of that file is immediately usable by whoever holds it, with no second factor. This command only derives and prints the public key. It never displays or transmits private key material, so it is safe to run for this check.

This one fact sets your urgency more than anything else in the audit. A stolen passphrase-locked key is a very different risk from a stolen bare one.

Audit every server the key can reach

Do this for every host the key config covers. Auditing one host and assuming the rest are fine defeats the purpose.

Confirm the log you query is the right one, before you trust silence from it. Two traps catch people here.

systemctl list-units --type=service | grep -i ssh

The SSH daemon unit name differs by distribution. It is ssh on Debian and Ubuntu, and sshd on RHEL-family systems. Querying the wrong name returns an empty result that looks reassuring and means nothing.

journalctl --list-boots

If this shows only the current boot or two, journald probably does not have persistent history enabled. Anything before that window is gone from this source. Fall back to the on-disk logs, which usually keep longer retention:

sudo zgrep -h "sshd" /var/log/auth.log* 2>/dev/null | grep -E "Accepted|Failed" | tail -200

Look for a second key fingerprint. Every successful publickey login records a fingerprint. One known fingerprint across the whole window is real evidence that no other credential got in. A fingerprint you do not recognize is the most direct finding this audit can produce.

Enumerate every trust anchor on the box, not only your own account:

sudo find / -xdev -name authorized_keys -exec echo "--- {} ---" \; -exec cat {} \;

Check root and every other account. An attacker who got in once often adds a key elsewhere. Rotating the key you know about then locks out nobody.

Check for accounts and persistence you do not recognize:

awk -F: '$3>=1000{print}' /etc/passwd
sudo crontab -l ; crontab -l
sudo ss -tlnp
last -20

For login history, read the pattern across weeks rather than the most recent entry. Your logins normally cluster in a few ranges: your ISP pool, your office, your VPN. One login from an unrelated network breaks that pattern. The break is a far stronger signal than any single address.

Investigating an account you are unsure about

Do not delete on sight. Gather evidence first. The account may be legitimate, and if it is not, you want a record before it disappears.

sudo id <account>                                     # group memberships: sudo, wheel, docker?
sudo grep -r "<account>" /etc/sudoers /etc/sudoers.d/ 2>/dev/null
sudo stat /home/<account>                             # creation and modification timestamps
sudo lastlog -u <account>                             # has it ever logged in?
sudo cat /home/<account>/.bash_history 2>/dev/null

Weigh three things.

Group membership matters more than the account itself. An unused account in sudo or docker is root-equivalent. That is a real finding regardless of how it got there.

Shell history is strong evidence in either direction. Mundane, internally consistent activity argues for an old account you forgot. References to infrastructure you do not know argue the opposite.

Timestamps place the account against the exposure window. An account that predates the window reads very differently from one that does not.

If you decide to remove it, archive first. Once deleted, the evidence is gone:

sudo tar -czf /root/<account>-backup-$(date +%Y%m%d).tar.gz /home/<account> /var/mail/<account> 2>/dev/null
sudo userdel -r <account>

Rotate a key without locking yourself out

A clean audit is evidence that nobody misused the key. It is not proof that nobody copied it. Say the key sat unencrypted on a machine that ran untrusted code. Retiring it is the only step that removes the exposure. Every other step just checks for consequences.

One rule matters: add the new key, verify it, then remove the old one. Never the other order. Remove first on a box where that key is your only way in, and routine rotation turns into a lockout.

1. Generate a new keypair on the client. Never generate it on the remote server and copy it down. That recreates the same problem: private key material touching a filesystem it did not need to touch.

   ssh-keygen -t ed25519 -f ~/.ssh/<keyname>_new -C "<your-identity>"
   

2. Add the new public key to every server the old one worked on, while still connected with the old key:

   echo "<contents of <keyname>_new.pub>" >> ~/.ssh/authorized_keys
   

3. Open a second, separate session. Confirm the new key works on every host before you touch anything else:

   ssh -i ~/.ssh/<keyname>_new <user>@<host>
   

4. Once every host works, remove the old key line from each authorized_keys. Two valid keys defeat the point. The exposed key stays exactly as usable as before until its line is gone.
5. Clean up locally. Your SSH config may reference the key by its original filename. Rename the new files to that name and skip the config edit.

Two pieces of hardening belong in the same session.

Set a passphrase on the new key. Use an agent so you do not retype it constantly. This one choice decides whether a future exposure is annoying or catastrophic.

Disable password authentication. Set PasswordAuthentication no in sshd_config on any server that still allows it. That removes a whole class of brute-force risk unrelated to this incident.

Extend the audit to cloud and platform accounts

The same principle applies. Read what the platform logs say happened before you assume anything needs rotating. Most of this is a pure read, not a rotation.

Two of these deserve extra attention. A new app registration with a freshly added client secret is a classic low-visibility persistence method in cloud identity systems. CI/CD is a specific target, because a compromise there spreads further than the original machine.

Closing it out

Rotate what was genuinely exposed, even after a clean audit. Cheap insurance beats probably fine, and rotation is the step that removes the exposure rather than checking for its consequences.

Fix the secondary gaps you noticed along the way, even where they had nothing to do with this incident. Password authentication still enabled, unencrypted keys, stale privileged accounts. You are already doing the audit. Closing those gaps is nearly free.

Update the written record you started in Part 2. List what you completed and what you still owe. That is the difference between a closed loop and one that only looks closed.

The short version

Every phase of this series asks the same question. Not "does this one fix make the symptom go away." Instead: "what else should I check before I call this done?" Ask that consistently and you have the transferable skill. Any single command here will go stale as tooling changes. The question will not.

The series

1. Part 1: Finding a Compromised Dependency
2. Part 2: Did the Payload Actually Run?
3. How Far Did It Reach? (this post)