If you’re a developer of any sort, you’ve probably used Secure Socket Shell (SSH) before. At its core, SSH is a protocol which lets users connect to computers running an SSH server. You can configure SSH to allow password based logins, but passwords can be brute-forced with enough time, resources, and ill will, so the industry standard is to use public/private key pair authentication to verify users’ access to systems. You simply generate an SSH key pair, drop your public key into the ~/.ssh/authorized_keys file on the target computer, configure its SSH server to accept public key based authentication requests, and then connect to it from anywhere in the world using your previously generated private key.
This sounds all well and good so far, but what happens when you want to access a computer which is located inside a private network? You could use a Virtual Private Network (VPN) to logically connect your local computer to the remote network; that would be a sensible plan. If, for some reason, a VPN was not available, you might be offered an SSH bastion host instead. An SSH bastion sits at the edge of an internal network and an external network, and is typically configured to allow authorized users to connect to it via SSH. This bastion host will typically have no services other than SSH running on it and it will have no ability for every user to connect to any computer on the private network it is connected to. So, what good is an SSH bastion, then?
Enter stage left: SSH Agent Forwarding. Forwarding your SSH agent allows you to connect to a single host, and setup a relay for future authorization requests so that you can connect to additional hosts (often called “hops”) beyond the first host in the chain via SSH. SSH agent forwarding ( the -A flag) is a powerful tool, which allows you to connect to any number of hops with the private key stored on your local computer without ever sharing your private key with any computer along the entire chain of connections. When used for legitimate development and administration work, SSH agent forwarding can be a real timesaver, but this magical agent forwarding system isn’t without its drawbacks.
If you configure an automation to do something which requires the use of SSH agent forwarding, you’re asking for trouble because this agent forwarding mechanism cannot differentiate between legitimate and illegitimate requests; it just replies to all of them the same way. The folks over at Skylight Cyber wrote an article detailing how SSH agent forwarding was responsible for a vulnerability allowing for root privilege escalation, affecting hundreds of thousands of computers. This particular vulnerability involved the server management software OnApp, which used to use SSH agent forwarding to relay commands from a single server to any other servers configured, which would allow for change management of the servers at an enterprise level. This software was configured to SSH into a machine via a forwarded SSH agent to perform certain tasks, which themselves would run executables on the target machine. The folks at Skylight Cyber realized they could replace one of those binaries with a small script which allowed them to SSH into any server within that provider’s cloud infrastructure as root and run arbitrary commands.
Hopefully this is a lesson learned for many of us about how dangerous SSH agent forwarding can be when applied to automations. If you have to use SSH connections for tasks, ensure that your source machine connects directly to the target machine, without the need for SSH agent forwarding to ever be enabled.
One last thing to note; this is the second week’s entry in a series of blog posts I’ll be doing, so stay tuned for more if you’re enjoying this content!
Stay safe out there!