SSH has become our universal (Unix) external access protocol

Drives me nuts that somewhere along the devops journey people decided that SSHing into a private server used for internal tools is an antiquated and outrageous thing to expect. People for some reason are actually excited about the prospect -- "we're gonna make it so you never have to SSH!". Little do they know that I like SSH. A lot more than I like clicking on the AWS console. And then somehow we're expected to debug the bastard using logz.io or similar -- unconscionable to me but maybe I'm old

Need to get a file from one box to another? Have SSH? `scp` is your friend: https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol

    scp remote.host:path.txt local
    scp local remote.host:path

Need a fast proxy to browse the internet securely and bypass restrictions without a VPN? SSH SOCKS proxy! Supported by most operating systems, but I tend to use it directly via Firefox so that it is isolated to one browser. This starts a socks proxy on the desired port:

    ssh -D $port $host

Firefox has network settings, simply choose SOCKS, 127.0.0.1 as the IP and the specified port as the port. Then you're off to the races.

Until we had proper VPN infrastructure to enter our VPC at AWS I would utilize this to view private vpc-only RMQ dashboards.

Love SSH!

An alternative I am seeing mentioned with some frequency is Tailscale, which doesn't need port 22 open to the internet, since it's using its own network's connectivity to facilitate your "tailscale SSH" connectivity. From what I read it's very similar to Amazon's SSM Agent.

The usefulness here is that you're closing off ports and reducing your exposure, the downside is that you need proprietary agents installed on the remote devices, and clients (tailscale itself or the SSM extension to AWS CLI) on proprietary networks doing the routing for you. Which might be perfectly fine for your use cases.

I've done even less reading but is Cloudflare's WARP client the same thing for their own network?

You can do all sorts of things with SSH. My favorite is SSHFS, which is couplings for SFTP that treat it like a proper filesystem, and it works on everything that uses SSH. Quicker to setup than a VPN and SMB, and about as secure (you could also theoretically use PAM to authenticate with LDAP or newer MFA protocols)

SSH port forwarding is one amazing aspect of this software. For one example, you can develop on a remote system by forwarding your local port 3000 to the remote 3000, using something like `ssh -nFL 3000:localhost:3000 user@remote`, all while going through SSH! It's an indispensable tool for modern development.

I wish that SSH would be disaggregated further. SSH has become the suite du jour for file transfer, remote access, and a handful of other things. Unfortunately, simultaneously, innovations have been made in transport protocols and elsewhere in the stack that we're unable to take advantage of.

SFTP is a great example of a protocol which has a discrete server (look! There's sftp-server on your computer. Nothing prevents it from running over TLS, or a web socket). I wish that this was the way the entire suite worked. I wish the multiplexing, and underlying shell implementation was transport agnostic (perhaps relying on SOCK_STREAM, or SOCK_SEQPACKET semantics), and the authentication, encryption, etc was its own thing.

I don't think that requiring a VPN to use SSH is good advice for big organizations, at least in terms of usability. My uni's (Leeds) comp sci department had this and it was extremely unpleasant to use. While it is "better" from a technical standpoint, I had peers who paid for private compute time instead of using the uni's free clusters. The reality is that even undergraduate comp sci students often don't know enough IT/sysadmin stuff to be able to figure out setting these things up and there's often a benefit to making the barrier to entry lower.

Something that few people remember is that if you have access to a filesystem through SSH, then you can have a remote Git repository with no configuration!

In the remote machine, you only need to create a bare repository:

git init --bare

And in your "client" machines you use it like any other remotes:

git remote add my_remote my_user@my_host:path_to_repo

It can be useful if for some reason you don't want to use GitHub/GitLab/Bitbucket/etc or as a glorified scp

I often find out that SSH is blocked in a lot of networks. This is frustrating, since I usually clone git repositories trough SSH (and I'm considering for this reason switching to HTTPS), since I find stupid having to use a VPN just to work with git.

If you really want to have something that gives you access to remote resources in any network the only solution is to use a VPN over TLS on the port 443, to make it impossible to distinguish it from any other normal HTTPS traffic. This is the reason why I run an OpenVPN server at my company, where normally I use Wireguard that is more performant (but it's blocked in a lot of networks).

At the end of the day port 443 with TLS traffic on it is the only thing that is guaranteed to not have been blocked (on port 80, 25, etc firewalls may check that you are effectively transferring HTTP traffic, they could not on 443 since the traffic is encrypted, tough a smart firewall can assume from traffic patterns that the connection is unlikely HTTPS, to this day I've jet to se a firewall this smart).

I think Amazon does a good job here.

Default SSH is with certificates, passwords not used. I like that. Hard to brute force a certificates.

In the old flat 10.X.X.X network amazon days - your new hosts were absolutely hammered when being brought up. There must have been folks on the amazon network itself just portscanning like crazy.

What about an IP whitelist managed on some other website (say in AWS). If you need remote access while you are travelling, you login to that website, which will add your current IP to the whitelist. The server refreshes its firewall with the new whitelist every 5 minutes. So within 5 minutes you get access.

That creates another layer of protection (authentication to the website). I would assume a linux firewall is very hard to bypass so almost as good as not exposing the server to the WAN. And doesn't have all the problems and complexity associated with VPNs, works on any device from anywhere.

One neat feature of OpenSSH server is the AuthorizedKeysCommand config, which lets you fetch (or generate!) a user's keys from anywhere, e.g. a curl response.

With this you can easily set up a centralized SSH keys system without the pitfalls of decentralized systems or running a CA. Have the user register their public key on your website in the typical fashion, and then write a simple secure endpoint and use AuthorizedKeysCommand to instantly integrate all your OpenSSH servers.

It also lets you implement more exotic authorization schemes with the full capabilities of your internal backend, which is often a million times more enjoyable than fighting through Linux PAM.

If you use SSSD and LDAP and don't like the idea of relying on a curl every login, you can also centrally-manage the keys in LDAP for a similar effects.

SSH is great but rapidly becomes less so once companies decide to put their servers behind some times cascaded SSH gateways, so you need to know which magic invocation of jump hosts to put in to have your connection go through anyway.

it has been for 20 years IIRC. its designed to be a general purpose way to do remote access and command exec, with auth/auth. like thats its whole ballgame. I think the confusion comes when people assumed it was merely a successor to telnet.

but yes this is why its so important for OpenSSL client and server sides to be as bulletproof as they can. its a giant worldwide SPOF and therefore a drool-inducing pinata for hackers.

Privilege Access Management platforms do unify external access protocol for SSH, k8s, databases etc.

For eg.

Cyberark

Checkpoint harmony

Teleport [https://goteleport.com]

Strongdm [https://www.strongdm.com/]

Adaptive [https://adaptive.dev/]

and there are many other tools like these.

Exposing SSH to the world really bothers me. Personally I firewall to my own IP when connecting to EC2 instances, it's a pain but I don't feel comfortable knowing there may be zero days out there, plus I don't want my CPU cycles wasted by script kiddies.

It seems like there should be a better solution - something like port-knocking but done properly.

If you have SSH access you can use it as door to setup a simple VPN https://github.com/sshuttle/sshuttle

If ssh was less useful it would have less of an attack surface. ..... at least this is my theory.

Hence why I rather like it. github uses ssh keys - surely developers can learn to set those up?

That's nice and all, but as TFA already mentions, ssh has a large attack surface. The most critical one though is that you usually grant people access to a shell, so if an account gets breached, you need to worry about local root exploits, which are actually pretty common.

Also, believe it or not, setting up key-based authentication is quite the challenge for a lot of people, especially if you demand encryption of the private key and setting up an agent. However, you cannot enforce private key encryption server-side, so you can't even guarantee some kind of 2FA is in place. Yes, ssh does nowadays support FIDO, but that's even more complicated for users...

OpenSSH has seen one hole recently (and the failed xz attempt) and somehow SSH is less safe than a VPN? How the VPN configured? What is the client OS people are using to connect to the VPN? What's the track record security wise of the various VPN offering?

The "SSH has a wide attack surface as seen by RCE ..." is a bit dishonest IMO. How s any VPN more secure?

If you want configure SSH to be pubkey only and hand over Yubikey to your users.

I wonder: in all the recent data leaks leaking billions of users data where attackers were inside company's networks (and not just on online facing servers), was it through SSH holes that these attack too place? Or are we talking about a corporate culture of Windows+VPN?

I've come down to the same conclusion and automating with fabric and paramiko