I'm not a server guy but I had to deploy a few web sites and web services recently and it has been an interesting journey so far, although sometimes it feels a bit scary: it's like stepping into the unknown. I have to ask myself many questions, such as: how do I know that my servers are secure? That's an important question but “security” is such a vague term that it makes it hard to grasp and answer this question confidently.
How to Secure a Server?
Security is a complex topic and an optimal solution usually depends on who or what we might need to secure our servers from. For instance, my hosting provider might access my servers and there is nothing I can do about that, aside from going fully self hosted which has its own drawbacks which I'm not ready to accept at the moment.
I picked a simple strategy which I believe should serve my goals pretty well. This strategy consists of 4 steps:
- keeping operating system up to date
- making sure that only the necessary ports are open (in my case,
- for each open port, keep a close look on who connects to it and log every interaction
- blocking clearly malicious attempts to use any of the open ports, if possible
SSH Login Attempts
I have a few Scaleway virtual servers and they have quite a limited disk space of 20 GB, which is more than enough for my needs but I've noticed that the free space started to decrease rapidly on a server running Ubuntu 18.04 with a default configuration. I've decided to check what takes all of that space and the most active abuser turned out to be
journalctl --since "24 hours ago" | wc -l
There were more than 75,000 log entries and most of them looked like this:
sshd: Failed password for root from x.x.x.x port 42788 ssh2 sshd: Received disconnect from x.x.x.x port 42788:11: [preauth] sshd: Disconnected from authenticating user root x.x.x.x port 42788 [preauth] sshd: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user=root sshd: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x user= sshd: Failed password for root from x.x.x.x port 19567 ssh2 sshd: Failed password for root from x.x.x.x port 19567 ssh2 sshd: Failed password for root from x.x.x.x port 19567 ssh2 sshd: Received disconnect from x.x.x.x port 19567:11: [preauth] sshd: Disconnected from authenticating user root x.x.x.x port 19567 [preauth]
Most of those logs are failed attempts to authenticate using a password. That was strange, since Scaleway didn't provide me with any passwords and I use public key auth which is more secure. I filed a ticket and asked them why their instances claim to accept passwords and they declined to answer this question twice but they said that I can turn off password auth if I wish to, which I did by changing the line
PasswordAuthentication yes to
PasswordAuthentication no in a file located at
You may need to restart sshd or reboot your server for changes to be applied.
Now, there should be less logs but it doesn't solve the problem: anyone can still try to log in as many times as possible which bloats the logs and I don't like that. Fortunately, there is a tool that can detect the most active abusers and update the system's firewall in order to deny them future connections. Strictly speaking, there are many tools, but Fail2ban is one of the most popular, reliable and easy to use.
Fail2ban is accessible from a default Ubuntu package list. I used Ubuntu 18.04 but I guess the setup process wouldn't be much different with any other version of Ubuntu.
apt update && apt install fail2ban
Fail2ban is much more than SSH “guard”, it can be used to parse many different types of logs in order to detect malicious patterns and to ban any bad actors. If you have HTTP pages that are protected with HTTP auth, you can consider using Fail2ban to put a cap on how many login attempts are allowed from a single IP address during a certain period of time. OK, back to our SSH now:
This file will override Fail2ban default configs so we can specify our preferences here. All Fail2ban modules are disabled by default so we have to explicitly enable an sshd module:
[DEFAULT] bantime = 1d findtime = 1d maxretry = 3 [sshd] enabled = true backend = systemd
This configuration file states that Fail2ban should give anyone 3 “chances” to offer valid login credentials and if they fail 3 times during 1 day, they will be banned for the next 24 hours. That config seemed to work for me but you can adjust it for your needs: it can be probably made a lot harsher.
Now, it's time to start
service fail2ban start
You can always check the status of
sshd module by using the following command:
fail2ban-client status sshd
Here is an example output:
Status for the jail: sshd |- Filter | |- Currently failed: 67 | |- Total failed: 542 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 6 |- Total banned: 76 `- Banned IP list: 184.108.40.206 220.127.116.11 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52
We can also watch Fail2ban in action in the real time:
tail -f /var/log/fail2ban.log
And finally, if you decide to change Fail2ban config later, don't forget to reload it in order for changes to take effect:
Changing SSH Port
sshd uses the port 22, which means that it doesn't even require scanning all of your ports in order to start those nasty login attempts. Having a non-standard SSH port can reduce that kind of spam tremendously.
First, let's generate a random port number:
shuf -i 1-65536 -n 5
That should give us 5 random numbers:
4961 54688 50904 18137 27714
Any of those numbers can be used so pick the one you like. Just make sure that the port with the same number is not in use by another program. Now, let's put it inside
The last thing we need to do is to restart the sshd service:
service sshd restart
That's all, now the amount of ssh auth spam should decrease by quite a bit. You can access your server remotely by specifying this new port as a value of the
ssh -p 4961 user@domain
It's relatively easy to weed out the bad actors from your SSH port by disabling the password login and banning anyone who tries to mess with your server using Fail2ban. One of the additional benefits of having an active banning tool is that it helps to keep your server logs lightweight and easy to read. Changing default SSH port doesn't make your server more secure in any significant way but it can reduce the amount of spam in your server's log by a lot.