I’m not a server guy but I had to deploy a few web sites and web services recently and it has been an interesting journey so far, although sometimes it feels a bit scary: it’s like stepping into the unknown. I have to ask myself many questions, such as: how do I know that my servers are secure? That’s an important question but “security” is such a vague term that it makes it hard to grasp and answer this question confidently.
How to Secure a Server?
Security is a complex topic and an optimal solution usually depends on who or what we might need to secure our servers from. For instance, my hosting provider might access my servers and there is nothing I can do about that, aside from going fully self hosted which has its own drawbacks which I’m not ready to accept at the moment.
I picked a simple strategy which I believe should serve my goals pretty well. This strategy consists of 4 steps:
- keeping operating system up to date
- making sure that only the necessary ports are open (in my case, ssh and http(s))
- for each open port, keep a close look on who connects to it and log every interaction
- blocking clearly malicious attempts to use any of the open ports, if possible
SSH Login Attempts
I have a few Scaleway virtual servers and they have quite a limited disk space of 20 GB, which is more than enough for my needs but I’ve noticed that the free space started to decrease rapidly on a server running Ubuntu 18.04 with a default configuration. I’ve decided to check what takes all of that space and the most active abuser turned out to be sshd.
journalctl --since "24 hours ago" | wc -l
There were more than 75,000 log entries and most of them looked like this:
Sep 22 09:51:17 server sshd: Failed password for root from 126.96.36.199 port 42788 ssh2 Sep 22 09:51:17 server sshd: Received disconnect from 188.8.131.52 port 42788:11: [preauth] Sep 22 09:51:17 server sshd: Disconnected from authenticating user root 184.108.40.206 port 42788 [preauth] Sep 22 09:51:17 server sshd: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=220.127.116.11 user=root Sep 22 09:51:30 server sshd: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=18.104.22.168 user= Sep 22 09:51:32 server sshd: Failed password for root from 22.214.171.124 port 19567 ssh2 Sep 22 09:51:33 server sshd: Failed password for root from 126.96.36.199 port 19567 ssh2 Sep 22 09:51:35 server sshd: Failed password for root from 188.8.131.52 port 19567 ssh2 Sep 22 09:51:35 server sshd: Received disconnect from 184.108.40.206 port 19567:11: [preauth] Sep 22 09:51:35 server sshd: Disconnected from authenticating user root 220.127.116.11 port 19567 [preauth]
Most of those logs are failed attempts to authenticate using a password. That was strange, since Scaleway didn’t provide me with any passwords and I use public key auth which is more secure. I filed a ticket and asked them why their instances claim to accept passwords and they declined to answer this question twice but they said that I can turn off password auth if I wish to, which I did by changing the line PasswordAuthentication yes to PasswordAuthentication no in a file located by a following path:
You may need to restart sshd or reboot your server for changes to be applied.
Now, there should be less logs but it doesn’t solve the problem: anyone can still try to log in as many times as possible which bloats the logs and I don’t like that. Fortunately, there is a tool that can detect the most active abusers and update the system’s firewall in order to deny them future connections. Strictly speaking, there are many tools, but Fail2ban is one of the most popular, reliable and easy to use.
Fail2ban is accessible from a default Ubuntu package list. I used Ubuntu 18.04 but I guess the setup process wouldn’t be much different with any other version of Ubuntu.
apt-get update && apt-get install fail2ban
Fail2ban is much more than SSH “guard”, it can be used to parse many different types of logs in order to detect malicious patterns and to ban any bad actors. If you have HTTP pages that are protected with HTTP auth, you can consider using Fail2ban to put a cap on how many login attempts are allowed from a single IP address during a certain period of time. OK, back to our SSH now:
This file will override Fail2ban default configs so we can specify our preferences here. All Fail2ban modules are disabled by default so we have to explicitly enable an sshd module:
[DEFAULT] bantime = 1d findtime = 1d maxretry = 3 [sshd] enabled = true backend = systemd
This configuration file states that Fail2ban should give anyone 3 “chances” to offer valid login credentials and if they fail 3 times during 1 day, they will be banned for the next 24 hours. That config seemed to work for me but you can adjust it for your needs: it can be probably made a lot harsher.
Now, it’s time to start Fail2ban service:
service fail2ban start
You can always check the status of sshd module by using the following command:
fail2ban-client status sshd
Here is an example output:
Status for the jail: sshd |- Filter | |- Currently failed: 67 | |- Total failed: 542 | `- Journal matches: _SYSTEMD_UNIT=sshd.service + _COMM=sshd `- Actions |- Currently banned: 6 |- Total banned: 76 `- Banned IP list: 18.104.22.168 22.214.171.124 126.96.36.199 188.8.131.52 184.108.40.206 220.127.116.11
We can also watch Fail2ban in action in the real time:
tail -f /var/log/fail2ban.log
And finally, if you decide to change Fail2ban config later, don’t forget to reload it in order for changes to take effect:
It’s relatively easy to weed out the bad actors from your SSH port by disabling the password login and banning anyone who tries to mess with your server using Fail2ban. One of the additional benefits of having an active banning tool is that it helps to keep your server logs lightweight and easy to read.