I’m not a server guy, but I had to deploy a few websites and web services recently, and it has been an interesting journey so far, although sometimes it’s a bit scary: it’s like stepping into the unknown. I have to ask myself many questions, such as: how do I know that my servers are secure? That’s an important question but security is such a vague term that it makes it hard to grasp and answer this question confidently.

How to Secure a Server?

Security is a complex topic and an optimal solution usually depends on who or what we might need to secure our servers from. For instance, my hosting provider might access my servers and there is nothing I can do about that, aside from going fully self-hosted which has its own set of drawbacks.

I picked a simple strategy which I believe should serve my goals pretty well. This strategy consists of four steps:

1. Keep operating system up to date.
2. Make sure that only the necessary ports are open. In my case, it’s ssh (22), http (80) and https (443).
3. For each open port, keep a close look on who connects to it and log every interaction.
4. Block clearly malicious attempts to use any of your open ports.

SSH Login Attempts

I have a few cheap virtual servers, and they have a limited disk space of 20 GB, which is more than enough for my current needs, but I’ve noticed that the free space started to decrease rapidly on a server running Ubuntu 18.04 with a default configuration. I’ve decided to check what takes all of that space and the most active abuser turned out to be sshd.

journalctl --since "24 hours ago" | wc -l

There were more than 75,000 log entries and most of them looked like this:

sshd[11616]: Failed password for root from x.x.x.x port 42788 ssh2
sshd[11616]: Received disconnect from x.x.x.x port 42788:11:  [preauth]
sshd[11616]: Disconnected from authenticating user root x.x.x.x port 42788 [preauth]
sshd[11616]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x  user=root
sshd[11620]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=x.x.x.x  user=
sshd[11620]: Failed password for root from x.x.x.x port 19567 ssh2
sshd[11620]: Failed password for root from x.x.x.x port 19567 ssh2
sshd[11620]: Failed password for root from x.x.x.x port 19567 ssh2
sshd[11620]: Received disconnect from x.x.x.x port 19567:11:  [preauth]
sshd[11620]: Disconnected from authenticating user root x.x.x.x port 19567 [preauth]

Most of those logs are failed attempts to authenticate using a password. That was strange, since Scaleway didn’t provide me with any passwords, and I use public key auth which is more secure. I filed a ticket and asked them why their instances claim to accept passwords, and they declined to answer this question twice, but they said that I can turn off password auth by myself, if I wish to, which I did by changing the line PasswordAuthentication yes to PasswordAuthentication no in a file located at /etc/ssh/sshd_config

PasswordAuthentication no

You may need to restart sshd or reboot your server for changes to be applied.

Now, there should be fewer logs, but it doesn’t solve the problem: anyone can still try to log in as many times as possible which bloats the logs, and I don’t like that. Fortunately, there is a tool that can detect the most active abusers and update the system’s firewall in order to deny them future connections. Strictly speaking, there are many tools, but fail2ban is one of the most popular, reliable and easy to use.

Installing Fail2ban

Fail2ban is accessible from a default Ubuntu package list. I used Ubuntu 18.04, but I guess the setup process wouldn’t be much different for Debian or any of its derivatives.

apt update && apt install fail2ban

Fail2ban is much more than SSH “guard”, it can be used to parse many types of logs in order to detect malicious patterns and ban any bad actors. If you have HTTP pages that are protected with HTTP auth, you can consider using Fail2ban to put a cap on how many login attempts are allowed from a single IP address during a certain period of time. OK, back to our SSH now:

nano /etc/fail2ban/jail.local

This file will override Fail2ban default configs, so we can specify our preferences here. All Fail2ban modules are disabled by default, so we have to explicitly enable sshd module:

[DEFAULT]
bantime = 1d  
findtime = 1d  
maxretry = 3

[sshd]
enabled = true
backend = systemd

This configuration file states that Fail2ban should give everyone three “chances” to offer valid login credentials and if they fail three times during a single day, they will be banned for the next 24 hours. That config seemed to work for me, but you can adjust it for your needs.

Now, it’s time to start fail2ban service:

service fail2ban start

You can always check the status of sshd module by using the following command:

fail2ban-client status sshd

Here is an example output:

Status for the jail: sshd
|- Filter
|  |- Currently failed:	67
|  |- Total failed:	542
|  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:	6
   |- Total banned:	76
   `- Banned IP list:	37.49.225.93 188.92.77.12 92.222.136.169 193.201.224.232 116.110.219.162 171.254.227.12

We can also watch Fail2ban in action in the real time:

tail -f /var/log/fail2ban.log

And finally, if you decide to change Fail2ban config later, don’t forget to reload it in order for changes to take effect:

fail2ban-client reload

Changing SSH Port

By default, sshd uses the port 22, which means that it doesn’t even require scanning all of your ports in order to start those nasty login attempts. Having a non-standard SSH port can reduce that kind of spam tremendously.

First, let’s generate a random port number:

shuf -i 1-65536 -n 5

That should give us 5 random numbers:

4961
54688
50904
18137
27714

Any of those numbers can be used so pick the one you like. Just make sure that the port with the same number is not in use by another program. Now, let’s put it inside /etc/ssh/sshd_config

Port 4961

The last thing we need to do is to restart the sshd service:

service sshd restart

That’s all, now the amount of ssh auth spam should decrease by quite a bit. You can access your server remotely by specifying this new port as a value of the -p argument:

ssh -p 4961 user@domain

Conclusion

It’s relatively easy to weed out the bad actors from your SSH port by disabling the password login and banning anyone who tries to mess with your server using Fail2ban. One of the additional benefits of having an active banning tool is that it helps to keep your server logs lightweight and easy to read. Changing default SSH port doesn’t make your server more secure in any significant way but it can reduce the amount of spam in your server’s log by a lot.