I continued my experiments with Lightning Network. My focus was on the things that can go wrong if you decide to operate a Lightning node. It turned out, a lot of things can go wrong ranging from the typical ops problems such as hardware failures to the exotic and counterintuitive ones, such as Lightning backups.

Let’s start with the counterintuitive issues, as they’re more interesting. Every good sysadmin knows that having regular backups is essential to make sure your data is safe. Having a backup is always good, it protects you from losing your precious data. When it comes to managing Lightning nodes, having a full backup can be an extremely dangerous thing because it can lead to the loss of your funds. The thing is: Lightning Network prevents cheating by punishing people for presenting an outdated states of their channels.

The reason is pretty simple, let’s say you opened a channel with a local bar, and it started with $100 on your side and $0 on the side of the bar. At some time in the future, all the money ends up on the side of the bar, since you paid with Lightning Network. What makes Lightning Network so fast and cheap compared with Bitcoin is the fact that you can move money from one side of your channel to another without triggering a costly and slow Bitcoin transaction, but as some point, you’ll want to close the channel and both sides will be able to reclaim their bitcoins. Both sides know the latest state of the channel and if any side tries to cheat and broadcast a previous state ($100 on your side, for example), the other side can prove that it has a newer state and get all the money as a reward for detecting cheating. Well, the problem is: backups tend to have older states, so they can be seen as cheating, that’s why it isn’t a good idea to use full backups.

That’s crazy, right? How can we back up a Lightning node? There is a mechanism called Static Channel Backup. It’s not ideal, but it should work. The problem with this kind of backup is that it won’t recover your channels in case of a node failure. It can only recover your bitcoins by force-closing all of your channels.

There are currently no reliable ways to back up and restore Lightning channels. The best thing an operator can do is to minimize the probability of a node failure. What can fail if you’re running Raspiblitz on a Raspberry Pi 4?

• The board itself. Just buy more boards and you’ll be fine.
• An SD card. Again, redundancy is you friend. Just keep a spare SD card and you’ll be able to recover your node without using backups.
• An SSD/HDD. Failure of a main storage device would be a problem. Luckily for us, SSD/HDD failure rates are pretty low, and it’s unlikely to happen. If you’re paranoid, set up a RAID array with a comfortable level of redundancy. I think it’s an overkill unless your node is handling a lot of money.
• Electricity shortage. Unexpected shutdowns can corrupt your data, and you have to avoid such situations. Buying a good UPS should solve that problem.

As you can see, good old redundancy can solve most of your problems if you want to operate a highly available Lightning node. Backups are tricky because using wrong backups can lead to the loss of money and using proper backups leads to force-closing all of your channels. Losing your channels is unfortunate, but it’s a “worst case” that shouldn’t really happen. Even if you’re unlucky, the only thing you would lose is blockchain transaction fees.