Igor Bubelov

Core Lightning Post Mortem

Thu, 12 Feb 2026 00:00:00 +0000

The Old Setup

I set up BTC Map’s first Lightning node in late 2022, and I picked CLN, mostly because it was ahead of the curve on BOLT-12 adoption. Installing lightningd wasn’t easy, though. It only officially supports Ubuntu, and BTC Map’s fleet runs on Arch.

So let’s take a closer look at the CLN package.

tar --extract --file clightning-v24.08.1-Ubuntu-24.04.tar.xz --verbose

The CLN Shenanigans

What follows is a complete explosion of files. Dozens of binaries, plugins, Python scripts, documentation, and even .github folders with screenshots, shipped accidentally inside the package.

I’ve trimmed the output, but the picture is clear:

./
./usr/
./usr/bin/
./usr/bin/lightning-cli
./usr/bin/lightning-hsmtool
./usr/bin/lightningd
./usr/bin/reckless
./usr/libexec/
./usr/libexec/c-lightning/
./usr/libexec/c-lightning/lightning_channeld
./usr/libexec/c-lightning/lightning_closingd
./usr/libexec/c-lightning/lightning_connectd
./usr/libexec/c-lightning/lightning_dualopend
./usr/libexec/c-lightning/lightning_gossipd
./usr/libexec/c-lightning/lightning_hsmd
./usr/libexec/c-lightning/lightning_onchaind
./usr/libexec/c-lightning/lightning_openingd
./usr/libexec/c-lightning/lightning_websocketd
./usr/libexec/c-lightning/plugins/
./usr/libexec/c-lightning/plugins/autoclean
./usr/libexec/c-lightning/plugins/bcli
./usr/libexec/c-lightning/plugins/bookkeeper
./usr/libexec/c-lightning/plugins/chanbackup
./usr/libexec/c-lightning/plugins/cln-askrene
./usr/libexec/c-lightning/plugins/cln-grpc
./usr/libexec/c-lightning/plugins/cln-renepay
./usr/libexec/c-lightning/plugins/clnrest/
./usr/libexec/c-lightning/plugins/clnrest/.github/
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Postman-bkpr-plugin.png
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Postman-with-body.png
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Postman.png
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Swagger-auth.png
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Swagger-list-methods.png
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Swagger-rpc-method.png
./usr/libexec/c-lightning/plugins/clnrest/.github/screenshots/Swagger.png
./usr/libexec/c-lightning/plugins/clnrest/Makefile
./usr/libexec/c-lightning/plugins/clnrest/__init__.py
./usr/libexec/c-lightning/plugins/clnrest/clnrest
./usr/libexec/c-lightning/plugins/clnrest/clnrest.py
./usr/libexec/c-lightning/plugins/clnrest/poetry.lock
./usr/libexec/c-lightning/plugins/clnrest/pyproject.toml
./usr/libexec/c-lightning/plugins/clnrest/utilities/
./usr/libexec/c-lightning/plugins/clnrest/utilities/__init__.py
./usr/libexec/c-lightning/plugins/clnrest/utilities/generate_certs.py
./usr/libexec/c-lightning/plugins/clnrest/utilities/rpc_plugin.py
./usr/libexec/c-lightning/plugins/clnrest/utilities/rpc_routes.py
./usr/libexec/c-lightning/plugins/clnrest/utilities/shared.py
./usr/libexec/c-lightning/plugins/commando
./usr/libexec/c-lightning/plugins/funder
./usr/libexec/c-lightning/plugins/keysend
./usr/libexec/c-lightning/plugins/offers
./usr/libexec/c-lightning/plugins/pay
./usr/libexec/c-lightning/plugins/recklessrpc
./usr/libexec/c-lightning/plugins/recover
./usr/libexec/c-lightning/plugins/spenderp
./usr/libexec/c-lightning/plugins/sql
./usr/libexec/c-lightning/plugins/topology
./usr/libexec/c-lightning/plugins/txprepare
./usr/libexec/c-lightning/plugins/wss-proxy/
./usr/libexec/c-lightning/plugins/wss-proxy/Makefile
./usr/libexec/c-lightning/plugins/wss-proxy/poetry.lock
./usr/libexec/c-lightning/plugins/wss-proxy/pyproject.toml
./usr/libexec/c-lightning/plugins/wss-proxy/wss-proxy
./usr/libexec/c-lightning/plugins/wss-proxy/wss-proxy.py
./usr/libexec/c-lightning/plugins/wss-proxy/wss_proxy/
./usr/libexec/c-lightning/plugins/wss-proxy/wss_proxy/__init__.py
./usr/share/
./usr/share/doc/
./usr/share/doc/c-lightning/
./usr/share/doc/c-lightning/LICENSE
./usr/share/doc/c-lightning/README.md
./usr/share/man/... (OMITTED)

This is complete insanity. Usually, you just need a single binary, like bitcoind, and you’re done. But CLN takes an unorthodox approach: instead of one self-contained executable, you get dozens of moving parts glued together in weird ways.

Core Lightning is a maintenance nightmare.

The Dynamic Linking Problem

It gets worse. For reference, Bitcoin Core binaries work on virtually any Linux system because they’re statically linked. I expected the same from CLN, but no. CLN relies heavily on dynamic linking, which means it makes bold assumptions about the host environment.

My last attempt at updating CLN failed because the system had a newer version of libsodium or something like that, it adds completely unnecessary friction and a constant source of uncertainty.

Combine that with the convoluted packaging, and you get the least sane Lightning implementation I’ve ever dealt with.

LND Gets It Right

LND, on the other hand, is almost perfect. It follows the bitcoind playbook to the letter:

tar --extract --file lnd-linux-amd64-v0.20.0-beta.tar.gz --verbose

lnd-linux-amd64-v0.20.0-beta/lnd
lnd-linux-amd64-v0.20.0-beta/lncli

That’s it. Two binaries. No hacky plugins, no screenshots, no sprawling directory trees, no Python dependencies, no .github folders.

Check the dependencies:

ldd lnd

not a dynamic executable

Statically linked. Self-contained. Upgrade is a simple file swap. This is the only deployment pattern that makes sense for a decentralized network with a huge number of self-hosted nodes.

The Counterarguments

Of course, there are always Ubuntu maxis and the Docker crowd, ready to sell you their “solutions” while conveniently ignoring the baggage such as invasive abstraction layers and all the extra complexity.

What I Miss

The one thing LND still lacks is full BOLT-12 support, and that’s the only reason I originally chose CLN. In practice, though, BOLT-11 has stayed relevant far longer than I expected and it will keep being a reasonable default option for the years to come. BOLT-12 development is progressing slowly but steadily, and we’ll get there eventually.

Conclusion

If you’re running a Lightning node and value your sanity, pick LND. It’s the only implementation that aims to be usable.

Wireless Charging

Sun, 08 Feb 2026 00:00:00 +0000

The Accident

It all started with a deep clean. Buried in a tangle of old cables, I found a dusty wireless charger I’d completely forgotten about, a “Mi Wireless Charger”. I have no idea where it came from.

On a whim, I dropped my Google Pixel 6 onto it. To my surprise, the screen lit up: “Charging Wirelessly”. My power meter showed a solid 4W flowing into the phone, occasionally bumping up to 5W. Not exactly fast-charging, but for a forgotten gadget, it was above my expectations.

That got me curious. This old thing was using the Qi standard, something I’d known about for years but never really paid attention to. Had it gotten any better over time? And were there newer, faster options out there now?

The WPC

A quick dive into the standard revealed the orchestrator: the Wireless Power Consortium (WPC). This isn’t a shadowy cabal, but an open alliance of hundreds of companies, from Apple to Xiaomi, that agreed on one crucial thing: ending the cord war.

Before Qi, wireless charging was a mess of incompatible pads and proprietary tech. The WPC’s mission was to create a single, global standard. Their success is the only reason that old, forgotten pad could still charge my relatively modern phone. It’s a rare and beautiful case of industry cooperation that actually benefited users.

The WPC hasn’t been idle, though. The standard evolved, and today Qi2, the next generation with magnetic alignment, is finally taking off.

Magnets

The big improvement in Qi2 isn’t just raw speed, it’s alignment. And now I understand why my old pad was so flaky, bouncing between 4W and 5W.

The new Magnetic Power Profile (MPP) borrows the best idea from Apple’s MagSafe: built-in ring magnets. Suddenly, all the side effects of misalignment go away: the slight buzzing, the flickering light, the heat buildup as the charger tries and fails to lock on.

Magnets solve that. Perfect alignment means more efficient power transfer, so charging is not only faster, but also noticeably cooler.

A nice bonus: now that alignment is guaranteed, manufacturers are getting creative with angles. You can find magnetic chargers that hold your phone at 30° or 45°, turning your phone into a handy video screen while it powers up.

The Bridge to the Old World

The problem is, my Pixel 6 isn’t getting a Qi2 update cuz it’s a hardware limitation, not a software patch. The only way to upgrade is to buy a new phone, like a Pixel 10, and I’m not ready for that. I plan to keep using my Pixel 6 for another year, or at least until the battery gives out.

Even so, I think getting a Qi2 charger now is still a smart move. You can buy a phone case with a strong magnetic ring built in, which gives you most of the alignment benefits of Qi2 without needing a new phone. It’s a perfect bridge between the old standard and the new one.

Conclusion

Curiosity won. I ordered a Qi2 charger from Ugreen.

Qi was a great foundation, and Qi2 builds on it by finally fixing the frustrating user experience.

This all started with a dusty old Qi pad I’d completely forgotten about. That accidental find sent me down a rabbit hole that ended with a much better charging setup. It was a good reminder that sometimes the best tech upgrades aren’t about raw speed, but about a simple, satisfying snap that just makes everything work better.

Wi-Fi is the Best Protocol for Smart Lights

Mon, 19 Jan 2026 00:00:00 +0000

Once you’ve had lights you can actually control (brightness, color, scheduling and all the other stuff), there’s zero chance you’re going back to the dumb ones. The whole smart light scene is a total mess though, because it’s split between Bluetooth, Zigbee, Wi-Fi, Matter over Thread, and Matter over Wi-Fi.

I have a bunch of Wi-Fi lights (Philips WiZ) and a whole zoo of Zigbee lights, from premium Philips Hue all the way down to no-name AliExpress brands for $3. In practice, I like my Wi-Fi WiZ lights much more.

While meshes like Zigbee and Thread are pitched as the best choice because every new device supposedly strengthens your network, most homes don’t actually need a mesh. In reality, it just adds trouble. This isn’t just my take; it’s basically how the industry is evolving. Zigbee coordinators are quickly moving from tiny USB sticks to giant dildos, centralizing the network around a single powerful node.

Another serious flaw in Zigbee is letting lights act as router nodes instead of end nodes. When a router node goes offline, which happens anytime someone uses a physical light switch to cut its power, it disrupts the whole network. Zigbee networks do self-heal, but it takes time and makes the whole network slower and less predictable.

Finding a good alternative isn’t easy. Bluetooth has range issues, and “naked” Wi-Fi lacks a standardized pairing and device-control protocol, so you end up needing a special app for every vendor, which is a hard no.

While I don’t recommend “naked” Wi-Fi, a lot of the newer Wi-Fi lights do support a protocol called Matter. It acts as a common language between Wi-Fi devices and smart home hubs. Since they all speak it, you don’t need any vendor apps, and using physical switches won’t break your network.

That said, Zigbee and Thread mesh networks aren’t going anywhere; there’s no real alternative for battery-powered devices such as sensors and buttons that need to have extremely low power draw to be usable. But for everything that plugs into the wall, I prefer Matter over Wi-Fi.

$4 Zigbee Lightbulb Deep Dive

Wed, 01 Oct 2025 00:00:00 +0000

Intro

So, my first Zigbee bulbs were a bunch of Philips Hues. I snagged a pretty good deal on the bundle, they were still pricey even with the discount. I was expecting top-notch quality, but two of them kicked the bucket in under two years, which got me looking for other options.

The first (and only) alternative I’ve tried is this eWeLink model CK-BL702-AL-01(7009_Z102LG03-1). Yeah, the name sounds kinda scary, but they were only four bucks each on AliExpress. I ordered six of them to swap out the old Philips WiZ WiFi downlights in my kitchen.

The lights showed up surprisingly fast and they were all working perfectly. Pairing them with Home Assistant was a breeze, they just worked with the standard Zigbee 3.0 pairing, no funny business.

Now, let’s take a look at the Zigbee signature for these lights:

{
  "node_descriptor": {
    "logical_type": 1,
    "complex_descriptor_available": 0,
    "user_descriptor_available": 0,
    "reserved": 0,
    "aps_flags": 0,
    "frequency_band": 8,
    "mac_capability_flags": 142,
    "manufacturer_code": 4742,
    "maximum_buffer_size": 127,
    "maximum_incoming_transfer_size": 242,
    "server_mask": 11264,
    "maximum_outgoing_transfer_size": 242,
    "descriptor_capability_field": 0
  },
  "endpoints": {
    "1": {
      "profile_id": "0x0104",
      "device_type": "0x010d",
      "input_clusters": [
        "0x0000",
        "0x0003",
        "0x0004",
        "0x0005",
        "0x0006",
        "0x0008",
        "0x0300",
        "0x1000",
        "0xef00",
        "0xfc11",
        "0xfc57"
      ],
      "output_clusters": [
        "0x0019"
      ]
    }
  },
  "manufacturer": "eWeLink",
  "model": "CK-BL702-AL-01(7009_Z102LG03-1)",
  "class": "zigpy.device.Device"
}

Node Descriptor

Logical Type

Logical Type set to 1 means these lights are Zigbee Routers, which is great news. Routers help extend your network’s range and make it more reliable.

There is a debate about lightbulbs though, since those can be turned off with a physical switch, potentially disturbing the network. I would prefer downlights being leaf nodes.

Complex Descriptor Available

This device doesn’t have a complex descriptor, which is totally normal. Most devices skip this.

User Descriptor Available

This is different from the complex descriptor, it’s meant for storing a custom label on the device itself. It’s empty here, which is standard if you’ve never set one. I don’t think Home Assistant even lets you edit this field right now anyway.

HA’s own labeling system is way more flexible, since it’s not limited to a measly 16 characters like this Zigbee field.

Reserved

This is just an unused field saved for future features, so 0 is exactly what we’d expect.

APS Flags

The Application Support Sub-layer (APS) flags are all zeros, which is pretty common.

Frequency Band

Value 8 means it’s using the 2.4 GHz band, the same crowded-but-universal frequency everything else uses.

Medium Access Control Capability Flags

142 is 1000 1110 in binary (LSB 0111 0001):

Bit	Name	Value	Note
0	Alternate PAN coordinator	0	Can’t become a coordinator
1	Device type	1	Full function, can route
2	Power source	1	Mains-powered (not battery)
3	Receiver ON when idle	1	Always awake and listening
4	Reserved	0	Reserved
5	Reserved	0	Reserved
6	Security capability	0	Doesn’t handle security at this level
7	Allocate address	1	Can help joining new devices

This looks… interesting. The security capability bit being zero seems suspicious, but this is my first time digging this deep into a Zigbee descriptor. Maybe Zigbee 3.0 handles security at a different level and this flag is just legacy stuff?

Manufacturer Code

4742 appears to be Signify Netherlands B.V. (the company behind Philips Hue). This is pretty clever, these bulbs are basically pretending to be Hue bulbs to sneak into a Hue Bridge ecosystem.

Maximum Buffer Size

127 is the most common value and means it can handle the largest standard packet size.

Maximum Incoming Transfer Size

242 is weird here. To actually transfer chunks this big, the device would need to support packet fragmentation… but that capability isn’t advertised in the APS flags. Something’s not adding up.

Server Mask

All the flags set here are “reserved”, meaning the device is using them for some non-standard, manufacturer-specific services. Not unusual for Zigbee devices trying to add (or pretending to have) extra features.

Maximum Outgoing Transfer Size

Same issue as before, 242 suggests it should support large outgoing transfers, but the APS flags don’t back that up. Another inconsistency.

Descriptor Capability Field

0 is standard, just confirms there are no extra descriptors beyond the basics.

Endpoint 1 of 1

Profile ID

0x0104 is Zigbee Light Link (ZLL), the standard profile for smart lights.

Device Type

0x010D means “Extended Color Light.” This is exactly what we’d expect for a tunable white + RGB bulb, so Home Assistant should recognize it without any issues.

Input Clusters

ID	Note
`0x0000`	Basic: Mandatory info like version and manufacturer.
`0x0003`	Identify: Makes the light blink, useful for finding it in a group.
`0x0004`	Groups: Lets you control multiple lights together.
`0x0005`	Scenes: Can save and recall preset colors/brightness.
`0x0006`	On/Off: The basics.
`0x0008`	Level Control: Dimming.
`0x0300`	Color Control: Handles all the colors and white temperatures.
`0x1000`	Touchlink Commissioning: Lets you pair by holding it close to a bridge.
`0xef00`	Manufacturer Specific: Who knows? Custom feature.
`0xfc11`	Manufacturer Specific: Another custom thing.
`0xfc57`	Manufacturer Specific: Could be for custom firmware updates.

Output Clusters

ID	Note
`0x0019`	OTA Updates: Can receive firmware over-the-air.

Manufacturer

It’s showing as eWeLink, but it’s super easy to spoof this field when compiling firmware, so this doesn’t necessarily mean it’s officially from them.

Model

CK-BL702-AL-01(7009_Z102LG03-1), that’s a model number alright.

Class

zigpy.device.Device means Home Assistant is using a generic driver with no special quirks. It should just work without any fancy custom handling.

Conclusion

Take this with a grain of salt, since it’s the first Zigbee device I’ve really dug into. But I’ll be reviewing a lot more of these soon, I’m going all-in on Zigbee and want to really understand how it works under the hood.

Road to Zigbee

Sun, 28 Sep 2025 00:00:00 +0000

I’ve been using Home Assistant for a couple of years now. I got started, like most people, with some basic Philips lights. They have this cheap line called WiZ (like $15 a light), but you need a special app to set them up. I also tried their fancy Hue line (~$60 per light!) because I wanted to get into motion sensors and have lights that turn on automatically. The big difference is that WiZ uses WiFi, and Hue uses Zigbee.

At first, I was kinda pissed that WiZ didn’t have a motion sensor. But it turns out there’s a legit reason, and it’s not just the company being greedy. Motion sensors have to be super power-efficient, spending most of their time in a deep sleep to run for a year or more on a battery. WiFi is a total battery hog, so slapping it into a sensor would drain the thing in days. That’s the real reason you don’t see WiZ sensors.

Home Assistant totally saved my wallet by letting me mix and match, I could use a Hue Zigbee sensor to trigger my cheap WiZ WiFi lights. But once I got a Zigbee router, I started looking into the protocol and realized it’s way better than WiFi for smart home stuff, like in every way. The only thing holding me back from buying more Hue devices was their crazy price.

Luckily, I found out AliExpress is flooded with no-name Zigbee devices, and I’ve been buying them like crazy. The best part is that every new device makes my whole mesh network stronger and more reliable. You really can’t beat the prices from China at the moment, it’s a total game changer and there are no sane reasons to prefer WiFi.

Philips Hue

Sun, 28 May 2023 00:00:00 +0000

Matter is an open and privacy-friendly protocol which allows smart home devices to communicate with each other. I’ve been reading a lot of good things about it, and I finally decided to buy a few of those Matter-enabled things, namely a Hue Bridge and a bunch of Hue light bulbs.

Those toys aren’t cheap, so my expectations were high, and I must say, I’m really impressed with the quality of those light bulbs. Everything works like a charm, and I really like the flexibility of that system. I set my outdoor lights to turn on automatically fifteen minutes before sunset and then turn off right after midnight.

My bedroom doesn’t really need any timers, but I often change the light temperature and intensity based on what I’m doing. I like to start my evening with a bright cold light and then gradually move to a dimmed down warm light as I’m preparing for sleep.

Nostr

Sun, 14 May 2023 00:00:00 +0000

I’ve been playing with the Nostr protocol for a few hours, and here are my main takeaways:

It has much better fundamentals than Mastodon, and the people behind it are nicer and more competent.
It can scale, although it won’t be trivial. I don’t believe it would need to achieve massive scale in order to be useful, anyway.
It’s trivial to own and backup your data, you’re in no way dependent on any server.

Although the success of Nostr is not certain, I decided to create a simple Nostr to Atom (or RSS) bridge which will allow me to subscribe to Nostr accounts and see their posts in my News app feed.

Tailscale

Wed, 19 Oct 2022 00:00:00 +0000

I’ve been running WireGuard for years and it’s been solid. Lately, there’s a whole cottage industry of projects trying to slap a shiny GUI on WireGuard to make it “easy” for non-tech people. Tailscale is the one everyone won’t shut up about, so I decided to check it out.

Verdict: total letdown. It was more annoying to set up than plain WireGuard, and it forces you to use some third-party login. Maybe that makes sense in a corporate hellscape, but certainly not for my self-hosted stuff.

Beej's Guide to Network Programming (2019)

Tue, 26 Jul 2022 00:00:00 +0000

Although I don’t use C professionally, I enjoy reading low-level documentation because it sheds more light on how things work under the hood. Beej’s Guide to Network Programming is a great book focused on network sockets and the ways we can interact with them.

Things like select and epoll are the secret sauce that enables higher-level non-blocking IO abstractions such as Kotlin coroutines and async Rust. Anyone curious to see how the sausage gets made will find it worthwhile.

If you’re curious about the author, Brian “Beej” Jorgensen Hall has been writing software since the early 1990s and first published the guide around 1995. He now teaches computer science at Oregon State University-Cascades in Bend, Oregon.

CLBOSS

Sun, 08 May 2022 00:00:00 +0000

It wouldn’t be an exaggeration to say that the success of Bitcoin depends on Lightning Network. This is a level 2 solution which is intended to solve Bitcoin’s scalability issues by introducing quite a few clever concepts which every node operator should understand. It’s worth noting that all Lightning Network implementations are work in progress, but there are plenty of people who’re using them in production without any issues.

Managing Lightning node is rather hard and time-consuming, and there is a huge demand for automation. CLBOSS is one of the most interesting developments in that space. This plugin can take control of your Core Lightning node and manage it automatically. It can open and close your channels, and it can also make sure that your inbound and outbound liquidity is well-balanced, which is pretty important for routing nodes. It can also watch for network transaction fees, and it prefers to operate when fees are low. Watching how CLBOSS operates can also be a good learning experience if you want to learn the best practices of Lightning node management.

Mastodon

Wed, 20 Apr 2022 00:00:00 +0000

The Mastodon logo

I was a heavy Twitter user once, but those times are long gone. It’s not that I don’t enjoy Twitter anymore; I just find their business model to be pretty nasty. I had a few Twitter pages in my bookmarks, mostly some people from the Bitcoin community. Apparently, this community decided to migrate to Mastodon, and it was such a relief for me. Mastodon allows me to subscribe to certain blogs via RSS. Twitter used to be like this, until they decided that sharing content doesn’t suit their financial interests.

I even created my own Mastodon blog, and it turned out to be pretty helpful. Some folks from the Bitcoin community helped me set up a Bitcoin and Lightning node. Mastodon feels like microblogging done right. Highly recommend.

Mastodon devs finally released an official Android client a few days ago, which could make Mastodon more appealing to a wider audience. Truth be told, Mastodon is much less addictive than Twitter for various reasons, so it’s going to be really hard for new users to stick with it. It’s not like we have any open alternatives though, so it might be worth the effort.

Miniflux

Wed, 29 Sep 2021 00:00:00 +0000

I use webfeeds to fetch news and follow niche blogs. Most websites that are worthy of your attention provide webfeeds, and you can easily add them to your feed reader.

Feed readers allow you to merge all of your content sources into a single coherent timeline. By the way, that’s exactly what Vesti does. In many cases, having a simple standalone feed reader app isn’t enough because some people use multiple devices and want to keep them in sync.

Let’s say I’ve already read some news in an Android app and now want to finish reading today’s news on my laptop. My Android app should let my laptop know which articles I’ve already read so that they aren’t shown as unread. These problems can be solved by setting up your own webfeed server and connecting all your apps to this common source of truth.

Initially, my app supported only one kind of server: Nextcloud News. It works reasonably well, but it’s pretty slow and hard to set up. I recently learned about a server called Miniflux, and it turned out to be the best feed server I’ve ever seen. It’s ridiculously simple to set up and blazing fast, so I decided to support Miniflux in the next Vesti release.

USB Everywhere

Wed, 15 Sep 2021 00:00:00 +0000

Two good USB-related events this month:

EU plans to force all device manufacturers to adopt USB-C.
USB accessories will soon have a special markings, making it easy to figure out their capabilities.

It’s sad that Apple provoked more regulations, other market participants seem to converge on USB-C voluntarily. Anyway, this regulation is good for iPhone users and for the environment. I think it’s mostly a failure of antitrust regulations though.

Tor Switches to Rust

Sat, 24 Jul 2021 00:00:00 +0000

Writing software is hard, mostly because the expectations are rather high. We want our software to be cheap, safe, fast, reliable, easy to use, and that’s just a few things to start with. There is also a reason why we have so many tools and languages: different software projects have different sets of priorities.

What do we want from a project like Tor? Well, we want everything, but, if we have to prioritize, safety comes on the top of the list. That’s why I’m very excited to hear that Tor developers have decided to embrace Rust. Here is what they’re saying:

For us, these problems mean that programming in C is a slow and painstaking process. Everything we write takes more code than we’d like it to, and we need to double-check even the safest-looking code to make sure it doesn’t fall prey to any of C’s list of enormous gotchas. This slows us down seriously, and increases the cost of adding new features.

Rust seems like the clearest way out of our bind. It’s a high-level language, and significantly more expressive than C. What’s more, it’s got some really innovative features that let the language enforce certain safety properties at compile-time. To a first approximation, if the code compiles, and it isn’t explicitly marked as “unsafe”, then large categories of bugs are supposed to be impossible.

That’s a huge win for us in programming and debugging time, and a huge win for users in security and reliability.

Setting Up SSH Passwordless Auth

Wed, 23 Jun 2021 00:00:00 +0000

There are many ways to access remote servers, but the most popular is Secure Shell Protocol (SSH). All popular operating systems ship with an SSH client and server packages installed by default, which means you can connect to other computers as well as to allow them to use your machine. That’s cool, but also a bit concerning: how can we make sure only the right people can access our computers?

There are many ways to authorize via SSH, but the most popular are password auth and key-based auth. Among those two methods, key-based auth is preferable because it’s much harder to guess an average cryptographic key than the average password. You can use SSH to generate so-called public-private key pair. After that, you can copy your public key to a machine you want to access remotely and place it in authorized_keys directory. After that, you can go ahead and connect to that computer without typing any passwords. This method is both convenient and secure.

Let’s say we generated a key pair already, how do we copy our public key to a remote host? Looks like a chicken and egg problem. Many VPS providers solve this problem by asking for your public key before creating your first server, so they can add it to authorized_keys every time they bring up a fresh server for you. Another way to set up key-based auth is to start with password auth and then use a program called ssh-copy-id. It will attempt normal password auth and, once you type your password, it will add your public key to authorized_keys directory on the remote host. In other words, this program “pushes” your public key to a remote server. The downside? You have to enable less secure password auth first and then manually turn it off.

I’ve been happily using ssh-copy-id for years, but this mouth, I’ve found out that it’s also possible to “pull” my public key without setting up password auth. It’s especially useful when I have physical access to a new machine, and I want to be able to connect to it remotely from my main computer at a latter stage. How is it possible to pull a public key from a machine which might be out of reach? Even if it’s reachable, it won’t give any files to an unrecognized client. We have a chicken and egg problem again. Well, pretty much everyone uses GitHub, and it will happily give you public keys of any user. Public keys aren’t secret, so there is no reason to hide them anyway. The tool that can pull public keys of any GitHub user is called ssh-import-id and it’s installed by default on most Linux distributions.

Wi-Fi 6 Router

Sat, 17 Apr 2021 00:00:00 +0000

First, let me say that Wi-Fi 6 is not a revolutionary change in wireless communications. Yes, it’s a bit faster, and it has certain features aimed to make Wi-Fi more efficient in crowded places, but overall, I didn’t notice any difference in my home setup. I just saw a banner on my Internet provider’s website, and I decided to ask my provider about the possibility of upgrading the router. It uses GPON, so I can’t really go shopping and pick any router I like, unfortunately. It turned out, they offer free router upgrades, so I decided to use this opportunity. The router hardware is faster, which allows it to keep latency low under load, and their new firmware finally got advanced enough to allow custom MAC to IP mappings. I really missed that feature. Those improvements have nothing to do with Wi-Fi 6, by the way.

State of the Lightning Network

Wed, 24 Feb 2021 00:00:00 +0000

I continued my experiments with Lightning Network. My focus was on the things that can go wrong if you decide to operate a Lightning node. It turned out, a lot of things can go wrong ranging from the typical ops problems such as hardware failures to the exotic and counterintuitive ones, such as Lightning backups.

Let’s start with the counterintuitive issues, as they’re more interesting. Every good sysadmin knows that having regular backups is essential to make sure your data is safe. Having a backup is always good, it protects you from losing your precious data. When it comes to managing Lightning nodes, having a full backup can be an extremely dangerous thing because it can lead to the loss of your funds. The thing is: Lightning Network prevents cheating by punishing people for presenting an outdated states of their channels.

The reason is pretty simple, let’s say you opened a channel with a local bar, and it started with $100 on your side and $0 on the side of the bar. At some time in the future, all the money ends up on the side of the bar, since you paid with Lightning Network. What makes Lightning Network so fast and cheap compared with Bitcoin is the fact that you can move money from one side of your channel to another without triggering a costly and slow Bitcoin transaction, but as some point, you’ll want to close the channel and both sides will be able to reclaim their bitcoins. Both sides know the latest state of the channel and if any side tries to cheat and broadcast a previous state ($100 on your side, for example), the other side can prove that it has a newer state and get all the money as a reward for detecting cheating. Well, the problem is: backups tend to have older states, so they can be seen as cheating, that’s why it isn’t a good idea to use full backups.

That’s crazy, right? How can we back up a Lightning node? There is a mechanism called Static Channel Backup. It’s not ideal, but it should work. The problem with this kind of backup is that it won’t recover your channels in case of a node failure. It can only recover your bitcoins by force-closing all of your channels.

There are currently no reliable ways to back up and restore Lightning channels. The best thing an operator can do is to minimize the probability of a node failure. What can fail if you’re running Raspiblitz on a Raspberry Pi 4?

The board itself. Just buy more boards and you’ll be fine.
An SD card. Again, redundancy is you friend. Just keep a spare SD card and you’ll be able to recover your node without using backups.
An SSD/HDD. Failure of a main storage device would be a problem. Luckily for us, SSD/HDD failure rates are pretty low, and it’s unlikely to happen. If you’re paranoid, set up a RAID array with a comfortable level of redundancy. I think it’s an overkill unless your node is handling a lot of money.
Electricity shortage. Unexpected shutdowns can corrupt your data, and you have to avoid such situations. Buying a good UPS should solve that problem.

As you can see, good old redundancy can solve most of your problems if you want to operate a highly available Lightning node. Backups are tricky because using wrong backups can lead to the loss of money and using proper backups leads to force-closing all of your channels. Losing your channels is unfortunate, but it’s a “worst case” that shouldn’t really happen. Even if you’re unlucky, the only thing you would lose is blockchain transaction fees.

Webfeeds

Mon, 21 Dec 2020 00:00:00 +0000

I stopped using social networks a couple of years ago, and I never looked back. Social networks are full of bullshit and self-promotion and this kind of content doesn’t really resonate with me. I get it, some people want to know what’s up with their closest friends, but the thing is, most of the people we get constant updates from via the social media are not our close friends. Using a social network feels like being a guinea big in some kind of social experiment in human deception.

I get most of the news from RSS and Atom feeds. It’s so much better this way. The closest thing to this experience is Reddit. I was addicted to Reddit for quite a while, and it felt much better than other mainstream social media, probably because it was built around certain topics and not people, so it was much easier to find something interesting.

The World Needs Another Feed Reader

Wed, 30 Sep 2020 00:00:00 +0000

Okay, maybe it’s just me. Anyway, here is the source code:

https://github.com/bubelov/news

I’ve been working on this thing for the last few weeks, and it’s still a bit rough around the edges, but I already use it, and it serves me well. I tried to post it to Google Play, but it seems like Google is trying to force the developers to use Chrome instead of Firefox, again:

It only works with Nextcloud, but I’m planning to add a standalone mode in the coming weeks.

WireGuard

Sat, 30 May 2020 00:00:00 +0000

I have a Nextcloud server in my house, and it works great inside my local network, but it also needs to be reachable from the Internet and that’s where it gets a bit tricky. My server has a local IP, but it doesn’t have a globally reachable IP, and there are no easy ways to get one from a regular residential ISP.

The most convenient way to reach a website is by its domain name. Any browser can execute a DNS query to figure out the actual IP addresses behind a particular domain name you’re trying to access, and once that address is found, your browser of choice can finally initiate a connection.

So, my Nextcloud server needs the following things to be globally reachable:

A domain.
A unique IP address to point that domain to.

I already had a domain and the only thing missing was a unique IP address to put inside an A record. It turned out that Digital Ocean has a data center close enough (ping < 30 ms) to my house, and I decided to rent a cheap virtual machine there and use its globally reachable IP address for my local Nextcloud server.

It’s fair to stop there and ask yourself “why didn’t this guy just host his stupid Nextcloud on that droplet directly? What’s the point of hosting stuff from home?”. Those are good questions and the answer is: privacy and cost. I don’t want to hold my private data anywhere except in my house, and I also don’t want to pay a lot of money for storage space that I already have and can use for free. Even buying a new HDD or an SSD is much, much cheaper than renting the equivalent space in the cloud.

So, my domain’s A record can point all the traffic to that DO droplet, which has a unique and globally reachable IP address, so the only extra thing I needed to do was to connect my home server with that VM somehow. I had heard that setting up OpenVPN is not particularly easy, and I wanted to find an easier solution. A few web searches later, I ended up reading about a tool called autossh.

This excellent post helped me get it up and running in no time but, unfortunately, autossh is kind of cryptic and sometimes it just stops working without logging any reason why. That’s why I had to get rid of autossh and find a way to set up a proper VPN.

It turned out that setting up a VPN can be pretty easy, thanks to WireGuard. It’s also the fastest out there. I highly recommend using WireGuard if you need a fast and reliable connection between your home computer and some server on the Internet.

RSS Tricks

Thu, 16 Jan 2020 00:00:00 +0000

RSS is a great way to subscribe to various blogs, but it can be used for a lot of other things. For instance, you can get notified of new releases of any GitHub project right from your RSS feed. Here is an example:

https://github.com/rust-lang/rust/releases.atom

It’s also possible to add your favorite YouTube channels to your RSS feed. For example, here is a link to a BBC Documentary feed.

So, YouTube feed links have the following format:

https://www.youtube.com/feeds/videos.xml?channel_id=<id>

BBC Documentary’s channel_id is UCZEPItn2Nb62Zso5eohHAAA and you can get the ID’s of your favorite channels by inspecting your address bar when they are open.

Thunderbolt 4 and USB 4

Mon, 13 Jan 2020 00:00:00 +0000

Thunderbolt 3 is an interface that is available on pretty much every high-end laptop nowadays. It’s almost too good to be true. Thunderbolt-enabled hardware have a set of unified USB ports that can be connected to up to four PCIe 3.0 lanes. One such lane gives us up to 8 GT/s (985 MB/s), so it should give us up to 32 GT/s, although Intel claims that it’s capable of 40 GT/s. I’m not sure where those extra 8 GT/s come from, but many people claim that their Thunderbolt 3 devices are only capable of ~22 GT/s. Anyway, it’s crazy fast!

Fast transfer speed is not the only benefit of Thunderbolt 3, it also allows us to charge our devices or send video output to external monitors. It’s also possible to use an external GPU and even Ethernet over Thunderbolt. Think of a small GPU enclosure that you can connect to your laptop with a single USB cable and aside from being able to do many GPU-intensive tasks, it would also keep your laptop charged and route your traffic via a wired interface which may improve your network throughput and latency. It’s a truly amazing little port.

So, on CES 2020 keynote, Intel announced a new generation of Thunderbolt: Thunderbolt 4, which left many people excited and confused at the same time. Intel promised some speed improvements, so it’s supposed to be 4 times faster than… USB 3.2. 2x1. Well, that was unexpected. Why do they compare Thunderbolt 4 to USB when they have Thunderbolt 3 as a more clear point of reference? They announced that Thunderbolt 4 is 4 times faster than USB 3.2. 2x1, which means 40 GT/s, same speed as Thunderbolt 3. That’s very strange, maybe they’ll clarify things later.

Aside from Thunderbolt standards, now we also have USB4 out in the wild. It’s practically the same thing as Thunderbolt 3, nothing new and exciting, but at least it’s not semi-proprietary which is nice. It looks like we’re going to stick with 40 GT/s for a while.

Actually, I expected that Intel would use the full potential of PCIe 4.0 which doubles the bandwidth to 16 GT/s (1970 MB/s) per every lane. A man can dream.

Wi-Fi vs Wired Networking on Raspberry Pi 4

Mon, 06 Jan 2020 00:00:00 +0000

I ran a few tests to find out how to boost network transfer speed on the Raspberry Pi 4, and whether it’s worth the hassle.

TL;DR

Prefer wired connections when it’s not too inconvenient. For maximum data throughput, you should also use an SSD because regular SD cards won’t let you fully utilize the potential of your wired connection.

Preface

The Raspberry Pi 4 is a capable SBC that comes with two built-in network interfaces: Gigabit Ethernet and IEEE 802.11ac Wi-Fi. Having options is good, but it’s also a common source of confusion. Which interface should we use and why?

In this article, I compare wired and Wi-Fi interfaces in terms of their maximum data throughput. If you’re new to networking, you might also benefit from reading about network latency and its role in the performance of networked software. In short, wired connections usually have an order of magnitude better latency than wireless ones, and that might be more important than the amount of megabytes you can push per second. That said, let’s do some tests!

First, it’s good to keep in mind that we can’t trust the benchmarks completely. Real-world performance can depend on many factors, and it’s always a good idea to reproduce the same tests with your own hardware and in your own environment.

Test Structure

Here is what my test does:

Generate 1 GB of random data on my laptop
Send this data to my Raspberry Pi and measure the average speed
Fetch this data from my Raspberry Pi and measure the average speed

I repeated this test with four different configurations:

A Raspberry Pi that uses Wi-Fi and stores data on an SD card
A Raspberry Pi that uses Wi-Fi and stores data on an SSD
A Raspberry Pi that uses Ethernet and stores data on an SD card
A Raspberry Pi that uses Ethernet and stores data on an SSD

That’s it. Now, it’s time to generate that random data.

Generating Test Data

We can use dd to generate a new file and /dev/urandom is a great data source:

dd if=/dev/urandom of=big.file bs=64M count=16 iflag=fullblock

Let’s clarify these arguments and their purpose:

if - input file. In our case, we attach our input to the stream of random data available on any Linux distribution.
of - output file. It’s the name of the file we want to create. Feel free to use any name you want.
bs - in our case, it means that we want to generate our random file piece by piece and there should be count pieces in it. It also means that we can tweak both arguments; the only important thing is that bs * count should be close to our target size (1 GB).
count - as mentioned above, it’s just a multiplier for bs

Testing Data Throughput Over Wi-Fi and Ethernet

Let’s send some data to the Raspberry Pi (Laptop --> Pi):

rsync --progress big.file pi@cloud.local:~/big.file

big.file
  1,073,741,824 100%   32.10MB/s    0:00:31 (xfr#1, to-chk=0/1)

I used rsync because it’s easy to use, and it reflects my real workloads (syncing a lot of files, doing backups and so on).

Now we can download that big file from the Raspberry Pi and measure the average transfer speed (Laptop <-- Pi):

rsync --progress pi@cloud.local:~/big.file big.file.back

big.file
  1,073,741,824 100%   42.69MB/s    0:00:23 (xfr#1, to-chk=0/1)

That’s it. I repeated this test for each of the tested setups, and here are the results:

Results

From	To	Interface	Storage	Speed (Mbit/s)	Speed (MB/s)
Dell XPS 13 (2018)	Raspberry Pi 4	Wi-Fi	SD	58.24	7.28
Dell XPS 13 (2018)	Raspberry Pi 4	Wi-Fi	SSD	55.84	6.98
Dell XPS 13 (2018)	Raspberry Pi 4	Gigabit Ethernet	SD	159.60	19.95
Dell XPS 13 (2018)	Raspberry Pi 4	Gigabit Ethernet	SSD	340.56	42.57
Raspberry Pi 4	Dell XPS 13 (2018)	Wi-Fi	SD	80.72	10.09
Raspberry Pi 4	Dell XPS 13 (2018)	Wi-Fi	SSD	82.00	10.25
Raspberry Pi 4	Dell XPS 13 (2018)	Gigabit Ethernet	SD	339.20	42.40
Raspberry Pi 4	Dell XPS 13 (2018)	Gigabit Ethernet	SSD	341.52	42.69

Interesting results, but let’s try an alternative approach to measuring network throughput before jumping to any conclusions.

Testing Raspberry Pi 4 Throughput With iPerf

We can easily measure “raw” network performance by moving as much data as possible between two computers for a certain period of time. One of those computers has to play the role of a client by generating a lot of gibberish and sending it to the second machine, which can simply discard all of that traffic upon arrival.

There are many ways to perform such a test, but iperf is something of an industry standard, and it’s very easy to use, which makes it a good choice for our quick test.

Let’s make Raspberry Pi 4 a server:

ssh pi@pi.local
sudo apt install iperf
iperf --server

Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)

Now, we can use iperf from another computer and instruct it to connect to the Raspberry Pi in order to measure the network throughput between them:

sudo apt install iperf
iperf --client pi.local

Client connecting to cloud.local, TCP port 5001
TCP window size: 85.0 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes   942 Mbits/sec

Here are my results for wired and Wi-Fi connections:

Interface	Speed (Mbit/s)
Gigabit Ethernet	942
Wi-Fi (IEEE 802.11ac)	86

Both tests point to the same conclusion.

Conclusion

Although using Wi-Fi is more convenient because it gives you more flexibility, don’t expect your network transfer speed to be anywhere above 86 Mbit/s if you connect your Raspberry Pi wirelessly. 86 Mbit/s is actually not that bad if you use an SD card because many modern SD cards can’t go beyond ~80 Mbit/s when they write data, although they can read a bit faster. Not being able to take advantage of that extra read speed is the main drawback of using Wi-Fi on a Raspberry Pi 4.

Things get a bit different if you prefer to store your data on an SSD. Modern solid-state drives can move gigabytes in seconds, but your Wi-Fi connection wouldn’t let you take advantage of that extra throughput. Switching to a wired connection boosts transfer speed up to about 30-40 MB/s when using rsync, which is better, but still a far cry from the theoretical limit of 125 MB/s (that’s the max speed of a Gigabit Ethernet adapter that is used in Raspberry Pi 4). It might be due to encryption overhead, which puts some pressure on the CPU and makes it a bottleneck. That’s precisely why benchmarking is hard: there are many components that might affect the results.

RSS is Live and Kicking

Mon, 21 Oct 2019 00:00:00 +0000

I’ve been experimenting with using Nextcloud as my RSS aggregator for the last couple of weeks. It works, but I wouldn’t call it the best option.

The main reason I’m hesitant is the lack of visible development activity in the backend code and the official Android app. When a core feature feels stagnant, it’s hard to trust its long-term future.

There is a notable upside, though: the built-in RSS server is compatible with third-party clients like the popular FeedReeder for Linux. This interoperability is a solid feature that keeps it usable even if the native apps lag behind.

My trial is ongoing. If Nextcloud doesn’t feel solid by the end of it, my fallback plan is to set up a dedicated Tiny Tiny RSS server, which has a reputation for being robust and actively maintained.

Igor Bubelov

Core Lightning Post Mortem

Table of Contents

The Old Setup

The CLN Shenanigans

The Dynamic Linking Problem

LND Gets It Right

The Counterarguments

What I Miss

Conclusion

Wireless Charging

The Accident

The WPC

Magnets

The Bridge to the Old World

Conclusion

Wi-Fi is the Best Protocol for Smart Lights

$4 Zigbee Lightbulb Deep Dive

Table of Contents

Intro

Node Descriptor

Logical Type

Complex Descriptor Available

User Descriptor Available

Reserved

APS Flags

Frequency Band

Medium Access Control Capability Flags

Manufacturer Code

Maximum Buffer Size

Maximum Incoming Transfer Size

Server Mask

Maximum Outgoing Transfer Size

Descriptor Capability Field

Endpoint 1 of 1

Profile ID

Device Type

Input Clusters

Output Clusters

Manufacturer

Model

Class

Conclusion

Road to Zigbee

Philips Hue

Nostr

Tailscale

Beej's Guide to Network Programming (2019)

CLBOSS

Mastodon

Miniflux

USB Everywhere

Tor Switches to Rust

Setting Up SSH Passwordless Auth

Wi-Fi 6 Router

State of the Lightning Network

Webfeeds

The World Needs Another Feed Reader

WireGuard

RSS Tricks

Thunderbolt 4 and USB 4

Wi-Fi vs Wired Networking on Raspberry Pi 4

Table of Contents

TL;DR

Preface

Test Structure

Generating Test Data

Testing Data Throughput Over Wi-Fi and Ethernet

Results

Testing Raspberry Pi 4 Throughput With iPerf

Conclusion

RSS is Live and Kicking