I was hanging out on #lobsters on IRC (Libera Chat), and people were discussing different flavours of Linux which prompted me to reflect whether Nix solves the problems I encounter from day to day.

Fearless Upgrades

On non declarative systems, there is a lot of manual maintenance to ensure the system runs smoothly. Double or triple that effort if you have more than one machine. When I first got into Linux, I put Arch Linux on my laptop. The install went smoothly and I was off to the races, until I received an email explaining that I need to update the package manager, Pacman. So I followed the instructions, and I broke something and I couldn't access the mirrors. Given my general inexperience, I just reinstalled the whole thing because I wasn't sure how to fix it.

My experience with Arch left an impression on me that my system should reliable and not break when I'm doing routine maintenance. Luckily, I found out about NixOS and its declarative setup so I never have to feel scared to upgrade my system because I know that I can always rollback to a previous change.

This allows me to easily bisect issues to find the exact version where it broke. For example on Nix Darwin (Nix for MacOS), there was a linker error when I tried to update audacity because it was built for newer MacOS versions than my system. My first solution was to add another input for nixos-26.05, and rebuild an older version of audacity which didn't have the issue. This upgrade strategy lets me update everything else to the bleeding edge while simultaneously pinning certain packages, that haven't caught up, to a stable version. This kind of reproducibility issue is inherent to MacOS not being designed with the same guarantees of Nix in mind.

With the assurance that I can easily roll back a disastrous update, I've had great success managing a fast update cycle on my desktop and production server. I believe that a good way to manage updates is with small frequent changes that let you easily pinpoint the issue, rather than deal with them all at once during one upgrade. That's why the guarantee of fearless upgrades is so appealing to my sensibilities.

Configuration All in One Place

Dotfiles and config files are usually all over the place, which is why I find Nix incredibly useful for organizing and deploying them to my server and desktop PC. For instance, I manage SSH authorized keys and configuration, a myriad of system users with varying permissions, Git, cgit, soju (IRC bouncer), nix-serve-ng (Nix bin cache), NFS, WireGuard, GnuPG, CDN, and split-horizon DNS.

Recently, I was experimenting with split-horizon DNS, which routes DNS queries like git.skullheadx.com directly to my homelab if I'm on my local network, otherwise it routes through my public VPS to my homelab over WireGuard. It turns out adding split-horizon DNS had an unintended side effect on my git ssh autotunnel from my homelab to VPS causing an incident leading to degraded service on git over ssh operations. In other words, it broke because the ssh autotunnel relied on the routing for git.skullheadx.com to route to the VPS, but since the homelab is on my local network, the DNS resolver just told my homelab to look at its own local IP. I was able to hone in on the issue because I can turn off certain services, like dnsmasq, to test my entire server configuration. Moreover, thanks to being tracked in Git history, I bisected the changelog, and was able to find when the issue started to occur, and I resolved it by telling the git auto ssh tunnel to use the IP of the VPS directly, rather than relying on the faulty assumption that it would get routed where I wanted it to.

I strongly prefer this style of system administration because I have specifically designed my flake to be abundantly clear, following the suckless philosophy. Since my configuration is deployed on real hardware (not some nebulous cloud), I explicitly define the services and programs that I need for each host all in one place, which allows me to see what's deployed there at a glance.

Actually Solving the Reproducibility Problem

I did a nix store optimise which deletes identical files and replaces them with hard links to a single copy. I ran it on my laptop which cleared 40 GB of duplicates. It made me reflect on the problem that Nix is trying to solve: how can we reproduce a package exactly the same on everyone's computer?

This is a really hard problem to solve and has given rise to many solutions like containers. Yes they can work, but Nix solves the problem better than containers because it lockfiles and reproduces only the exactly dependencies required, meanwhile the containers will install a whole OS image so that the packages used can be reproduced.

In my opinion, it's pretty overkill to download and install a whole new image for every single deploy. If you were engineering a proper solution, you should make only the things you are trying to deploy reproducible, including all its dependencies. Take this to the scale of an entire OS, and Nix becomes the obvious answer since it tracks all those dependencies.

But taking a step even further back, why is this a problem in the first place? The whole philosophy of UNIX and Linux systems is that it's intended for any developer to come and add programs, and build off of the community's work. Because there are so many contributors, many anonymous, that makes it extremely difficult to standardize an environment that everyone should use. Some may say that the culture of replacing the existing programs with any of your choice is a selling point of Linux, but this approach is not fully without downsides, since we now have no idea what libraries and dependencies exist on the user's system. I've recently started using OpenBSD as as a router for all my internet traffic towards my servers. Most of the programs that I use are first party, and they all just integrate seamlessly with each other. Moreover, the general attitude towards system upgrades is to update everything to the next release altogether. It's a completely different approach to the goal of achieving a stable operating system, which works with significantly less complexity.

To Conclude This Reflection

I still think that NixOS and OpenBSD have their places. For everything that needs to be rock solid, low resource, and highly secure I've opted for OpenBSD. My BSD router handles TLS termination, relayd, firewall to the public internet and static web hosting (my business).

Nix is invaluable for trying out new, and 3rd party code. For example, I'm running cgit, soju IRC bouncer, Dnsmasq split horizon DNS, Nix bin cache, and NFS. The flexibility and power from the three advantages we've explored make NixOS a joy to quickly deploy new services, and debug when if they go wrong.