NixOS on Raspberry Pi: The Good, the Bad and "the Linux way"
Gran Torino (Film) © 2008 Warner Bros. Pictures
Disclaimer: this is not a tutorial, but a story of me trying the new hype thing that I am probably too old for 😀
I have a confession to start this article with: I have never enjoyed “the Linux way” of doing things:
- “Everything at once” package updates via distributives
- Having a strong allergy to binaries
- Piping like a 19 y.o. (Urban Dictionary is your friend)
- Being religious about everything and how things should be done (but always having 42 ways of achieving the same!)
“Do one thing, and do it well!”. Ummm… if that “one thing” is “arguing on Internet on why your way is the only way” then yeah, well done!
And it comes with no surprise that I always tried to avoid “the Linux way”. Be it with WORA (“Write Once Run Anywhere”) in Java, static Go binaries, or… with Docker.
Ever since discovering Docker in 2014 (a decade ago!), I was always consistent with my perception of this technology. No, Docker did not invent containers, they existed in many shapes or forms for many more decades.
What they did is they broke the vicious circle of “the Linux way” and democratized Linux containers by finally providing a userdeveloper-friendly tooling for building, distributing and running them. No more piping of the output of one command to another, tons of flags and outdated documentation (if any!).
Revolutionary! Why couldn’t we have it from the beginning?
But containers mean that you forcibly isolate processes from the rest of the system, by design.
There is nothing bad with it, just limiting.
Sure, I can share filesystems and even networks between the containers.
There are even ways to make it look like everything is running in the same space.
But what if that’s not enough? Or maybe I am not ready to pay the “containerization tax”?
These questions were always bothering me, but I always paid the “containerization tax” because the benefits of immutability were outweighing the cost. And, to be fair, not like there were many (if any) alternatives.
And those of you who are capable of writing a Hello World without Copilot’s help may remember CoreOS - a package manager-less Linux distributive that doubled down on containers (although not with Docker, but with “Prt Sc Sys Rq” inspired rkt).
And while CoreOS has been discontinued in 2020, I did like the idea of the “immutable” Linux OS a lot.
Imagine being able to read your server’s configuration from a single source of truth?
Almost as good as not having to pipe the outputs of commands to each other just to achieve a simple thing!
And when I heard of NixOS for the first time, I couldn’t stop thinking “can it be it?”.
It promises to have the immutable design, can be configured as code, reproducible, “pure” and “hermetic” - a property of Nix builds, which isolates them from the host system via various mechanisms.
Inherently, it helps to avoid the “DLL hell”, or at least claims to do so.
So I’ve decided to try it by learning on my most mission critical server - the homelab!
(because I need it, but never have enough time to fix it if it goes bad, nor am getting paid for working on it, despite the stuff I’m doing being even more advanced than the things I am getting paid for, typically 😀)
A brand new way of managing the OS, on an exotic hardware (ARM 64-bit via Raspberry Pi 400), what could go wrong?
The Good
When I was young, I tended to rush and try every new emerging technology. Well, I am not yet old, but I learned the power of letting the dust settle before trying something. iOS-like “.1 version” mentality, if I may.
But, with Nix/NixOS, it totally went past me (and it let a lot of dust to settle since the invention in 2003). To be honest, I didn’t even know about its existence up until 2020s 🤯 But, to be fair, many didn’t.
So I had to learn both Nix and NixOS at the same time to get going. And, if you are reading this article, there are likely only two reasons why: you either came here from Reddit/HN to cancel me for my Linux takes or you want to follow the same process for your own needs. And, if the latter, be prepared to learn both as well, although IMO NixOS adds only ~10% learning overhead on top.
The good news is that NixOS was mature enough for me to get going, even in Raspberry PI.
The documentation exists, first and foremost, and is extensive (too extensive to the point of misleading, if you ask me, but better than no documentation at all). Here are the two starting links:
- https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi
- https://nixos.wiki/wiki/NixOS_on_ARM/Raspberry_Pi_4
If you are impatient like me, or prefer shorter quick start instructions, the tl;dr: is:
Download the generic SD image
⚠️ WARNING!
The provided link (that I copied from the linked docs!) leads to… a build server.
I am not a CISO at a large Enterprise so I don’t mind the latest and greatest, but “the very latest and sometimes brokest” is too much even for me, but I couldn’t find a quick way to download the latest release build of the SD image.
nixos-sd-image-25.05beta724962.d70bd19e0a38-aarch64-linuxworked fine for me, but oh my…Use your preferred way of burning the image onto the SD card (the good Raspberry Pi Imager (v1.8.5) worked just fine for me), like you would normally do with Raspbian
Burn it onto the SD card and proceed booting Raspberry Pi as you would normally do
Not much of an instruction but that’s the point.
But this is only where the fun starts!
Because I am impatient, I totally borked my brand new system after attempting to nixos-rebuild switch it.
Why? Because the configuration was empty, and (I guess?) it deleted my nixos user!
But don’t worry, NixOS is very “all thumbs” friendly: just reboot your RPi and you will be presented with a selector of the latest builds. The one from 1970 (no, NixOS is not THAT old, we all know where this year is coming from) is the stock configuration that gives you a second chance.
To fix it, you need to run nixos-generate-config (don’t ask me why it is not generated by default. Just don’t.),
that will create the main configuration file /etc/nixos/configuration.nix and /etc/nixos/hardware-configuration.nix that it will import.
But don’t rush switching to it, let’s create a user for ourselves and enable SSH so we don’t have to edit it directly on RPi.
I will not go deep into the Nix syntax and NixOS semantics. Can’t say that it is easy to google and that the docs are easy to follow, but it is just too much to cover in this article.
The simpliest config to start with would be something like this:
# /etc/nixos/configuration.nix
{ config, lib, pkgs, ... }:
{
system.stateVersion = "25.05";
imports = [
./hardware-configuration.nix
./user.nix
];
boot.loader.grub.enable = false;
boot.loader.generic-extlinux-compatible.enable = true;
networking = {
hostName = "raspberrypi";
firewall = {
enable = true;
allowPing = true;
};
};
services = {
openssh.enable = true;
};
}# /etc/nixos/user.nix
{ pkgs, ... }:
{
users.users.bsideup = {
isNormalUser = true; # lies...
extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
openssh.authorizedKeys.keys = [
"ssh-ed25519 AAAASOMUCHTIMEWASTEDBECAUSEEXAMPLESSHOWPASSWORDBASEDAUTH/5Kf0zSINGbU3ZLU6nUNqsWimF52 bsideup@Non-Linux-Maybe-Next-Year-Desktop.local"
];
};
security.sudo.extraRules = [
{
users = [ "bsideup" ];
commands = [
{
command = "ALL" ;
options = [ "NOPASSWD" ];
}
];
}
];
# Surprise your CISO with this simple trick!
services.getty.autologinUser = "bsideup";
}(Obviously replace bsideup with your username and authorize your own public key. Ironically, many examples on the internet suggest providing the seed password, despite the pubkey being easy to use -_-)
Now if we nixos-rebuild switch it, the user will be created, SSH server activated, and we should be able to just ssh raspberrypi.local to it (or whatever the hostname you picked).
I did break the leg multiple times going through the green path but you should have seen my excitement when I got something working for the first time 😀
After a few hours, I could finally do something useful with it and replicate my old, pre-NixOS system.
I’ve decided to start with the Time Machine backup destination (Avahi + Samba), it was pretty easy:
Mount the external USB SSD drive by adding the following or similar block to
configuration.nix:fileSystems."/mnt/backups" = { # Yours will be different device = "/dev/disk/by-uuid/b773149d-bf3a-46b9-8a91-af4c3f7907fb"; fsType = "ext4"; options = [ "noexec" "nodev" "noatime" "nodiratime" ]; };Create
time-machine.nix(don’t forget to import it fromconfiguration.nix!) with something like this (wherebsideupis your username):{ pkgs, ... }: { services = { samba = { package = pkgs.samba4Full; # ^^ `samba4Full` is compiled with avahi, ldap, AD etc support # (compared to the default package, `samba`) # Required for samba to register mDNS records for auto discovery # See https://github.com/NixOS/nixpkgs/blob/592047fc9e4f7b74a4dc85d1b9f5243dfe4899e3/pkgs/top-level/all-packages.nix#L27268 enable = true; openFirewall = true; settings.timemachine = { path = "/mnt/backups"; browseable = "yes"; "read only" = "no"; "guest ok" = "no"; "create mask" = "0600"; "directory mask" = "0700"; comment = "Raspberry Pi Time Capsule"; "writeable" = "yes"; "valid users" = "bsideup"; "write list" = "bsideup"; "force user" = "bsideup"; "vfs objects" = "catia fruit streams_xattr"; "fruit:aapl" = "yes"; "fruit:time machine" = "yes"; }; }; avahi = { enable = true; openFirewall = true; nssmdns4 = true; publish = { enable = true; addresses = true; domain = true; hinfo = true; userServices = true; workstation = true; }; extraServiceFiles = { smb = '' <?xml version="1.0" standalone='no'?><!--*-nxml-*--> <!DOCTYPE service-group SYSTEM "avahi-service.dtd"> <service-group> <name replace-wildcards="yes">%h</name> <service> <type>_smb._tcp</type> <port>445</port> </service> <service> <type>_device-info._tcp</type> <port>9</port> <txt-record>model=TimeCapsule8,119</txt-record> </service> <service> <type>_adisk._tcp</type> <port>9</port> <txt-record>dk0=adVN=backups,adVF=0x82</txt-record> <txt-record>sys=adVF=0x100</txt-record> </service> </service-group> ''; }; }; }; }Authorize your user with
smbpasswd -u bsideup(wherebsideupis your username)
One more nixos-rebuild switch and my precious filesjunk is regularly backed up again, yay.
In retrospective, that samba4Full instead of samba was already hinting at why this looked so simple so far…
The Bad
While getting the Samba + Avahi running was easy, my next target was the Unifi Controller - Ubiquiti’s free software that you can self host to control your Unifi (big fan!) network. And good news! It is available as a NixOS package. Well… kinda.
Starting with it was easy:
# unifi.nix
{ config, lib, pkgs, ... }:
{
# Poor FOSS kittens
nixpkgs.config.allowUnfree = true;
networking.firewall.allowedTCPPorts = [ 8443 ];
fileSystems."/var/lib/unifi/data/backup" = {
device = "/mnt/backups/unifi";
fsType = "none";
options = [ "bind" ];
};
services.unifi = {
enable = true;
openFirewall = true;
};
}But, when I attempted to switch to it, I was presented with a surprise… MongoDB.
I don’t know who at Ubiquiti thought that it is a good idea to use MongoDB as a database for it, but it is what it is. And while it is not a huge (but big) deal anywhere else, on my Raspberry Pi 400 it would mean that I have to wait for many hours for it to compile.
Not to mention that it made my RPi hang during the compilation, because apparently I forgot to configure the swap and no swap was configured by default! Fixed it with the following:
#hardware-configuration.nix
swapDevices = [{
device = "/swapfile";
size = 16 * 1024; # 16GB
}];Why compile? Because, unlike many other packages, the MongoDB package is not available as a pre-built package due to its non-OSS license. So I went down the rabbit hole…
So close…
After a quick scan on GitHub for the usage of the unifi package, I’ve found the mongodb-ce package that uses the pre-built binaries for MongoDB.
I’ve quickly changed the config to it:
services.unifi = {
enable = true;
mongodbPackage = pkgs.mongodb-ce.overrideAttrs {
# Unifi Controller does not support MongoDB v8,
# and the latest NixOS distributive (why are we doing it again...)
# comes with 8.0.4 by default
version = "7.0.14";
};
openFirewall = true;
};The build has passed and… nothing.
I knew that the Unifi Controller starts MongoDB daemon as a subprocess from Java (yeah…),
and a quick htop has identified that the MongoDB process crashes.
Apparently, the MongoDB Community binaries do not support Raspberry Pi 4 / 400: https://www.umpah.net/install-mongodb-on-a-raspberry-pi/
Awww… my first Nix package
After giving up with available MongoDB options, I’ve decided to bite the bullet and build my own Nix package for that MongoDB dependency.
Luckily, there are pre-built MongoDB binaries from a MongoDB employee: https://github.com/themattman/mongodb-raspberrypi-binaries
A quick read on how packages are structured gave me the following:
services.unifi = {
enable = true;
mongodbPackage = pkgs.stdenv.mkDerivation rec {
pname = "mongodb";
version = "7.0.14";
src = pkgs.fetchurl {
url = "https://github.com/themattman/mongodb-raspberrypi-binaries/releases/download/r7.0.14-rpi-unofficial/mongodb.ce.pi4.r7.0.14.tar.gz";
hash = "sha256-b0F8ihngN20CUc/tXFHao8JVroDiEfsSfDs2QErkma0=";
};
installPhase = ''
mkdir -p $out/bin
install -Dm 755 mongod $out/bin/mongod
runHook postInstall
'';
};
openFirewall = true;
};Result? A failure. 15 minutes of googling later, I learned that Nix assumes a single sub-folder in the archive. Obviously…
So we need to specify sourceRoot = "." to override that assumption.
Result? Successful build! Hooray, champagne popping, Linux commands piping, everyone is happy… not. WTH?…
That’s how I learned another thing about Nix: it does not like dynamic libraries (neither do I!).
But the MongoDB binaries were compiled with them in mind. NGL I was almost ready to give up, but then I looked at the mongodb-ce package for inspiration, and a-ha! Nix build can patch the binaries with autoPatchelfHook and a bit of hinting with buildInputs. Sweet.
So I copied their config and replaced the URL:
services.unifi = {
enable = true;
mongodbPackage = pkgs.stdenv.mkDerivation rec {
pname = "mongodb";
version = "7.0.14";
src = pkgs.fetchurl {
url = "https://github.com/themattman/mongodb-raspberrypi-binaries/releases/download/r7.0.14-rpi-unofficial/mongodb.ce.pi4.r7.0.14.tar.gz";
hash = "sha256-b0F8ihngN20CUc/tXFHao8JVroDiEfsSfDs2QErkma0=";
};
sourceRoot = ".";
buildInputs = [
pkgs.curl.dev
pkgs.openssl.dev
(lib.getLib pkgs.stdenv.cc.cc)
];
nativeBuildInputs = [ pkgs.autoPatchelfHook ];
dontStrip = true;
installPhase = ''
mkdir -p $out/bin
install -Dm 755 mongod $out/bin/mongod
runHook postInstall
'';
};
openFirewall = true;
};Resut? Build failure. What a great reminder that NixOS is still Linux - a never ending loop of trying and mostly failing…
What this time? An obscure Libssl.so.1 not found error! But… don’t we have it… there?
Turns out that particular binaries need OpenSSL 1.1, and that openssl.dev isn’t it -_-.
Remind me again, what’s wrong with Java?
I will spare you another code snippet, because just adding pkgs.openssl_1_1 will not work, because… it is deprecated due to vulnerabilities!
But luckily there is a way to still use it by adding the package to permittedInsecurePackages. Phew!
So here is my final, working (as of today, lol) config for Unifi on Raspberry Pi 4 / 400:
{ config, lib, pkgs, ... }:
{
networking.firewall.allowedTCPPorts = [ 8443 ];
fileSystems."/var/lib/unifi/data/backup" = {
device = "/mnt/backups/unifi";
fsType = "none";
options = [ "bind" ];
};
nixpkgs.config.permittedInsecurePackages = [
"openssl-1.1.1w"
];
services.unifi = {
enable = true;
mongodbPackage = pkgs.stdenv.mkDerivation rec {
pname = "mongodb";
version = "7.0.14";
src = pkgs.fetchurl {
url = "https://github.com/themattman/mongodb-raspberrypi-binaries/releases/download/r7.0.14-rpi-unofficial/mongodb.ce.pi4.r7.0.14.tar.gz";
hash = "sha256-b0F8ihngN20CUc/tXFHao8JVroDiEfsSfDs2QErkma0=";
};
sourceRoot = ".";
buildInputs = [
pkgs.curl.dev
pkgs.openssl_1_1
(lib.getLib pkgs.stdenv.cc.cc)
];
nativeBuildInputs = [ pkgs.autoPatchelfHook ];
dontStrip = true;
installPhase = ''
mkdir -p $out/bin
install -Dm 755 mongod $out/bin/mongod
runHook postInstall
'';
};
openFirewall = true;
};
}The Ugly
While I did achieve the result that I wanted, I can’t yet agree that Nix/NixOS is “the next Docker”, as some people say.
It is definitely an improvement over the traditional Linux experience. That said, it is still very much “the Linux way” in many aspects:
- Poor documentation. I can’t emphasize this enough! It is not non-existent, but oftentimes misleading, or outdated. Nix feels like a system where copy/pasting snippets from internet is how 90% of tasks will be achieved, so it has to be good.
- The syntax, while powerful, can be very confusing.
I found this thread fascinating: https://discourse.nixos.org/t/what-is-the-added-benefit-of-let-in-compared-to-plain-with-with-rec/14248
- idk, maybe it is a “me” issue, but the way it turned into a potato after I applied
nixos-rebuild switchon a stock installation was really surprising!
It would be super cool to have a “–safe” flag innixos-rebuild switchthat will rollback to the previous configuration if no confirmation was received in 30s or like so (Display Settings-style), to avoid having to restart the machine if you loose the access. - Not everything has to be FOSS: https://discourse.nixos.org/t/petition-to-build-and-cache-unfree-packages-on-cache-nixos-org/17440
A regular user won’t care about the legalities of distributing binaries, they just want to havedocker run mongo:7-like experience - Probably it made sense for “traditional” Linux distributives, but I don’t see much value in having a distributive-like approach in NixOS, with over 100,000 packages in https://github.com/NixOS/nixpkgs versioned under a single version.
I did learn how to use an older version of the package to pinmongodb-ceto the pre-24.05 implementation before discovering the attributes override and it was nowhere close todocker run mongo:7, sorry.
I find the tech at the core of it to be very interesting but the UX could use a major improvement.
Kinda reminds me of Linux containers and how Docker did it.
Maybe, similar to LXC, someone just needs to take the good parts and slap a fantastic UX/DX on it? 🤓
You can find the final result as a Gist here:
https://gist.github.com/bsideup/8bb24c0c29d6329f70e55fe7ccb3e1e0
Bonuses
Bonus 1: edit NixOS configuration remotely with VSCode over SSH
As you probably have guessed, I am not a Vim ninja. So I wanted to be able to edit the Nix configuration with comfort of the IDE like VS Code. My initial attempt to use the VS Code Remote support has miserably failed, because it was unable to install the required components on the remote machine (NixOS is not a regular Linux distro!).
I started looking for alternatives and discovered the SSH FS extension that allows editing files over SSH. Almost perfect!
It even supports elevating the permissions with sudo, to be able to edit the root-owned configuration.nix.
The problem is that it uses the sftp-server binary that is not easily available on NixOS, and I couldn’t understand why it fails, despite being able to establish an SSH connection.
After some GitHub issues digging, I’ve finally came up with a config that worked!
"sshfs.configs": [
{
"name": "raspberrypi-nixos",
"host": "raspberrypi.local",
"username": "$USERNAME",
"debug": false,
"privateKeyPath": "$HOME/.ssh/id_ed25519",
"sftpCommand": "nix-shell -p openssh --run '$buildInputs/libexec/sftp-server'",
"sftpSudo": true
}
]It uses nix-shell with the openssh package, in which the sftp-server binary is available.
This enabled me to edit the config remotely from my MacBook as if it was available locally. Sweet!
Bonus 2: How to modify EEPROM
While it was literally in the docs, I somehow missed it.
But what made it worse is that the “normal” way (with rpi-eeprom-update) worked, but the result was never saved due to the way how NixOS works.
So I will “cheat” and just copy/paste the docs here, to point your attention to the fact that you must point the tool to the “right” location, because the default /boot/ is managed by NixOS and can’t be changed with the tool:
$ nix-shell -p raspberrypi-eeprom
$ sudo mount /dev/disk/by-label/FIRMWARE /mnt/firmware
$ sudo BOOTFS=/mnt/firmware FIRMWARE_RELEASE_STATUS=stable rpi-eeprom-update -d -aBonus 3: cloning/enlarging the SD card
Quickly after starting to use NixOS I realized that the stock 32Gb SD Card may not be enough, lol. So I decided to order a V30 256Gb SD card instead.
I totally could have utilized the reproducible nature of NixOS and I even regret not doing so,
but my lazy ass shortsightedness made me want to clone the old SD card instead.
After a bit of googling, I’ve found balenaEtcher to be the easiest solution to copy the card and keep it bootable. One big advantage of the tool is that it can copy from one card to another (if you have two card readers).
Sure, you can use dd (you can even pipe its output! 🫠) but good luck getting a “sparse” image instead of the full 32Gb ISO, even if only 1-2Gb are taken.
Then you can use resize2fs /dev/mmcblk1p2 (where mmcblk1p2 is your root partition, check with df -H) after booting into the system (yes, I did it live. YOLO when you have a reproducible system, I guess :)).