This repository allows you to build a custom SD image of NixOS for your Raspberry Pi (or any other
supported AArch64 device) in about 15-20 minutes on a modern x86_64
system or about 5 minutes on a
powerful AArch64
box, without installing any additional dependencies.
The default configuration enables OpenSSH out of the box, allowing to install NixOS on an embedded device without attaching a display.
This works both on x86_64
systems and native AArch64
builders. When needed, QEMU is
used to emulate AArch64
and binfmt_misc
is used to
allow transparent execution of AArch64 binaries.
A Packer specification is provided in packer/
which allows to build an
SD image using a native AArch64 instance provided by Amazon EC2. It takes less than 10 minutes!
Out of the box this supports:
- any device supported by the
sd-image-aarch64
builder of NixOS. This includes the Raspberry Pi 3 and other devices listed here. - Raspberry Pi 4. Please note that the latest Raspberry Pi 4 model with 8 GiB of RAM
is supported upstream only when building an image from the unstable branch. To do that,
open
docker/docker-compose.yml
and changeNIXPKGS_BRANCH
tomaster
.
Any other device can be supported by changing the configuration files in config/
.
Start by cloning this repo and opening config/sd-image.nix
in your favorite
editor. Then:
- choose the target device by following the instructions in the file;
- add your SSH key(s) by replacing the existing
ssh-ed25519 ...
placeholder.
Customize sd-card.nix
(or add more files) as you like, they will be copied to the container.
Protip: if you're building for a Raspberry Pi 4 and don't need ZFS, enable
DISABLE_ZFS_IN_INSTALLER
in docker/docker-compose.yml
to speed
up the build. Please note that if you have already executed run.sh
once, you need to rebuild
the images after changing this flag using ./run.sh up --build
.
Finally, ensure that your Docker is set up and you have a working installation of Docker Compose, then just run:
./run.sh
The script is just a wrapper around docker-compose
which makes sure that the right parameters
are passed to it.
And that's all! Once the execution is done, a .img
file will be produced and copied in this
directory. To free up the space used by the containers, just run:
./run.sh down --rmi all -v
WARNING: This interacts with the host kernel to set up a binfmt_misc
handler to execute
AArch64 binaries. Due to this, some containers have to be executed with the --privileged
flag.
To quickly build an SD image using a native AArch64 EC2 instance, head over to the
packer/
subdirectory which has a Packer
specification to do it in two commands and less than 10 minutes.
Before Packer, there was also a Terraform specification, but I removed it in favor of the Packer
one. It is still accessible in the
terraform
branch.
Once an image is produced by the container it's sufficient to flash it to the SD card of your
choice with any tool which can flash raw images onto block devices. There have been some reports
of issues using Etcher on macOS, thus it might be easier to just use
dd
.
Hopefully, the flashed SD card should just work on your device. The unofficial wiki contains lots of resources for possible things you might need or that might go wrong when using NixOS on a Raspberry Pi. Check out the page for all ARM devices too.
Once your Pi boots and you're logged in, you can generate a barebones configuration using
nixos-generate-config
.
Please keep in mind the following:
- the installer configuration (which is the one you edited in the
config/
folder) is different than the system configuration. The installer configuration is only used to build the image -- using an installer configuration on a production system is an error and will lead to weirdness. - the system configuration will not inherit from the installer configuration, thus any
relevant options (such as users, SSH keys, networking etc.) have to be configured again. Please
note that once you switch to the main system configuration the
nixos
user will be removed.
That being said, here are some example system configurations that are mostly ready to use for both the Pi 3 and Pi 4:
Example configuration for the Pi 3
# Please read the comments!
{ config, pkgs, lib, ... }:
{
# Boot
boot.loader.grub.enable = false;
boot.loader.raspberryPi.enable = true;
boot.loader.raspberryPi.version = 3;
boot.loader.raspberryPi.uboot.enable = true;
# Kernel configuration
boot.kernelPackages = pkgs.linuxPackages_latest;
boot.kernelParams = ["cma=32M"];
# Enable additional firmware (such as Wi-Fi drivers).
hardware.enableRedistributableFirmware = true;
# Filesystems
fileSystems = {
"/" = {
device = "/dev/disk/by-label/NIXOS_SD";
fsType = "ext4";
};
};
swapDevices = [ { device = "/swapfile"; size = 1024; } ];
# Networking (see official manual or `/config/sd-image.nix` in this repo for other options)
networking.hostName = "nixpi"; # unleash your creativity!
# Packages
environment.systemPackages = with pkgs; [
# customize as needed!
vim git htop
];
# Users
# === IMPORTANT ===
# Change `yourName` here with the name you'd like for your user!
users.users.yourName = {
isNormalUser = true;
# Don't forget to change the home directory too.
home = "/home/yourName";
# This allows this user to use `sudo`.
extraGroups = [ "wheel" ];
# SSH authorized keys for this user.
openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ];
};
# Miscellaneous
time.timeZone = "Europe/Rome"; # you probably want to change this -- otherwise, ciao!
services.openssh.enable = true;
# WARNING: if you remove this, then you need to assign a password to your user, otherwise
# `sudo` won't work. You can do that either by using `passwd` after the first rebuild or
# by setting an hashed password in the `users.users.yourName` block as `initialHashedPassword`.
security.sudo.wheelNeedsPassword = false;
# Nix
nix.gc.automatic = true;
nix.gc.options = "--delete-older-than 30d";
boot.cleanTmpDir = true;
# https://nixos.wiki/wiki/FAQ/When_do_I_update_stateVersion
system.stateVersion = "20.03";
}
Example configuration for the Pi 4
# Please read the comments!
{ config, pkgs, lib, ... }:
{
# Boot
boot.loader.grub.enable = false;
boot.loader.raspberryPi.enable = true;
boot.loader.raspberryPi.version = 4;
# Kernel configuration
boot.kernelPackages = pkgs.linuxPackages_rpi4;
boot.kernelParams = ["cma=64M" "console=tty0"];
# Enable additional firmware (such as Wi-Fi drivers).
hardware.enableRedistributableFirmware = true;
# Filesystems
fileSystems = {
# There is no U-Boot on the Pi 4 (yet) -- the firmware partition has to be mounted as /boot.
"/boot" = {
device = "/dev/disk/by-label/FIRMWARE";
fsType = "vfat";
};
"/" = {
device = "/dev/disk/by-label/NIXOS_SD";
fsType = "ext4";
};
};
swapDevices = [ { device = "/swapfile"; size = 1024; } ];
# Networking (see official manual or `/config/sd-image.nix` in this repo for other options)
networking.hostName = "nixpi"; # unleash your creativity!
# Packages
environment.systemPackages = with pkgs; [
# customize as needed!
vim git htop
];
# Users
# === IMPORTANT ===
# Change `yourName` here with the name you'd like for your user!
users.users.yourName = {
isNormalUser = true;
# Don't forget to change the home directory too.
home = "/home/yourName";
# This allows this user to use `sudo`.
extraGroups = [ "wheel" ];
# SSH authorized keys for this user.
openssh.authorizedKeys.keys = [ "ssh-ed25519 ..." ];
};
# Miscellaneous
time.timeZone = "Europe/Rome"; # you probably want to change this -- otherwise, ciao!
services.openssh.enable = true;
# WARNING: if you remove this, then you need to assign a password to your user, otherwise
# `sudo` won't work. You can do that either by using `passwd` after the first rebuild or
# by setting an hashed password in the `users.users.yourName` block as `initialHashedPassword`.
security.sudo.wheelNeedsPassword = false;
# Nix
nix.gc.automatic = true;
nix.gc.options = "--delete-older-than 30d";
# https://nixos.wiki/wiki/FAQ/When_do_I_update_stateVersion
system.stateVersion = "20.03";
}
Once you have a valid configuration in /etc/nixos/configuration.nix
, run nixos-rebuild switch
as root
, and optionally run nix-collect-garbage -d
to remove all the leftover stuff from the
installation that is not required.
For the Pi 3, see also this excellent blog post which has step-by-step instructions for the whole process.
For the Pi 4, you might want to check these amazing instructions written by @chrisanthropic. If you're running out of space on your firmware partition, this Gist also includes instructions on how to make an image with a bigger one.
- If the execution fails due to missing permissions, sorry -- you need to be able to run containers
with the
--privileged
Docker flag. - Ensure you have enough memory/swap and disk space. This can require up to 8 GiB of RAM and ~6-7 GiB of disk space.
- If the build fails with
cptofs
related errors or something like:This is a known issue (see [1] and [2]). Please editResizing to minimum allowed size resize2fs 1.45.5 (07-Jan-2020) Please run 'e2fsck -f temp.img' first.
docker-compose.yml
, uncommentAPPLY_CPTOFS_PATCH
and rerun with./run.sh up --build
. If you want to learn more, I investigated this issue and wrote about it. - If you are running Docker Toolbox on Windows, you might encounter weird "file not found" errors
when Nix attempts to find your configuration files. This is due to the fact that Docker Toolbox
uses VirtualBox to run Docker and
C:\Users
is the only directory shared by default -- thus, if you're storing your files in any other path you might run into the issue. Follow the instructions detailed in this great post for ways to solve this. Thanks @dsferruzza! - Failing commands like
bsdtar: Error opening archive: Can't initialize filter; unable to run program "zstd -d -qq"
might be due a preexisting alpine image. Delete it and run the script again. - For any other problem, open an issue or email me!
To build an SD image for a foreign architecture, NixOS requires that the host system is able to run
executables for the target architecture. Most people though don't have a powerful ARM64v8 machine at
their disposal to do that, which is the reason why I have made this. Plus, containers reduce the
friction of the entire process to zero. Feel free to check out docker-compose.yml
, the
documentation should (hopefully) be clear.
Here's how it works in detail:
- When needed, QEMU and
binfmt_misc
are used to emulate AArch64 and to allow the host kernel to understand and run AArch64 binaries. To limit the risk of security issues, the build process itself runs on an unprivileged container -- the containers that deal with QEMU andbinfmt_misc
are separate and do not interact with the build process or untrusted binaries. - When running
docker-compose up
, here's what happens:- if emulation is required, the first image to be built is
setup-qemu
, which will:- download a pinned version of QEMU from the Debian archives, required for proper emulation. At the time of writing, the downloaded version is 5.0.
- verify the integrity of the downloaded binaries with an hardcoded hash.
- extract
qemu-aarch64-static
from the package.
- then, Docker builds
build-nixos
, which will:- create an unprivileged user for the NixOS build.
- download and bootstrap Nix with the default configuration.
- download the specified version/checkout of
nixpkgs
. By default, this downloadsnixpkgs
for the current stable version (20.03). - prepare an environment file which adds Nix to
$PATH
, setsNIX_HOME
and sets a trap which notifies the cleanup container using TCP when the build is done.
- once all the images are built, if emulation is required,
setup-qemu
runs (with privileges), and it will:- check if a
binfmt_misc
entry which has the same interpreter name exists (qemu-aarch64-docker-nixos
), removing it if so - register
qemu-aarch64-bin
as abinfmt_misc
handler for AArch64 with thefix-binary
flag, which allowsbinfmt_misc
to keep working when the container is destroyed
- check if a
build-nixos
will be started concurrently (without privileges), and it will:- wait until the system is able to understand and execute AArch64 binaries
- bootstrap the environment
- build the image
- copy the image to
/build
asroot
(shared volume) - notify
cleanup-qemu
via a simplenc
call
- last but not least, if emulation is required,
cleanup-qemu
will also be started concurrently (with privileges), and it will:- listen on TCP port
11111
and wait untilbuild-nixos
connects and unlocks the process - after that happens, it will remove
binfmt_misc
handlers that start withqemu
and leave the system clean
- listen on TCP port
- if emulation is required, the first image to be built is
- Use a unique name as the
binfmt_misc
handler so that it's not needed to nuke the other pre-existing QEMU handlers on the system. - Use a custom script to register QEMU as a
binfmt_misc
handler instead of patching the original one. - Support native
aarch64
compilation