NixOS on Brightbox

NixOS is a Linux distribution that is configured using a functional language in a declarative way.

I’ve been using it here at Brightbox as my main development machine for several months now and whilst it has some rough edges, it has some compelling features to look at.

It is possible to use it on Brightbox but there is a bit of work involved.

This post is for those who are curious about NixOS and want a brief introduction and for those who already use NixOS but didn’t realise how to get it running on Brightbox.

Nix

Nix is a package manager for Linux, OS X and other Unix systems. It uses a functional language to create reproducible, isolated versions of packages that do not change once built.

As many of the build inputs as possible are fixed so that the exact same piece of software can be rebuilt given the same parameters. This also allows using prebuilt binary packages from trustworthy sources.

It uses symlinks to perform atomic upgrades and rollbacks in a similar way to Capistrano deployments. The symlinks are managed by Nix and allow you to have multiple profiles, each with a different view of the system and it’s software.

Due to this isolation, you can run multiple versions of libraries, applications or services on one system alongside each other without having to resort to using containers.

Think of it as a system wide chruby where you can run nix-shell and be in Bash with the exact libraries and version of languages you need to develop and test.

NixOS

NixOS is a Linux distribution that builds on Nix that allows you to declare what the desired state of the system should be. Unlike something like Puppet or Chef, the system itself can do this.

As well as declaring which packages you want available, one or more config files can specify which services to run, which kernel modules to blacklist, containers to run, disks to mount and some other 3000 documented options.

It also includes the same atomic upgrades and rollbacks that nix does but on a system level. You can rebuild a system and it will causes any altered services to restart and they pick up new version of servers and configs immediately.

When you make a change to the system, it can even automatically create a new GRUB entry for it. So you can switch back to earlier generations in case something went wrong.

Preparing NixOS for Brightbox

Thanks to Rob Vermass and other members of the community, there has been the means to build an image for Brightbox for the last year (since version 15.09).

Using the nix-build tool NixOS (or any Linux system with Nix) you can build a QCOW2 image based on a NixOS configuration.

You’ll need access to a Linux machine with Nix installed for this. So this means there’s a bit of bootstrapping.

Whilst OS X supports Nix, the x86_64-darwin platform can not build the images you need to upload.

Installing Nix

You can either install Nix on an existing Linux system or create a Cloud Server to use.

You can install Nix following their installation guide. That should give you the nix-build command and a copy of Nixpkgs that you need.

Building a NixOS image

The command is a bit more complicated than it should be. You have to tell it the following things:

• Where the Nix packages for NixOS are located
• Where the NixOS configuration file is located
• The build command you want to run: config.system.build.brightboxImage
• The system architecture/platform you’d like to build: x86_64-linux

We can set the configuration file a number of ways. To keep things simple we’ll use the NIXOS_CONFIG environment setting.

We’ll start simple and try to build that as is.

$ export NIXOS_CONFIG=~/.nix-defexpr/channels/nixpkgs/nixos/modules/virtualisation/brightbox-config.nix
$ nix-build ~/.nix-defexpr/channels/nixpkgs/nixos \
      -A config.system.build.brightboxImage \
      --argstr system x86_64-linux

The build command will create an result/nixos.qcow2 image in the working directory. This can be uploaded and registered in your account’s image library and used to build new servers.

Using the system

Create a cloud server using your new image. If you don’t have IPv6, then map a Cloud IP so you can access it and you can log in as root. It should automatically pick up the SSH keys registered to the account using cloud-init.

You can now use NixOS as normal.

$ ssh root@public.srv-d3fs2.gb1.brightbox.com

[root@srv-d3fs2:~]# nixos-version
16.09pre88945.5120af0 (Flounder)

[root@srv-fbppf:~]# cat /etc/nixos/configuration.nix
{ config, pkgs, modulesPath, ... }:

{
  imports = [ "${modulesPath}/virtualisation/brightbox-image.nix" ];
}

/etc/nixos/configuration.nix is the file that defines the NixOS system. The default file inherits the brightbox-image module which in turn inherits various settings for a headless server running under virtualisation.

After the image is build, there is one more job to do. Nix uses Nixpkgs, which is the collection of definitions for packages and a NixOS system.

At the moment, it is not yet connected to a “channel” which is where the latest version of these are downloaded from.

[root@srv-fbppf:~]# nixos-rebuild switch --upgrade

This downloads the latest version of Nixpkgs and switches the system to be using it. It also sets up the environment to allow future rebuilds to find the packages without needing extra command line args.

Setting up a full system

NixOS has a substantial list of configurable options you can tweak to set up a system.

Be aware however that the online documentation runs close to the latest version so some setting may not be available unless you are using your own checkout of Nixpkgs on the master branch.

So let’s set up a webserver. Update /etc/nixos/configuration.nix to be:

{ config, pkgs, modulesPath, ... }:

{
  imports = [ "${modulesPath}/virtualisation/brightbox-image.nix" ];
  
  networking.firewall.allowedTCPPorts = [ 80 ];
  
  services = {
    lighttpd = {
      enable = true;
    };
  };
}

We need something to serve from the server so lets create a file and tell NixOS to switch to the new configuration.

[root@srv-fbppf:~]# mkdir -p /srv/www
[root@srv-fbppf:~]# echo hello >> /srv/www/index.html
[root@srv-fbppf:~]# nixos-rebuild switch
building Nix...
building the system configuration...fyppn6nyr387c6mcrww44mzjd-vconsole.conf’
these derivations will be built:v3s33drxsazwagdqm59ygg3923yzp-vlock.pam’
  /nix/store/y7sdfs7cipqw2pz1f7swanmkfabdnpc1-system-path.drv-unit-systemd-vcon  /nix/store/zbqqi20hda0q9mzwvbx1d1r2m04rhm1g-dbus-conf.drv
  /nix/store/2grckp2wy4m482s7ls49s86a5vbxmnjw-unit-dbus.service.drv.pam’
  /nix/store/2s2lgv7apxic7yb425syhwkxx3xphi7l-unit-polkit.service.drvunits’
  /nix/store/9x620l3rkfmhh87w259xnnpf4hvassb5-firewall-start.drvcreensaver.pam’
  /nix/store/2vr12m0jm5w0z5ygpl5f4lkl8ydv4vmv-firewall-reload.drv’
  /nix/store/5r1ami1qd7w9c0lj5b9xdy6ygplls2gh-lighttpd.conf.drvixos-system-unna  /nix/store/9dfkzr76m3m7pjn1lf23zmzkjm0aw4id-unit-lighttpd.service.drv
  /nix/store/zbxs8557qck12ws3is2m4lvs2lijmfds-unit-firewall.service.drv
  /nix/store/sii9s784hgwhg900ik1aly5f6414cxjp-system-units.drvrvice, kmod-stati  /nix/store/j4krbskh09svlnxmc3msbg7qjz78kasn-etc.drvork-setup.service, nginx.s  /nix/store/xzxp08dv516mvig2hx09bh2y8fpcvsmy-users-groups.json.drvrnald.servic  /nix/store/kvqzlki1ys3x6iq1cf0j79pxn0zq5wv8-nixos-system-unnamed-16.09pre89188.f25006b.drvmd-tmpfiles-setup-dev.service, systemd-udev-trigger.service, systethese paths will be fetched (0.22 MiB download, 0.90 MiB unpacked):
  /nix/store/hzdykwhy13gj9qs7b54c79c7scmfjfcq-lighttpd-1.4.40ivated by:
building path(s) ‘/nix/store/y9gpy3a3113zjd8a9g25sn1ydgv8labw-users-groups.json’arning: Stopping systemd-journald.service, but it can still be activated by:
fetching path ‘/nix/store/hzdykwhy13gj9qs7b54c79c7scmfjfcq-lighttpd-1.4.40’...
  systemd-journald-dev-log.socket
*** Downloading ‘https://cache.nixos.org/nar/02mfxl8bbmzhj8dra2mk1iah3hdjbsca8gd15iq66dpa1vpwlzr9.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/hzdykwhy13gj9qs7b54c79c7scmfjfcq-lighttpd-1.4.40’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speedst100  227k  100  227k    0     0  1749k      0 --:--:-- --:--:-- --:--:-- 1767krvice, systemd-update-utmp.service, systemd-user-sessions.service, user@0.servicbuilding path(s) ‘/nix/store/xi1kmhv7din74vgq418rv6wpwkw93h5r-firewall-start’
building path(s) ‘/nix/store/g1i15pbc7vbmrrfxlzfmn9r6i1s9p9c0-lighttpd.conf’
building path(s) ‘/nix/store/kkpw534nh89j2lsvcfivyad8amp0lcb8-firewall-reload’
building path(s) ‘/nix/store/hlhs4gbfi3xv9gqcvbhya7ja1p8rg1z4-system-path’
created 3009 symlinks in user environment1/conf.d/60-nixos-generic-alias.conf’.install-info: warning: no info dir entry in `/nix/store/hlhs4gbfi3xv9gqcvbhya7ja1p8rg1z4-system-path/share/info/time.info'conf.d/99-user.conf’...
building path(s) ‘/nix/store/1fg4wg50a7mnc762jsyylhxjmx5gfcdr-unit-firewall.service’ing group ‘nginx’
building path(s) ‘/nix/store/c6fdpwj7arg5wgvsjqvr6xfsskmb9bs6-dbus-conf’
building path(s) ‘/nix/store/5kjpb33vcwdcpa160n320qk02mfl6mzl-unit-lighttpd.service’ding the following units: dbus.service, dev-hugepages.mount, dev-mqueue.mobuilding path(s) ‘/nix/store/2d3kfvbg47p1gh267fs0ja9ik3iqp4in-unit-dbus.service’estarting the following units: dhcpcd.service, sshd.service, systemd-logind.sebuilding path(s) ‘/nix/store/lng22apqwim953gbx5wapfb3n3vcgr94-unit-polkit.service’rting the following units: audit.service, fetch-ec2-data.service, kmod-statibuilding path(s) ‘/nix/store/7lhfzxndnddh6p1sj6d5w2mw0ks522g7-system-units’-daebuilding path(s) ‘/nix/store/rac1l547i4dqyrxg343n357rkgrz8pny-etc’ systemd-modubuilding path(s) ‘/nix/store/q38ic52y324ypk0lb98sv6n0fii01n9x-nixos-system-unnamed-16.09pre89188.f25006b’e, systemd-udev-trigger.service, systemd-udevd.servicupdating GRUB 2 menu...up.service
activating the configuration...
setting up /etc...
reloading the following units: dbus.service, firewall.service
the following new units were started: lighttpd.service

Now each change in the system has created derivations which are exact instructions in how to create a package or file. It then builds theses packages and versioned file of that part of the configuration. lighttpd itself has already been built so it is downloaded from the trusted NixOS sources.

The new state of the system is prepared and then when the symlinks are setup and switched the services that have been touched are restarted.

You can visit the Cloud IP of the server and you should see “hello” or whatever you put in /srv/www/index.html.

Rolling back

Wait, we didn’t want a web server.

[root@srv-fbppf:~]# nixos-rebuild switch --rollback
switching from generation 3 to 2
updating GRUB 2 menu...
stopping the following units: lighttpd.service
activating the configuration...
setting up /etc...
removing group ‘lighttpd’
removing user ‘lighttpd’
reloading the following units: dbus.service, firewall.service

This switches back to the 2nd generation of our system. The 1st generation was the one before we first ran nixos-rebuild switch --update fresh from our uploaded image. The 2nd generation had the updates.

Switching back didn’t involved any downloads or rebuilding since the Nix store kept the old version of all the files and packages. The rollback was pretty managed by shutting off lighttpd, changing a symlink and restarting.

Try the web page again and the firewall will be blocking you, and there isn’t a web server listening anymore anyway.

Now next time we try to rebuild it will try to recreate the system in /etc/nixos/configuration.nix which will turn the webserver back on.

However /srv/www/index.html still exists on the system. So there there is a mix of the declarative system and still some “imperative” parts to be aware of.

Summary

NixOS is trying something different in the Linux distribution space and it’s not going to be suitable for everyone.

The documentation and UX of the tools is a bit rough and is being worked on by the community. The Nix language itself has a learning curve as well.

If you have to manage your own development machine then the ability to pull in software, into an isolated shell and not pollute your normal shell is pleasing different to a world of homebrew, RVM and most package managers.

If a system update goes bad, most times a rollback can fix it. Including kernel panics during startup.

Having a local checkout of Nixpkgs allows developers to use tools like git-bisect to find where a problem was introduced to a system. You can rebase you customisations, test them and move on without waiting for a PR to be reviewed and accepted.

It’s certainly worth having a look at and if you are already a NixOS user, now you can try running it on Brightbox as well.

Try NixOS on Brightbox

You can sign up for Brightbox in just a couple of minutes and use your £50 free credit to play with NixOS on an SSD Cloud Server.