A Tutorial Introduction to Nix

Brief introduction to a nix based project workflow.

Background

For CarpentryCon@Home 2020, along with Amrita Goswami, I am to prepare and deliver a workshop on “Reproducible Environments with the Nix Packaging System”. In particular, as a community of practice lesson, the focus is not on packaging (as is typical of most Nix tutorials) nor on the Nix expression language itself, but instead on the use of Nix as a replacement for virtual environments using mkShell.

Materials

This is a Carpentries style single page lesson on setting up and working with Nix for reproducible environments. It was concieved to be a complimentary resource to the content of this repository, namely:

• Slides on Python packages with Nix
• Nix with R and devtools
• Statistical Rethinking and Nix
• An Etherpad
• Session recording

Ten seconds into Nix

A few words to keep in mind, in no particular order.

• Nix is based of good academic principles by Dolstra, de Jonge, and Visser (2004) and Dolstra, Löh, and Pierron (2010)
  • It has been used in large scientific projects for reproducibility (e.g. d-SEAMS of Goswami, Goswami, and Singh (2020))
• The Nix expression language is a domain specific language
  • Turing completeness is not a goal or a requirement
• Can leverage binary caches
  • Not always true, only when installed in /nix

Setup

For this particular tutorial, we will assume the standard Nix installation proceedure, that is, one where the installer has root access to create the initial /nix directory and set up the build users¹. This follows directly from the Nix Manual:

1# You need root permissions for this!!!
2sh <(curl -L https://nixos.org/nix/install) --daemon

At this point we will also install the canonnical first package, the hello package, which simply outputs a friendly greeting.

1nix-env -i hello

Note that the basic package search operation is nix search and it gives outputs which look like:

Though this is not bad by any standard, we will try to get a more interactive management tool.

Basic Helpers

The first few things to obtain are:

nox

This is a better package management helper

niv

For pinning dependencies as discussed later

lorri

For working seamlessly with project environments

Exercise 1

Try installing these and use nox emacs to test the output

More Dependable Dependencies

Standard Channels

Nix works by searching a repository (local or online) of package derivations. Indeed, we can pass nix-env any local fork of the main nixpkgs repo as well.

1# don't run this, it is a large repo
2git clone https://github.com/NixOS/nixpkgs.git $mynixdir
3# make changes..
4$EDITOR $nixpkgs/pkgs/applications/editors/emacs/default.nix
5nix-env -i emacs -f $nixpkgs

• This might serve as a way to mass modify the nixpkgs in a pinch
  • However, we will almost never use this in practice
  • The overlay approach is much better
• It is also useful if we need to build local derivations

Pinning Dependencies

Compared to globally tracking the branches of nixpkgs or even local changes and forks, for project oriented workflows it is better to use niv which we obtained previously. In a nutshell, niv will generate a json file to keep track of dependencies and wraps it in a nix file we can subsequently import and use.

Project Setup

We are now in a position to start working with a project oriented workflow.

1# Make directories
2mkdir myFirstNix
3cd myFirstNix
4# Setup
5niv init
6niv update nixpkgs -b nixpkgs-unstable

At this stage your project should have the following structure:

1tree myFirstNix

We can now move on to the heart of this tutorial, the nix-shell. In a nutshell, running nix-shell when there is a defined shell.nix will spawn a virtual environment with the nix packages requested.

lorri and direnv

Though we haven’t as yet generated a shell.nix we should point out that writing one by hand will mean that we need to rebuild the enviroment when we make changes using nix-shell every time. A more elegant approach is to offload the rebuilding of the environment to lorri which also has a neat direnv integrration. Let’s try that out.

1cd myFirstNix
2lorri init

At this point we should now have:

1tree -a myFirstNix

We might want to take a quick look at what is being loaded into the environment and the shell.nix at this point.

1let
2  pkgs = import <nixpkgs> {};
3in
4pkgs.mkShell {
5  buildInputs = [
6    pkgs.hello
7  ];
8}

1eval "$(lorri direnv)"

The .envrc output is not very useful at a glance, however when we cd into the directory it is very verbose and explicit about what is being set up.

Note that in order to set lorri up, we will need to set up a daemon.

1systemctl --user start lorri
2direnv allow

Pinning with niv

Note that inspite of having set up niv, we have not yet used the sources defined therein. We will now fix this, by modifying shell.nix.

 1let
 2  sources = import ./nix/sources.nix;
 3  pkgs = import sources.nixpkgs { };
 4  inherit (pkgs.lib) optional optionals;
 5in
 6pkgs.mkShell {
 7  buildInputs = [
 8    pkgs.hello
 9  ];
10}

Purity and Environments

There are a couple of things to note about this setup.

• The default shell is bash
• On occasion, depending on your Dotfiles you might have paths overriden in an annoying way

One workaround is to use nix shell with an argument:

1nix-shell --run "bash"

• We can also pass --pure to the function, but at the cost of having to define many more dependencies for our shell

mkShell

The mkShell function is the focus of our tutorial, and we will mostly work around passing in different environments and hooks. Let us start by defining a hook.

Shell Hooks

Often, we will want to set an environment variable in our shell in advance. We should not use direnv for this, and instead we will focus on the shellHook option. Syntactically, we note that this is of the form:

1let
2  hook = ''
3  export myvar="Test"
4  ''
5in pkgs.mkShell {
6  shellHook = hook;
7}

Often we will describe variables in the let section in favor of cluttering the actual function call itself.

Overriding Global Packages

For overriding global packages, it is best to leverage the config.nix (which is commonly in $HOME/.config/nixpkgs/config.nix) file instead of the current environment, though it could be managed in a per-project setup as well. Consider the case where we need to disable tests for a particular packages, say libuv.

1{
2  packageOverrides = pkgs:
3    with pkgs; {
4      libuv = libuv.overrideAttrs (oldAttrs: {
5        doCheck = false;
6        doInstallCheck = false;
7      });
8    };
9}

Python Dependencies

As R dependency management has been covered in an earlier post, we will focus on the management of python environments.

Generic Environments

We can define existing packages as follows (and can check for existence with nox) using the let..in syntax.

 1let
 2  # Niv
 3  sources = import ./nix/sources.nix;
 4  pkgs = import sources.nixpkgs { };
 5  inherit (pkgs.lib) optional optionals;
 6  # Python
 7  pythonEnv = pkgs.python38.withPackages (ps: with ps;[
 8    numpy
 9    toolz
10  ]);
11in pkgs.mkShell {
12  buildInputs = with pkgs; [
13    pythonEnv
14
15    black
16    mypy
17
18    libffi
19    openssl
20  ];
21}

Project Local Pip

We can leverage a trick from here to set a local directory for pip installations, which boils down to some path hacking.

 1let
 2  # Niv
 3  sources = import ./nix/sources.nix;
 4  pkgs = import sources.nixpkgs { };
 5  inherit (pkgs.lib) optional optionals;
 6  # Python
 7  pythonEnv = pkgs.python38.withPackages (ps: with ps;[
 8    numpy
 9    toolz
10  ]);
11  hook = ''
12     export PIP_PREFIX="$(pwd)/_build/pip_packages"
13     export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.8/site-packages:$PYTHONPATH"
14     export PATH="$PIP_PREFIX/bin:$PATH"
15     unset SOURCE_DATE_EPOCH
16  '';
17in pkgs.mkShell {
18  buildInputs = with pkgs; [
19    pythonEnv
20
21    black
22    mypy
23
24    libffi
25    openssl
26  ];
27  shellHook = hook;
28}

Note that this is discouraged as we will lose the caching capabilities of nix.

Non-Standard Python

For more control over the environment, we can define it in more detail with some overlays.

 1let
 2  python = pkgs.python38.override {
 3    packageOverrides = self: super: {
 4      pytest = super.pytest.overridePythonAttrs (old: rec {
 5        doCheck = false;
 6        doInstallCheck = false;
 7      });
 8    };
 9  };
10  myPy = python.withPackages
11    (p: with p; [ numpy pip pytest ]);
12in pkgs.mkShell {
13  buildInputs = with pkgs; [
14    myPy
15  ];
16}

We have used both overriden packages and standard packages in the above formulation.

Building Packages

For cases where we are certain that no existing package is present (use nox) we can also build them. Take f90wrap as an example, and we will use the Github version, rather than the PyPi version (the difference is in the source fetch function).

 1f90wrap = self.buildPythonPackage rec {
 2  pname = "f90wrap";
 3  version = "0.2.3";
 4  src = pkgs.fetchFromGitHub {
 5    owner = "jameskermode";
 6    repo = "f90wrap";
 7    rev = "master";
 8    sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23";
 9  };
10  buildInputs = with pkgs; [ gfortran stdenv ];
11  propagatedBuildInputs = with self; [
12    setuptools
13    setuptools-git
14    wheel
15    numpy
16  ];
17  preConfigure = ''
18    export F90=${pkgs.gfortran}/bin/gfortran
19  '';
20  doCheck = false;
21  doIstallCheck = false;
22};

This is quite involved, discuss.

Setting Versions

We can finally generalize our shell.nix to default to python 3.8 but also take a command through --argstr:

1nix-shell --argstr pythonVersion 36 --run "bash"

Where we need to simply define the option at the top of the file, with a default.

 1{ pythonVersion ? "38" }:
 2# Define
 3let
 4  sources = import ./nix/sources.nix;
 5  pkgs = import sources.nixpkgs { };
 6  inherit (pkgs.lib) optional optionals;
 7  hook = ''
 8    # Python Stuff
 9     export PIP_PREFIX="$(pwd)/_build/pip_packages"
10     export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.8/site-packages:$PYTHONPATH"
11     export PATH="$PIP_PREFIX/bin:$PATH"
12     unset SOURCE_DATE_EPOCH
13  '';
14  # Apparently pip needs 1980 or above
15  # https://github.com/ento/elm-doc/blob/master/shell.nix
16  python = pkgs."python${pythonVersion}".override {
17    packageOverrides = self: super: {
18      pytest = super.pytest.overridePythonAttrs (old: rec {
19        doCheck = false;
20        doInstallCheck = false;
21      });
22      ase = super.ase.overridePythonAttrs (old: rec {
23        doCheck = false;
24        doInstallCheck = false;
25      });
26      f90wrap = self.buildPythonPackage rec {
27        pname = "f90wrap";
28        version = "0.2.3";
29        src = pkgs.fetchFromGitHub {
30          owner = "jameskermode";
31          repo = "f90wrap";
32          rev = "master";
33          sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23";
34        };
35        buildInputs = with pkgs; [ gfortran stdenv ];
36        propagatedBuildInputs = with self; [
37          setuptools
38          setuptools-git
39          wheel
40          numpy
41        ];
42        preConfigure = ''
43          export F90=${pkgs.gfortran}/bin/gfortran
44        '';
45        doCheck = false;
46        doInstallCheck = false;
47      };
48    };
49  };
50  myPy = python.withPackages
51    (p: with p; [ ase ipython ipykernel scipy numpy f90wrap pip ]);
52in pkgs.mkShell {
53  buildInputs = with pkgs; [
54    # Required for the shell
55    zsh
56    perl
57    git
58    direnv
59    fzf
60    ag
61    fd
62
63    # Building thigns
64    gcc9
65    gfortran
66    openblas
67
68    myPy
69    # https://github.com/sveitser/i-am-emotion/blob/294971493a8822940a153ba1bf211bad3ae396e6/gpt2/shell.nix
70  ];
71  shellHook = hook;
72}

This is enough to cover almost all use-cases for python environments.

Build Helpers

Note that we can speed up some aspects of fetch with the prefetch commands:

1nix-prefetch-git $giturl
2nix-prefetch-url $url

In practice, some trial and error is easier.

Supplementary Reading Material

Though these are in no means exhaustive, they may offer a slightly more advanced or different focus than the material covered here.

Core Content

• Manuals
  • Nix and Nixpkgs
• Nix Wiki
  • Nix Cheatsheet
• Language Sections

Learning Paths

• Nix pills
• Official tutorials
• Nix dev has some nice opinionated tips

Personal Correspondence

Tyson Whitehead from Compute Canada was kind enough to bring the folllowing additional training materials:

• A wiki pertaining to usage of Nix on in an HPC setting
• SWC style workshop materials from TECC 2018
• SHARCNET live presentation materials from 2018

Conclusions

The standard dive into Nix is based on building derivations and playing with language, which is in no means a bad one, just too long for the time allocated. The best way to get into Nix is to start using it for everything.

References