11 minutes
Written: 2020-08-18 16:18 +0000
Updated: 2024-08-06 00:52 +0000
A Tutorial Introduction to Nix
Brief introduction to a nix based project workflow.
Background
For CarpentryCon@Home 2020, along with
Amrita
Goswami,
I am to prepare and deliver a workshop on “Reproducible Environments
with the Nix Packaging
System”. In
particular, as a community of practice lesson, the focus is not on
packaging (as is typical of most Nix tutorials) nor on the Nix
expression language itself, but instead on the use of Nix as a
replacement for virtual environments using mkShell
.
Materials
This is a Carpentries style single page lesson on setting up and working with Nix for reproducible environments. It was concieved to be a complimentary resource to the content of this repository, namely:
- Slides on Python packages with Nix
- Nix with R and devtools
- Statistical Rethinking and Nix
- An Etherpad
- Session recording
Ten seconds into Nix
A few words to keep in mind, in no particular order.
- Nix is based of good academic principles by Dolstra, de Jonge, and
Visser (2004) and Dolstra, Löh, and Pierron (2010)
- It has been used in large scientific projects for reproducibility (e.g. d-SEAMS of Goswami, Goswami, and Singh (2020))
- The Nix expression language is a domain specific language
- Turing completeness is not a goal or a requirement
- Can leverage binary caches
- Not always true, only when
installed in
/nix
- Not always true, only when
installed in
Setup
For this particular tutorial, we will assume the standard Nix
installation proceedure, that is, one where the installer has root
access to create the initial /nix
directory and set up the build
users1. This follows directly from the Nix
Manual:
1# You need root permissions for this!!!
2sh <(curl -L https://nixos.org/nix/install) --daemon
At this point we will also install the canonnical first package, the
hello
package, which simply outputs a friendly greeting.
1nix-env -i hello
Note that the basic package search operation is nix search
and it
gives outputs which look like:
Though this is not bad by any standard, we will try to get a more interactive management tool.
Basic Helpers
The first few things to obtain are:
- nox
- This is a better package management helper
- niv
- For pinning dependencies as discussed later
- lorri
- For working seamlessly with project environments
Exercise 1
Try installing these and use
nox emacs
to test the output
More Dependable Dependencies
Standard Channels
Nix works by searching a repository (local or online) of package
derivations. Indeed, we can pass nix-env
any local fork of the main
nixpkgs repo as well.
1# don't run this, it is a large repo
2git clone https://github.com/NixOS/nixpkgs.git $mynixdir
3# make changes..
4$EDITOR $nixpkgs/pkgs/applications/editors/emacs/default.nix
5nix-env -i emacs -f $nixpkgs
- This might serve as a way to mass modify the
nixpkgs
in a pinch- However, we will almost never use this in practice
- The overlay approach is much better
- It is also useful if we need to build local derivations
Pinning Dependencies
Compared to globally tracking the branches of nixpkgs
or even local
changes and forks, for project oriented workflows it is better to use
niv
which we obtained previously. In a nutshell, niv
will generate a
json
file to keep track of dependencies and wraps it in a nix
file
we can subsequently import and use.
Project Setup
We are now in a position to start working with a project oriented workflow.
1# Make directories
2mkdir myFirstNix
3cd myFirstNix
4# Setup
5niv init
6niv update nixpkgs -b nixpkgs-unstable
At this stage your project should have the following structure:
1tree myFirstNix
myFirstNix | |||
---|---|---|---|
└── | nix | ||
├── | sources.json | ||
└── | sources.nix | ||
1 | directory, | 2 | files |
We can now move on to the heart of this tutorial, the nix-shell
. In a
nutshell, running nix-shell
when there is a defined shell.nix
will
spawn a virtual environment with the nix packages requested.
lorri and direnv
Though we haven’t as yet generated a shell.nix
we should point out
that writing one by hand will mean that we need to rebuild the
enviroment when we make changes using nix-shell
every time. A more
elegant approach is to offload the rebuilding of the environment to
lorri
which also has a neat direnv
integrration. Let’s try that out.
1cd myFirstNix
2lorri init
Aug | 18 | 15:24:06.524 | INFO | wrote | file, | path: | ./shell.nix |
---|---|---|---|---|---|---|---|
Aug | 18 | 15:24:06.524 | INFO | wrote | file, | path: | ./.envrc |
Aug | 18 | 15:24:06.524 | INFO | done |
At this point we should now have:
1tree -a myFirstNix
myFirstNix | |||
---|---|---|---|
├── | .envrc | ||
├── | nix | ||
│ | ├── | sources.json | |
│ | └── | sources.nix | |
└── | shell.nix | ||
1 | directory, | 4 | files |
We might want to take a quick look at what is being loaded into the
environment and the shell.nix
at this point.
1let
2 pkgs = import <nixpkgs> {};
3in
4pkgs.mkShell {
5 buildInputs = [
6 pkgs.hello
7 ];
8}
1eval "$(lorri direnv)"
The .envrc
output is not very useful at a glance, however when we cd
into the directory it is very verbose and explicit about what is being
set up.
Note that in order to set lorri
up, we will need to set up a daemon.
1systemctl --user start lorri
2direnv allow
Pinning with niv
Note that inspite of having set up niv
, we have not yet used the
sources defined therein. We will now fix this, by modifying shell.nix
.
1let
2 sources = import ./nix/sources.nix;
3 pkgs = import sources.nixpkgs { };
4 inherit (pkgs.lib) optional optionals;
5in
6pkgs.mkShell {
7 buildInputs = [
8 pkgs.hello
9 ];
10}
Purity and Environments
There are a couple of things to note about this setup.
- The default shell is
bash
- On occasion, depending on your Dotfiles you might have paths overriden in an annoying way
One workaround is to use nix shell with an argument:
1nix-shell --run "bash"
- We can also pass
--pure
to the function, but at the cost of having to define many more dependencies for our shell
mkShell
The mkShell
function is the focus of our tutorial, and we will mostly
work around passing in different environments and hooks. Let us start by
defining a hook.
Shell Hooks
Often, we will want to set an environment variable in our shell in
advance. We should not use direnv
for this, and instead we will focus
on the shellHook
option. Syntactically, we note that this is of the
form:
1let
2 hook = ''
3 export myvar="Test"
4 ''
5in pkgs.mkShell {
6 shellHook = hook;
7}
Often we will describe variables in the let section in favor of cluttering the actual function call itself.
Overriding Global Packages
For overriding global packages, it is best to leverage the config.nix
(which is commonly in $HOME/.config/nixpkgs/config.nix
) file instead
of the current environment, though it could be managed in a per-project
setup as well. Consider the case where we need to disable tests for a
particular packages, say libuv
.
1{
2 packageOverrides = pkgs:
3 with pkgs; {
4 libuv = libuv.overrideAttrs (oldAttrs: {
5 doCheck = false;
6 doInstallCheck = false;
7 });
8 };
9}
Python Dependencies
As R dependency management has been covered in an earlier post, we will focus on the management of
python
environments.
Generic Environments
We can define existing packages as follows (and can check for existence
with nox
) using the let..in
syntax.
1let
2 # Niv
3 sources = import ./nix/sources.nix;
4 pkgs = import sources.nixpkgs { };
5 inherit (pkgs.lib) optional optionals;
6 # Python
7 pythonEnv = pkgs.python38.withPackages (ps: with ps;[
8 numpy
9 toolz
10 ]);
11in pkgs.mkShell {
12 buildInputs = with pkgs; [
13 pythonEnv
14
15 black
16 mypy
17
18 libffi
19 openssl
20 ];
21}
Project Local Pip
We can leverage a trick from
here
to set a local directory for pip
installations, which boils down to
some path hacking.
1let
2 # Niv
3 sources = import ./nix/sources.nix;
4 pkgs = import sources.nixpkgs { };
5 inherit (pkgs.lib) optional optionals;
6 # Python
7 pythonEnv = pkgs.python38.withPackages (ps: with ps;[
8 numpy
9 toolz
10 ]);
11 hook = ''
12 export PIP_PREFIX="$(pwd)/_build/pip_packages"
13 export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.8/site-packages:$PYTHONPATH"
14 export PATH="$PIP_PREFIX/bin:$PATH"
15 unset SOURCE_DATE_EPOCH
16 '';
17in pkgs.mkShell {
18 buildInputs = with pkgs; [
19 pythonEnv
20
21 black
22 mypy
23
24 libffi
25 openssl
26 ];
27 shellHook = hook;
28}
Note that this is discouraged as we will lose the caching capabilities of nix.
Non-Standard Python
For more control over the environment, we can define it in more detail with some overlays.
1let
2 python = pkgs.python38.override {
3 packageOverrides = self: super: {
4 pytest = super.pytest.overridePythonAttrs (old: rec {
5 doCheck = false;
6 doInstallCheck = false;
7 });
8 };
9 };
10 myPy = python.withPackages
11 (p: with p; [ numpy pip pytest ]);
12in pkgs.mkShell {
13 buildInputs = with pkgs; [
14 myPy
15 ];
16}
We have used both overriden packages and standard packages in the above formulation.
Building Packages
For cases where we are certain that no existing package is present (use
nox
) we can also build them. Take f90wrap
as an example, and we will
use the Github version, rather than the PyPi version (the difference is
in the source fetch function).
1f90wrap = self.buildPythonPackage rec {
2 pname = "f90wrap";
3 version = "0.2.3";
4 src = pkgs.fetchFromGitHub {
5 owner = "jameskermode";
6 repo = "f90wrap";
7 rev = "master";
8 sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23";
9 };
10 buildInputs = with pkgs; [ gfortran stdenv ];
11 propagatedBuildInputs = with self; [
12 setuptools
13 setuptools-git
14 wheel
15 numpy
16 ];
17 preConfigure = ''
18 export F90=${pkgs.gfortran}/bin/gfortran
19 '';
20 doCheck = false;
21 doIstallCheck = false;
22};
This is quite involved, discuss.
Setting Versions
We can finally generalize our shell.nix
to default to python 3.8
but
also take a command through --argstr
:
1nix-shell --argstr pythonVersion 36 --run "bash"
Where we need to simply define the option at the top of the file, with a default.
1{ pythonVersion ? "38" }:
2# Define
3let
4 sources = import ./nix/sources.nix;
5 pkgs = import sources.nixpkgs { };
6 inherit (pkgs.lib) optional optionals;
7 hook = ''
8 # Python Stuff
9 export PIP_PREFIX="$(pwd)/_build/pip_packages"
10 export PYTHONPATH="$(pwd)/_build/pip_packages/lib/python3.8/site-packages:$PYTHONPATH"
11 export PATH="$PIP_PREFIX/bin:$PATH"
12 unset SOURCE_DATE_EPOCH
13 '';
14 # Apparently pip needs 1980 or above
15 # https://github.com/ento/elm-doc/blob/master/shell.nix
16 python = pkgs."python${pythonVersion}".override {
17 packageOverrides = self: super: {
18 pytest = super.pytest.overridePythonAttrs (old: rec {
19 doCheck = false;
20 doInstallCheck = false;
21 });
22 ase = super.ase.overridePythonAttrs (old: rec {
23 doCheck = false;
24 doInstallCheck = false;
25 });
26 f90wrap = self.buildPythonPackage rec {
27 pname = "f90wrap";
28 version = "0.2.3";
29 src = pkgs.fetchFromGitHub {
30 owner = "jameskermode";
31 repo = "f90wrap";
32 rev = "master";
33 sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23";
34 };
35 buildInputs = with pkgs; [ gfortran stdenv ];
36 propagatedBuildInputs = with self; [
37 setuptools
38 setuptools-git
39 wheel
40 numpy
41 ];
42 preConfigure = ''
43 export F90=${pkgs.gfortran}/bin/gfortran
44 '';
45 doCheck = false;
46 doInstallCheck = false;
47 };
48 };
49 };
50 myPy = python.withPackages
51 (p: with p; [ ase ipython ipykernel scipy numpy f90wrap pip ]);
52in pkgs.mkShell {
53 buildInputs = with pkgs; [
54 # Required for the shell
55 zsh
56 perl
57 git
58 direnv
59 fzf
60 ag
61 fd
62
63 # Building thigns
64 gcc9
65 gfortran
66 openblas
67
68 myPy
69 # https://github.com/sveitser/i-am-emotion/blob/294971493a8822940a153ba1bf211bad3ae396e6/gpt2/shell.nix
70 ];
71 shellHook = hook;
72}
This is enough to cover almost all use-cases for python
environments.
Build Helpers
Note that we can speed up some aspects of fetch with the prefetch commands:
1nix-prefetch-git $giturl
2nix-prefetch-url $url
In practice, some trial and error is easier.
Supplementary Reading Material
Though these are in no means exhaustive, they may offer a slightly more advanced or different focus than the material covered here.
Core Content
- Manuals
- Nix and Nixpkgs
- Nix Wiki
- Language Sections
Learning Paths
- Nix pills
- Official tutorials
- Nix dev has some nice opinionated tips
Personal Correspondence
Tyson Whitehead from Compute Canada was kind enough to bring the folllowing additional training materials:
- A wiki pertaining to usage of Nix on in an HPC setting
- SWC style workshop materials from TECC 2018
- SHARCNET live presentation materials from 2018
Conclusions
The standard dive into Nix is based on building derivations and playing with language, which is in no means a bad one, just too long for the time allocated. The best way to get into Nix is to start using it for everything.
References
Dolstra, Eelco, Merijn de Jonge, and Eelco Visser. 2004. “Nix: A Safe and Policy-Free System for Software Deployment,” 15.
Dolstra, Eelco, Andres Löh, and Nicolas Pierron. 2010. “NixOS: A Purely Functional Linux Distribution.” Journal of Functional Programming 20 (5-6): 577–615. https://doi.org/dfrgtj.
Goswami, Rohit, Amrita Goswami, and Jayant K. Singh. 2020. “D-SEAMS: Deferred Structural Elucidation Analysis for Molecular Simulations.” Journal of Chemical Information and Modeling 60 (4): 2169–77. https://doi.org/10.1021/acs.jcim.0c00031.
For reasons pertaining to latency and ease-of-use, we will assume the multi-user installation ↩︎