Short post on using mach-nix with niv.

Background

In previous posts, there was a discussion on a ground up approach to adding packages which aren’t on the core nixpkgs channels using GitHub or PyPi sources. However, this lacked a way to do so programmatically, and also a way to convert existing python projects.

Python Dependency Management

This time, instead of the more pedagogical approach of building packages from PyPi or GitHub, we will use overlays and the excellent mach-nix to speed up the process. We will continue to use niv.

1niv init
2niv update nixpkgs -b nixpkgs-unstable

To leverage mach-nix we will simply need the following setup to work with niv.

 1let
 2  sources = import ./nix/sources.nix;
 3  pkgs = import sources.nixpkgs { };
 4  inherit (pkgs.lib) optional optionals;
 5  mach-nix = import (builtins.fetchGit {
 6    url = "https://github.com/DavHau/mach-nix/";
 7    ref = "refs/tags/3.1.1";
 8  }) {
 9    pkgs = pkgs;
10
11    # optionally specify the python version
12    # python = "python38";
13
14    # optionally update pypi data revision from https://github.com/DavHau/pypi-deps-db
15    # pypiDataRev = "some_revision";
16    # pypiDataSha256 = "some_sha256";
17  };
18  customPython = mach-nix.mkPython {
19    requirements = ''
20      copier
21      pytest
22    '';
23    providers = {
24      _default = "nixpkgs,wheel,sdist";
25      pytest = "nixpkgs";
26    };
27    pkgs = pkgs;
28  };
29in pkgs.mkShell { buildInputs = with pkgs; [ customPython ]; }

Note that we have essentially written out a requirements.txt and can actually pass a path there instead as well. The key point to make it work with niv is the pkgs parameter. To use the older method of overriding parts of the setup, we can use the overrides_pre hook as shown below:

 1let
 2  sources = import ./prjSource/nix/sources.nix;
 3  pkgs = import sources.nixpkgs { };
 4  inherit (pkgs.lib) optional optionals;
 5  mach-nix = import (builtins.fetchGit {
 6    url = "https://github.com/DavHau/mach-nix/";
 7    ref = "refs/tags/3.1.1";
 8  }) {
 9    pkgs = pkgs;
10
11    # optionally specify the python version
12    # python = "python38";
13
14    # optionally update pypi data revision from https://github.com/DavHau/pypi-deps-db
15    # pypiDataRev = "some_revision";
16    # pypiDataSha256 = "some_sha256";
17  };
18  customPython = mach-nix.mkPython {
19    requirements = ''
20      copier
21      pytest
22      f90wrap
23    '';
24    providers = {
25      _default = "nixpkgs,wheel,sdist";
26      pytest = "nixpkgs";
27    };
28    overrides_pre = [
29      (pythonSelf: pythonSuper: {
30        pytest = pythonSuper.pytest.overrideAttrs (oldAttrs: {
31          doCheck = false;
32          doInstallCheck = false;
33        });
34        f90wrap = pythonSelf.buildPythonPackage rec {
35          pname = "f90wrap";
36          version = "0.2.3";
37          src = pkgs.fetchFromGitHub {
38            owner = "jameskermode";
39            repo = "f90wrap";
40            rev = "master";
41            sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23";
42          };
43          buildInputs = with pkgs; [ gfortran stdenv ];
44          propagatedBuildInputs = with pythonSelf; [
45            setuptools
46            setuptools-git
47            wheel
48            numpy
49          ];
50          preConfigure = ''
51            export F90=${pkgs.gfortran}/bin/gfortran
52          '';
53          doCheck = false;
54          doIstallCheck = false;
55        };
56      })
57    ];
58    pkgs = pkgs;
59  };
60in pkgs.mkShell { buildInputs = with pkgs; [ customPython ]; }

We can also pull in overrides from poetry2nix with overrides_post as described here.

Conclusion

With the completion of this final remaining hurdle, nix is now fully realized as a python management system. At this point the “only” thing remaining is to find an optimal way of leveraging nix for setting up re-usable data science and scientific computing projects.