Short post on making minimal changes to derivations in nixpkgs at a project level using callPackage() along with GH-Actions for deployment of sphinx documentation.

Background

As part of my work on the Symengine documentation1, I had originally thought of leveraging nix for reproducible builds for each of the language bindings with GH-Actions. There exists a derivation in the upstream package repository, but it was outdated (v6.0.0 instead of v6.0.1) 2. Normally this would simply require a PR to be fixed; but since documenting different language bindings requires specific flags and tests were largely not an issue, I needed to make a project level derivation for each repository.

Project Layout

The standard niv setup, as described in the longer tutorial post will suffice.

1niv init -b master

Since the bindings were for python, mach-nix was leveraged for the shell.nix; reasons for which have been outlined elsewhere. This led to the following expression:

 1let
 2  sources = import ./nix/sources.nix;
 3  pkgs = import sources.nixpkgs { };
 4  nsymengine = pkgs.callPackage ./nix/nsymengine.nix { };
 5  mach-nix = import (builtins.fetchGit {
 6    url = "https://github.com/DavHau/mach-nix/";
 7    ref = "refs/tags/3.1.1";
 8  }) {
 9    pkgs = pkgs;
10  };
11  customPython = mach-nix.mkPython rec {
12    requirements = builtins.readFile ./requirements.txt;
13  };
14in pkgs.mkShell {
15  buildInputs = with pkgs; [
16    customPython
17    cmake
18    nsymengine
19    bashInteractive
20    sage
21    which
22  ];
23}

The highlighted line calls nsymengine.nix which is derived from the upstream expression. Since callPackage is being used from the niv pinned sources, the expression is completely project-specific and reproducible.

Upstream

For completion; the upstream expression is reproduced. We will actually grab this directly with show-derivation:

1nix show-derivation -f "<nixpkgs>" symengine > nix/nsymengine.nix

Which gives us:

 1{ lib, stdenv
 2, fetchFromGitHub
 3, cmake
 4, gmp
 5, flint
 6, mpfr
 7, libmpc
 8}:
 9
10stdenv.mkDerivation rec {
11  pname = "symengine";
12  version = "0.6.0";
13
14  src = fetchFromGitHub {
15    owner = "symengine";
16    repo = "symengine";
17    rev = "v${version}";
18    sha256 = "129iv9maabmb42ylfdv0l0g94mcbf3y4q3np175008rcqdr8z6h1";
19  };
20
21  nativeBuildInputs = [ cmake ];
22
23  buildInputs = [ gmp flint mpfr libmpc ];
24
25  cmakeFlags = [
26    "-DWITH_FLINT=ON"
27    "-DINTEGER_CLASS=flint"
28    "-DWITH_SYMENGINE_THREAD_SAFE=yes"
29    "-DWITH_MPC=yes"
30    "-DBUILD_FOR_DISTRIBUTION=yes"
31  ];
32
33  doCheck = true;
34
35  checkPhase = ''
36    ctest
37  '';
38
39  meta = with lib; {
40    description = "A fast symbolic manipulation library";
41    homepage = "https://github.com/symengine/symengine";
42    platforms = platforms.unix ++ platforms.windows;
43    license = licenses.bsd3;
44    maintainers = [ maintainers.costrouc ];
45  };
46
47}

Repo-native

The truncated expression meant to be bundled with the repo is reproduced below.

 1{ lib, stdenv
 2, fetchFromGitHub
 3, cmake
 4, gmp
 5, flint
 6, mpfr
 7, libmpc
 8}:
 9
10stdenv.mkDerivation rec {
11  pname = "symengine";
12  name = "symengine";
13  version = "44eb47e3bbfa7e06718f2f65f3f41a0a9d133b70"; # From symengine-version.txt
14
15  src = fetchFromGitHub {
16    owner = "symengine";
17    repo = "symengine";
18    rev = "${version}";
19    sha256 = "137cxk3x8vmr4p5x0knzjplir0slw0gmwhzi277si944i33781hd";
20  };
21
22  nativeBuildInputs = [ cmake ];
23
24  buildInputs = [ gmp flint mpfr libmpc ];
25
26  cmakeFlags = [
27    "-DWITH_FLINT=ON"
28    "-DINTEGER_CLASS=flint"
29    "-DWITH_SYMENGINE_THREAD_SAFE=yes"
30    "-DWITH_MPC=yes"
31    "-DBUILD_TESTS=no"
32    "-DBUILD_FOR_DISTRIBUTION=yes"
33  ];
34
35  doCheck = false;
36
37}
38
39# Derived from the upstream expression : https://github.com/r-ryantm/nixpkgs/blob/34730e0640710636b15338f20836165f29b3df86/pkgs/development/libraries/symengine/default.nix

Some project-specific notes:

  • m2r2 was used to inject the project README.md into the Sphinx reStructuredText index file
  • To prevent symengine from picking up the system python, a pure shell was required nix-shell --pure --run bash
  • symengine-version.txt was already being used by the CI setup for cloning the right commit, and though a text file for hash storage is inelegant, it is a valid approach without nix

GH-Actions and Deployment

Having obtained an environment; the penultimate step involved a trivial shell script which uses the fantastic nix-shell:

1#!/usr/bin/env nix-shell
2#!nix-shell ../shell.nix -i bash
3
4python setup.py build_ext --inplace
5sphinx-build docs/ genDocs/

Finally an action definition in .github/workflows/mkdocs.yaml:

 1on:
 2  push:
 3    branches:
 4      - main
 5      - master
 6
 7name: Make Sphinx API Docs
 8
 9jobs:
10  mkdocs:
11    runs-on: ubuntu-latest
12    steps:
13      - uses: actions/checkout@v2
14
15      - uses: cachix/install-nix-action@v12
16        with:
17          nix_path: nixpkgs=channel:nixos-unstable
18
19      - name: Build
20        run: ./bin/docbuilder.sh
21
22      - name: Deploy
23        uses: peaceiris/actions-gh-pages@v3
24        with:
25          github_token: ${{ secrets.GITHUB_TOKEN }}
26          publish_dir: ./genDocs

Conclusion

In the end, to ensure easy long-term maintenance by the other maintainers, it was decided that the nix derivations should be trashed in favor of conda madness. Though most changes should be contributed upstream, the efficient re-use of upstream expressions is still viable for certain projects, especially those related to documentation.

  1. Sponsored by the Google Season of Docs initiative ↩︎

  2. Part of the documentation design worked out involved the live at head concept (also described in this CPPCon17 talk) ↩︎