Publishing Doxygen and Sphinx with Nix and Rake

Automating documentation deployment with Travis, rake and nix

Background

In the previous post we generated documentation using Doxygen with Exhale to handle Sphinx. Now we will clean up the earlier workflow with rake and ensure the environment is reproducible with nix while deploying to Travis CI.

Setup

A quick reminder of the setup we generated in the last post:

1tree -d $prj/ -L 2

We had further setup files to enable documentation generation with a manual two stage process (handling doxygen and sphinx separately).

1cd docs/Doxygen
2doxygen Doxyfile-prj.cfg
3cd ../Sphinx
4make html
5mv build/html ../../public

This might be extracted into a simple build.sh script, and then we might decide to have a clean.sh script and then we might try to replicate all the functionality of a good build system with scripts.

Thankfully, we will instead start with a build script defined as above to transition to nix, before using an actual build tool for our dirty work.

Adding Nix

It wouldn’t make sense for me to not stick nix into this. I recall the dark days of setting up Dockerfiles to ensure reproducible environments on Travis.

At this point one might assume we will leverage the requirements.txt based workflow described earlier in Niv and Mach-Nix for Nix Python. While this would make sense, there are two barriers to its usage:

• It is slower than a poetry build, as dependency resolution is performed
• It does not play well with existing projects
  • Most python projects do not rely solely on requirements.txt ¹

Poetry2Nix

Recall that as sphinx is originally meant for and most often used for Python projects, we will need to consider the possibility (remote though it is) that there might be users who would like to test the documentation without setting up nix.

Thus we will look to the poetry2nix project instead. We note the following:

• The poetry2nix setup is faster (as it consumes a lockfile instead of solving dependencies from requirements.txt)
  • mach-nix however, is more flexible and can make use of the poetry2nix overrides
• In a strange chicken and egg problem, we will have to manually generate the lockfile, thereby creating an impure poetry project for every update, though the nix setup will not need it later
  • This is one of the major reasons to prefer mach-nix for newer projects

Shell Environment

We prep our sources in the usual way, by running niv init in the project root to generate the nix/ folder and the sources therein. With all that in mind, the shell.nix file at this point is fairly standard, keeping the general niv setup in mind (described in a previous Nix tutorial):

1# -*- mode: nix-mode -*-
2let
3  sources = import ./nix/sources.nix;
4  pkgs = import sources.nixpkgs { };
5  customPython = pkgs.poetry2nix.mkPoetryEnv { projectDir = ./.; };
6in pkgs.mkShell {
7  buildInputs = with pkgs; [ doxygen customPython rake darkhttpd ];
8}

Where the most interesting aspect is that the projectDir is to be the location of the project root, though both poetrylock and pyproject variables are supported.

Refactoring

We consider the problem of refactoring the build.sh script:

1#!/usr/bin/env bash
2cd docs/Doxygen
3doxygen Doxyfile-prj.cfg
4cd ../Sphinx
5make html
6mv build/html ../../public

Without resorting to methods such as nix-shell --run build.sh --pure.

Nix Bash

Script in hand, we would like to be able to run it directly in the nix environment. We modify the script as follows:

 1#! /usr/bin/env nix-shell
 2#! nix-shell deps.nix -i bash
 3
 4# Build Doxygen
 5cd docs/Doxygen
 6doxygen Doxyfile-syme.cfg
 7
 8# Build Sphinx
 9cd ../Sphinx
10make html
11mv build/html ../../public
12
13# Local Variables:
14# mode: shell-script
15# End:

This calls on a deps.nix² which we shall generate in a manner very reminiscent of the shell.nix ³ as follows:

1let
2  sources = import ./../nix/sources.nix;
3  pkgs = import sources.nixpkgs { };
4  customPython = pkgs.poetry2nix.mkPoetryEnv { projectDir = ./../.; };
5in pkgs.runCommand "dummy" {
6  buildInputs = with pkgs; [ doxygen customPython ];
7} ""

Only the paths have changed, and instead of creating and returning a shell environment with mkShell we instead “run” a derivation instead. At this point we can run this simply as:

1./scripts/build.sh

This is reasonably ready (as a first draft) for being incorporated into a continuous integration workflow.

Travis CI

Seeing as Travis provides first class nix support, as well as excellent integration with GitHub, we will prefer it.

Settings

A minor but necessary evil is setting up a PAP (personal access token) from here. Depending on what repositories are being used, the scope should encompass repo permissions (minimally public_repo), and admin:org permissions might be required.

Having obtained the token, we will need to navigate to the Settings section on the Travis web-UI and add the token as an environment variable, we might be partial to a name like GH_TOKEN.

Build Configuration

We will leverage the following configuration:

 1language: nix
 2
 3before_install:
 4  - sudo mkdir -p /etc/nix
 5  - echo "substituters = https://cache.nixos.org/ file://$HOME/nix.store" | sudo tee -a /etc/nix/nix.conf > /dev/null
 6  - echo 'require-sigs = false' | sudo tee -a /etc/nix/nix.conf > /dev/null
 7
 8before_script:
 9  - sudo mkdir -p /etc/nix && echo 'sandbox = true' | sudo tee /etc/nix/nix.conf
10
11script:
12  - scripts/build.sh
13
14before_cache:
15  - mkdir -p $HOME/nix.store
16  - nix copy --to file://$HOME/nix.store -f shell.nix buildInputs
17
18cache:
19  nix: true
20  directories:
21    - $HOME/nix.store
22
23deploy:
24  provider: pages
25  local_dir: ./public/
26  skip_cleanup: true
27  github_token: $GH_TOKEN # Set in the settings page of your repository, as a secure variable
28  keep_history: true
29  target_branch: master # Required for user pages
30  on:
31    branch: src

Where all the action is essentially in script and deploy. Note however, that the before_cache step should change if there is a default.nix instead. We will in this case, consider the situation of having an organization or user page being the deploy target.

Rake

Usable though the preceding setting is, it is still rather unwieldy in that:

• there are a bunch of artifacts which need to be cleaned manually
• it is fragile and tied to the folder names

We can fix this with any of the popular build systems, however here we will focus on the excellent rake ⁴. We shall commit to our course of action by removing make.

1cd docs/Sphinx
2rm Makefile make.bat # other make cruft

Components

Variables

We will begin by requiring rake and setting basic variables.

1require 'rake'
2
3CWD = File.expand_path(__dir__)
4DOXYFILE = "Doxyfile-prj.cfg"
5OUTDIR = File.join(CWD,"public")
6SPHINXDIR = File.join(CWD,"docs/Sphinx")

This section should give a fairly clear idea of how the Rakefile itself is essentially pure ruby code. We are now beginning to have more holistic control of how our project is structured.

Tasks

The general form of a task is simply:

1desc "Blah blah"
2task :name do
3# Something
4end

Some variations of this will be considered when appropriate.

Clean

A clean task is a good first task, being as it is almost trivial in all build systems.

1desc "Clean the generated content"
2task :clean do
3  rm_rf "public"
4  rm_rf "docs/Doxygen/gen_docs"
5  rm_rf "docs/Sphinx/build"
6end

Serve

We will use the lightweight darkhttpd server for our generated documentation.

1desc "Serve site with darkhttpd"
2task :darkServe, [:port] do |task, args|
3  args.with_defaults(:port => "1337")
4  sh "darkhttpd #{OUTDIR} --port #{args.port}"
5end

Note that we have leveraged the args system in this case, and also used the top-level OUTDIR variable.

Doxygen

Since the doxygen output is a pre-requisite, it makes sense to set it up early on.

1desc "Build doxygen"
2task :mkDoxy do
3  Dir.chdir(to = File.join(CWD,"docs/Doxygen"))
4  system('doxygen', DOXYFILE)
5end

Sphinx

This task will depend on having the doxygen output, so we will express this idiomatically by making the doxygen task run early on.

1desc "Build Sphinx"
2task :mkSphinx, [:builder] => ["mkDoxy"] do |task, args|
3  args.with_defaults(:builder => "html")
4  Dir.chdir(to = File.join(CWD,"docs/Sphinx"))
5  sh "poetry install"
6  sh "poetry run sphinx-build source #{OUTDIR} -b #{args.builder}"
7end

There are some subtleties here, notably:

• The task is meant to run without nix
• We use the args setup as before

No Nix Meta

With this we can now set up a task to build the documentation without having nix.

1desc "Build site without Nix"
2task :noNixBuild => "mkSphinx" do
3  Rake::Task["darkServe"].execute
4end

The main take-away here is that we finally call the Rake library itself, but within the task, which means the dependency tree is respected and we get doxygen->sphinx->darkhttpd as required.

Nix Builder

For nix use we note that we are unable to enter the nix environment from within the Rakefile itself. We work around this by being more descriptive.

1desc "Build Nix Sphinx, use as nix-shell --run 'rake mkNixDoc' --pure"
2task :mkNixDoc, [:builder] => "mkDoxy" do |task, args|
3  args.with_defaults(:builder => "html")
4  Dir.chdir(to = SPHINXDIR)
5  sh "sphinx-build source #{OUTDIR} -b #{args.builder}"
6end

Final Form

The final Rakefile shall be (with a default task defined):

 1require 'rake'
 2
 3# Variables
 4CWD = File.expand_path(__dir__)
 5DOXYFILE = "Doxyfile-prj.cfg"
 6OUTDIR = File.join(CWD,"public")
 7SPHINXDIR = File.join(CWD,"docs/Sphinx")
 8
 9# Tasks
10task :default => :darkServe
11
12desc "Clean the generated content"
13task :clean do
14  rm_rf "public"
15  rm_rf "docs/Doxygen/gen_docs"
16  rm_rf "docs/Sphinx/build"
17end
18
19desc "Serve site with darkhttpd"
20task :darkServe, [:port] do |task, args|
21  args.with_defaults(:port => "1337")
22  sh "darkhttpd #{OUTDIR} --port #{args.port}"
23end
24
25desc "Build Nix Sphinx, use as nix-shell --run 'rake mkNixDoc' --pure"
26task :mkNixDoc, [:builder] => "mkDoxy" do |task, args|
27  args.with_defaults(:builder => "html")
28  Dir.chdir(to = SPHINXDIR)
29  sh "sphinx-build source #{OUTDIR} -b #{args.builder}"
30end
31
32desc "Build site without Nix"
33task :noNixBuild => "mkSphinx" do
34  Rake::Task["darkServe"].execute
35end
36
37desc "Build doxygen"
38task :mkDoxy do
39  Dir.chdir(to = File.join(CWD,"docs/Doxygen"))
40  system('doxygen', DOXYFILE)
41end
42
43desc "Build Sphinx"
44task :mkSphinx, [:builder] => ["mkDoxyRest"] do |task, args|
45  args.with_defaults(:builder => "html")
46  Dir.chdir(to = File.join(CWD,"docs/Sphinx"))
47  sh "poetry install"
48  sh "poetry run sphinx-build source #{OUTDIR} -b #{args.builder}"
49end

Travis

We are now in a position to fix our travis build configuration. Simply replace the old and fragile build.sh script section with the following:

1script:
2  - nix-shell --run "rake mkNixDoc" --show-trace --verbose --pure

Direnv

As a bonus section, consider the addition of the following .envrc for those who keep multiple ruby versions:

1eval "$(rbenv init -)"
2rbenv shell 2.6.2
3rake -T

Activate this with the usual direnv allow. This has the added benefit of listing the defined tasks when cd‘ing into the project directory.

Conclusions

A lot has happened on the tooling end, even though the documentation itself has not been updated further. We have managed to setup a robust environment which is both reproducible and also amenable to users who do not have nix. We have also setup a build system, which can help us in many more ways as well (asset optimization through the rails pipeline). In the next post, we will return to the documentation itself for further tinkering.