Documenting C++ with Doxygen and Sphinx - doxyrest

Unifying C++ and Python documentation.

Background

For almost half a decade my C++ projects have been either scoped to where plain Doxygen works well enough ¹, or largely undocumented at the C++ API level ². While integrating with Sphinx via exhale or breathe shown earlier ³ works well, it often fails to provide a seamless source code view or struggles with modern C++ constructs.

For smaller, well formed libraries, the drawbacks of not being able to read the source are manifold, tending towards over-verbose documentation strings. Back in 2020 during the season of docs for Symengine, I stuck to pure doxygen with a custom theme, doxyYoda, a setup which served me well for Fortran projects like GaussJacobiQuad ⁴, and continues to be attractive.

Given the advent of pixi, however, it is worth revisiting the doxyrest setup ⁵. For the most part, the focus will be on implementation within a meson based project, rgpot, which defines an RPC interface for accessing forces and energies, typically for an atomistic calculation, before demonstrating the benefit for a much larger project, metatomic.

Baseline – rgpot

In general, the pipeline relies on Doxygen emitting XML, which doxyrest parses via Lua templates to generate reStructuredText, finally consumed by Sphinx. This allows the C++ API to sit seamlessly alongside Python documentation in a single Sphinx project, utilizing modern themes like furo or pydata-sphinx-theme.

The pixi task workflow

For rgpot, I opted to encapsulate the build complexity within pixi tasks. This avoids polluting the repository with git submodules for tooling that is essentially a build-time dependency, and has better support for managing dependencies.

The workflow is broken down into dependency retrieval, patching, and generation:

1[feature.docs.tasks.setup-doxyrest]
2description = "Clones the doxyrest dependency into the subprojects directory"
3cmd = """
4test -d subprojects/doxyrest || \
5(mkdir -p subprojects && \
6git clone --depth 1 --single-branch \
7https://github.com/vovkos/doxyrest subprojects/doxyrest)
8"""

The setup-doxyrest task handles the “shallow clone” approach to keep the checkout light.

The Build Cycle

With the infrastructure in place, the actual build is a two-step process defined in doxybuild:

1[feature.docs.tasks.doxybuild]
2description = "Generates XML via Doxygen and converts it to RST via Doxyrest"
3cwd = "docs/source"
4depends-on = ["setup-doxyrest"]
5cmd = "doxygen Doxyfile_potlib.cfg && doxyrest -c doxyrest-config.lua"

We first generate the XML blobs with Doxygen. Immediately after, doxyrest is invoked with a Lua configuration file, which defines the files used to map the generated XML into frames. Internally, doxyrest relies on Lua Frames which are templates that mix standard reStructuredText with injected Lua logic (delimited by %{ ... }) and variable substitution (via $var).

In principle, one could write custom frames to radically alter the output, for C++ projects we simply point FRAME_DIR_LIST to the standard “C-family” frames provided by the repo we cloned earlier.

 1INPUT_FILE = "xml/index.xml"
 2OUTPUT_FILE = "api/index.rst"
 3INDEX_TITLE = "API Reference"
 4
 5FRAME_DIR_LIST = {
 6    "../../subprojects/doxyrest/frame/common",
 7    "../../subprojects/doxyrest/frame/cfamily"
 8}
 9
10LANGUAGE = "cpp"
11ESCAPE_ASTERISKS = true
12ESCAPE_TRAILING_UNDERSCORES = true

With this, doxyrest digests the Doxygen XML into produce RST files that Sphinx can ingest.

Sphinx Configuration

For Sphinx to process the generated RST correctly, it needs to load the doxyrest extension. Since we are downloading/cloning this into a local subdirectory, we must explicitly update the python path in docs/src/conf.py so Sphinx can find it.

 1# For doxyrest, the location of this can change
 2sys.path.insert(0, os.path.abspath("subprojects/doxyrest/sphinx"))
 3
 4extensions = [
 5    # ... other extensions
 6    # c++ helpers
 7    "doxyrest",
 8    "cpplexer",
 9    # ...
10]

Finally, the standard Sphinx build can pick up the artifacts:

1[feature.docs.tasks.sphinxbld]
2depends-on = [ { task = "mkrst", environment = "docs" }, ]
3cmd = """ sphinx-build docs/source/ docs/build """

CI deployment

The primary advantage of encapsulating the build logic within pixi tasks lies in the simplification of Continuous Integration (CI) pipelines. Since the environment and build steps reside within pixi.toml, the CI configuration need only setup pixi and invoke the target task.

1- uses: prefix-dev/setup-pixi@v0.8.10
2  with:
3    activate-environment: true
4    environments: docs
5- name: Generate Docs
6  run: pixi r docbld

This reduces the friction of maintenance significantly, as local reproduction of CI failures becomes a matter of running pixi r docbld, and integrates easily with other actions, like the doc-previewer to post artifact links directly on Pull Requests, facilitating easier review of the generated output without digging through Action logs.

Metatomic integration

For larger projects like metatomic which do not use pixi, the process requires a bit more manual orchestration. Here, the challenge is replacing the declarative task runner with a robust script that ensures dependencies (like the doxyrest lua frames) are present before invoking the build.

While a Makefile or bash script is the traditional route, using nushell provides a nice balance of readability and cross-platform handling without the foot-guns of strict POSIX shell scripting, which are best avoided, despite my return to bash.

The ideal orchestrator script

Instead of relying solely on tox and file driven configurations to do the heavy lifting of environment setup, we use a nu script to bootstrap the documentation build.

First, we define our environment and ensure the local directory structure exists. We check if doxyrest is available in the path (useful for local development) or if we need to fetch it.

1let root = ($env.PWD)
2let deps_dir = ($root | path join "docs/subprojects")
3let doxyrest_src = ($deps_dir | path join "doxyrest")
4let install_dir = ($deps_dir | path join "install")
5let has_doxyrest = (which doxyrest | is-not-empty)
6
7if not ($deps_dir | path exists) { mkdir $deps_dir }

Next, if the binary is missing, we download the pre-compiled release. This avoids the complexity of setting up a C++ build chain in the CI environment just for a doc tool.

 1if not $has_doxyrest and not ($install_dir | path exists) {
 2    print "Downloading Doxyrest binary release..."
 3    let version = "2.1.3"
 4    let target = "linux-amd64"
 5    let archive = $"doxyrest-($version)-($target).tar.xz"
 6    let url = $"https://github.com/vovkos/doxyrest/releases/download/doxyrest-($version)/($archive)"
 7
 8    cd $deps_dir
 9    http get $url | save $archive
10    tar -xJf $archive
11
12    if ($install_dir | path exists) { rm -rf $install_dir }
13    mv $"doxyrest-($version)-($target)" $install_dir
14    rm $archive
15    cd $root
16}

Even with the binary, we still need the Lua frames (themes/templates) from the source repository. We clone this shallowly.

1if not ($doxyrest_src | path exists) {
2    print $"Cloning doxyrest frames into ($doxyrest_src)..."
3    git clone --depth 1 --single-branch https://github.com/vovkos/doxyrest $doxyrest_src
4}

Finally, we run the pipeline. Note that we prepend the local install directory to the PATH only for the doxyrest execution block.

 1print "Generating XML with Doxygen..."
 2cd docs
 3if not ("build/doxygen/xml" | path exists) { mkdir build/doxygen/xml }
 4doxygen Doxyfile.metatomic
 5
 6print "Generating RST with local Doxyrest..."
 7with-env { PATH: ($env.PATH | prepend ($install_dir | path join "bin")) } {
 8    doxyrest -c doxyrest-config.lua
 9}
10
11print "Building HTML with Sphinx (tox)..."
12tox -e docs

Sphinx and CI configurations are similar to the previous project, with the complete including a bash orchestrator in the PR here.

 1#!/usr/bin/env nu
 2
 3def main [] {
 4    let root = ($env.PWD)
 5    let deps_dir = ($root | path join "docs/subprojects")
 6    let doxyrest_src = ($deps_dir | path join "doxyrest")
 7    let install_dir = ($deps_dir | path join "install")
 8    let has_doxyrest = (which doxyrest | is-not-empty)
 9
10    if not ($deps_dir | path exists) { mkdir $deps_dir }
11
12    # 1. Grab Doxyrest Binary if missing
13    if not $has_doxyrest and not ($install_dir | path exists) {
14        print "Downloading Doxyrest binary release..."
15        let version = "2.1.3"
16        let target = "linux-amd64"
17        let archive = $"doxyrest-($version)-($target).tar.xz"
18        let url = $"https://github.com/vovkos/doxyrest/releases/download/doxyrest-($version)/($archive)"
19
20        cd $deps_dir
21        http get $url | save $archive
22        tar -xJf $archive
23
24        if ($install_dir | path exists) { rm -rf $install_dir }
25        mv $"doxyrest-($version)-($target)" $install_dir
26        rm $archive
27        cd $root
28    }
29
30    # 2. Clone the frames (still needed for the Lua templates)
31    if not ($doxyrest_src | path exists) {
32        print $"Cloning doxyrest frames into ($doxyrest_src)..."
33        git clone --depth 1 --single-branch https://github.com/vovkos/doxyrest $doxyrest_src
34    }
35
36    # 3. Run the Pipeline
37    print "Generating XML with Doxygen..."
38    cd docs
39    if not ("build/doxygen/xml" | path exists) { mkdir build/doxygen/xml }
40    doxygen Doxyfile.metatomic
41
42    print "Generating RST with local Doxyrest..."
43    with-env { PATH: ($env.PATH | prepend ($install_dir | path join "bin")) } {
44        doxyrest -c doxyrest-config.lua
45    }
46
47    print "Building HTML with Sphinx (tox)..."
48    tox -e docs
49}

Conclusions

Personally, I like the look of Doxygen still, and for some themes like shibuya, manual patches are required (e.g. monkeypatch @ rgpot). Nevertheless there are multi-language projects ⁶ which benefit from having a consistent style across languages, so this works out quite well. Probably the next step would consistency across even more languages and esoteric builds, like those in Metatensor.