HPC Configuration 2021

A post on configuring a user account for heavy interactive HPC usage with Spack, Nix and self-hosted applications.

Background

I have in the past written a few posts about configuring my HPC setup. Since the Research and University Network of Iceland hf. (RHnet) recently recieved funding for a new machine, called Elja ¹, I decided to jot down my notes from my setup as part of the user interaction team. Without abusing the ability to bend the ear of the system administrators, I decided to document a user-facing workflow. All the commands here assume an interactive session (example in slurm):

1# 26 should be the number of processors on a node
2srun -n 26 --pty /bin/bash

Rust and Utilities

rustup is a fantastic way to get cargo up and running and immediately allows for the installation of many relevant command line utilities.

1curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
2source $HOME/.cargo/env

The ones I use mostly are enumerated in Table 1.

Table 1: Language bindings and tools

Utility	Rust Alternative	Rationale
`cat`	bat	File reader with syntax highlighting
`top`	bottom	Process management
`ls`	exa	Listing things with colors
`ps`	procs	Better looking process lists
`grep`	ripgrep	Search recursively
`find`	fd-find	Find things
`sed`	sd	Search and replace things
`time`	hyperfine	Better benchmarking with stats
`man`	tealdeer	A fast `tldr` page viewer
`fzf`	skim	A fuzy finder
–	navi	Personalized cheatsheets
`cd / z`	zoxide	Directory jumping aid
`gibo`	bliss	`gitignore` generation utility
`diff`	delta	Nicer diffs
`watch`	watchexec	Watch and execute
`CTRL+R`	mcfly	Reverse search

These are trivial to obtain:

1cargo install bottom ripgrep hyperfine skim mcfly
2# The following need some system libraries (OpenSSL) and shall be dealt with later, or with micromamba
3# bat exa fd-find tealdeer starship sd procs zoxide bliss navi watchexec

Not immediately obvious, but easy to update as well.

1cargo install cargo-update
2cargo install-update --all

MicroMamba

We will set up a micromamba installation to get git and other base utilities.

1mkdir -p ~/.local/bin
2export PATH=$HOME/.local/bin:$PATH
3wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
4mv bin/micromamba ~/.local/bin
5rm -rf bin
6micromamba shell init -s bash -p $HOME/.micromamba
7. ~/.bashrc

Now we can use mamba to get a temporary interactive environment.

1# Spack requirements
2PKGS="git patch curl make tar gzip unzip bzip2 xz zstd subversion lsof m4 openssl pkg-config"
3mkdir ~/.mamba_venvs
4micromamba create -p ~/.mamba_venvs/intBase $PKGS
5micromamba activate ~/.mamba_venvs/intBase

What we need micromamba for is really the openssl library.

Configuration

A simple .condarc will suffice for the above.

1channels:
2  - conda-forge
3  - defaults
4channel_priority: disabled

GitHub Client

We can grab the GitHub client binary for interacting with GitHub for now. Configuring and using this spares the necessity of explaining SSH authentication.

 1ghcliVer=1.13.1
 2DST=~/.local
 3wget https://github.com/cli/cli/releases/download/v${ghcliVer}/gh_${ghcliVer}_linux_amd64.tar.gz
 4tar xf "gh_${ghcliVer}_linux_amd64.tar.gz"
 5mkdir -p ~/.local/
 6cd "gh_${ghcliVer}_linux_amd64"
 7# Merges directories
 8# From: https://unix.stackexchange.com/a/603209/246530
 9find -type d -exec mkdir -vp "$DST"/{} \; -or -exec mv -nv {} "$DST"/{} \;
10cd ../
11rm -rf "gh_${ghcliVer}_linux_amd64" "gh_${ghcliVer}_linux_amd64.tar.gz"

Now we can set this up.

1gh auth login # Follow the instructions

The rest of the post assumes gh is present.

Dotfiles

Some of this post is simplified by using my Dotfiles with bombadil:

1cargo install toml-bombadil
2mkdir -p $HOME/Git/Github
3cd ~/Git/Github
4gh repo clone HaoZeke/Dotfiles
5cd Dotfiles
6git checkout bombadil
7bombadil install -c bombadil.toml

We will need to make some directories since bombadil does not currently handle this. Make changes to the variables as required and then link them.

1mkdir -p ~/.config/nvim ~/.config/alacritty/ \
2    ~/.spack/
3bombadil link -p hpc elja # Combining profiles (variables)

Spack

For setting multi compiler workflows on the HPC, spack is a great tool. It is also well documented and easy to add to. Rather than setting up a hierarchical Lmod structure, we shall use a spack environment. Although we have already a micromamba environment, spack configurations are more suited to the HPC workflow; since they are more easily customized.

1# Install
2cd $HOME
3gh repo clone spack/spack

Now we can start by getting a new compiler.

1. ~/spack/share/spack/setup-env.sh
2spack install gcc@9.3.0
3spack compiler add $(spack location -i gcc@9.3.0)
4spack install gcc@10.2.0 %gcc@9.3.0
5spack compiler add $(spack location -i gcc@10.2.0)
6spacktivate interactive # After setting the environment

Environments

This step is not required if one uses my dotfiles.

 1# $SPACK_HOME//var/spack/environments/interactive/spack.yaml
 2spack:
 3  definitions:
 4    - compilers: ['gcc@10.2.0']
 5    - packages: [
 6      'zsh', 'tmux', 'git',
 7      'openssl', 'gmp', 'mpfr',
 8      'mpc', 'perl', 'boost', 'subversion',
 9      'zlib', 'cmake', 'bison', 'bdw-gc',
10      'flex', 'pcre', 'jq', 'patch',
11      'bzip2', 'sqlite', 'editline',
12      'ncurses', 'texinfo', 'brotli'
13      ]
14  specs:
15    - $compilers
16    - 'subversion%gcc@9.3.0'
17    - matrix:
18       - [$packages]
19       - [$%compilers]
20  view: true

Note that subversion depends on serf which currently does not build with gcc@10.x.x.

1spacktivate interactive
2spack install

Alternatively, one can just spack load packages of interest after spack install.

Cleaning Modules

This step is not required if one uses the dotfiles.

 1modules:
 2  enable::
 3    - lmod
 4  lmod:
 5    core_compilers:
 6      - 'gcc@10.2.0'
 7    hierarchy:
 8      - mpi
 9    hash_length: 0
10    whitelist:
11      - gcc
12    blacklist:
13      - '%gcc@10.2.0'
14    all:
15      filter:
16        environment_blacklist:
17          - "C_INCLUDE_PATH"
18          - "CPLUS_INCLUDE_PATH"
19          - "LIBRARY_PATH"
20      environment:
21        set:
22          '{name}_ROOT': '{prefix}'
23    openmpi:
24      environment:
25        set:
26          SLURM_MPI_TYPE: pmi2
27          OMPI_MCA_btl_openib_warn_default_gid_prefix: '0'
28    projections:
29      all:          '{name}/{version}'
30      ^lapack:      '{name}/{version}-{^lapack.name}'

We can then rebuild the setup.

1spack module lmod refresh --delete-tree -y

Nix

Portable

nix-portable is one of the best projects for working with local nix installations. Where it works, it works very simply:

1wget https://github.com/DavHau/nix-portable/releases/download/v006/nix-portable
2chmod +x nix-portable
3mv nix-portable ~/.local/bin

Mamba

1# Spack requirements
2PKGS="git patch curl make tar gzip bzip2 xz zstd brotli openssl sqlite bdw-gc nlohmann_json editline libxml2 libxslt libseccomp bison flex lsof libarchive automake pkg-config"
3micromamba create -p ~/.mamba_venvs/nixIns $PKGS
4micromamba activate ~/.mamba_venvs/nixIns

It is best to have a clean, maximal boost.

1git clone --recursive https://github.com/boostorg/boost.git
2cd boost
3git checkout tags/boost-1.73.0
4./bootstrap.sh
5./b2 headers
6./b2
7./b2 install --prefix=$HOME/.hpc/boost/boost-1.73.0
8ml load boost/boost-1.73.0

Now we can try to compile a patched older nix.

1export myprefix=$HOME/.hpc/nix/nix-boot
2export nixdir=$HOME/.nix
3./configure  --enable-gc --prefix=$myprefix \
4--with-store-dir=$nixdir/store --localstatedir=$nixdir/var \
5\
6--disable-seccomp-sandboxing --disable-doc-gen \
7CPPFLAGS="-I$HOME/.mamba_venvs/nixIns/include" LDFLAGS="-L$HOME/.mamba_venvs/nixIns/lib -Wl,-R$HOME/.mamba_venvs/nixIns/lib"
8make -j $(nproc)
9make install

Spack

When the existing methods don’t work, we can fall back to a more rustic method. We need some additional packages, which are not part of spack yet.

1spack create https://github.com/anrieff/libcpuid/releases/download/v0.5.1/libcpuid-0.5.1.tar.gz
2spack install libcpuid

1mkdir -m 0755 ~/.nix
2spack install nix storedir=$nixdir/store statedir=$nixdir/var doc=False sandboxing=False nfs=True
3spack load nix
4# Basic setup
5mkdir -p  ~/.nix/var/nix/profiles
6nix-env --switch-profile .nix/var/nix/profiles/default

Now we can install this natively with spack.

 1mkdir -p $HOME/Git/Github
 2gh repo clone nixos/nixpkgs
 3cd nixpkgs
 4export NIX_PATH=$(pwd)
 5export nixdir=$HOME/.nix
 6export PATH=$(spack location -i nix)/bin/:$PATH:$nixdir/bin
 7export INCLUDE=$(spack location -i nix)/include/:$INCLUDE
 8export LD_LIBRARY_PATH=$(spack location -i nix)/lib:$LD_LIBRARY_PATH
 9export PKG_CONFIG_PATH=$(spack location -i nix)/lib/pkgconfig/:$PKG_CONFIG_PATH
10nix-env -i nix -f $HOME/Git/Github/nixpkgs -j$(nproc) --keep-going --show-trace -v --cores $(nproc) 2>&1 | tee nix-no-root.log

 1export BOOST_ROOT=$(spack location -i boost)
 2./configure --enable-gc \
 3 --prefix=$myprefix \
 4 --with-store-dir=$nixdir/store \
 5 --localstatedir=$nixdir/var \
 6 --with-boost=$BOOST_ROOT \
 7 --disable-seccomp-sandboxing \
 8 --disable-doc-gen \
 9 --includedir="$(spack location -i bdw-gc)" \
10 --includedir="$(spack location -i bzip2)" \
11 --includedir="$(spack location -i nlohmann-json)" \
12 --libdir="$(spack location -i bzip2)" \
13 --libdir="$(spack location -i bdw-gc)" \
14 PKG_CONFIG=$(spack location -i pkg-config)/bin/pkg-config \
15 OPENSSL_CFLAGS=-I$(spack location -i openssl)/include \
16 OPENSSL_LIBS=-L$(spack location -i openssl)/lib \
17 CPPFLAGS="-I$(spack location -i nlohmann-json)/include"
18make -j $(nproc)
19make install
20ml load nix/user # Hooray!

Self hosted binaries

Code Server

The first of the selfhosted set is the browser based IDE inspired by Visual Studio Code.

1export VERSION=3.12.0
2mkdir -p ~/.local/lib ~/.local/bin
3curl -fL "https://github.com/cdr/code-server/releases/download/v$VERSION/code-server-$VERSION-linux-amd64.tar.gz" \
4  | tar -C ~/.local/lib -xz
5mv "$HOME/.local/lib/code-server-$VERSION-linux-amd64" "$HOME/.local/lib/code-server-$VERSION"
6ln -s "$HOME/.local/lib/code-server-$VERSION/bin/code-server" "$HOME/.local/bin/code-server"
7PATH="$HOME/.local/bin:$PATH"

This has the advantage of not needing any port forwarding (with --link):

1code-server --link

Naturally this is often disabled by more security oriented SysAdmins. This can be circumvented with the same ngrok scheme as outlined earlier.

Conclusions

This is a work in progress, and would likely need to be updated ever so often, perhaps like a “State Of My Dots” or a yearly “HPC Configuration $YEAR” or something like that.

A step in the right direction in terms of female representation among the university servers ↩︎