8 minutes
HPC Dotfiles and LMod
Background
My move away from the powerful, but unimaginatively named HPC clusters of IITK 1 brought me in close contact with the Lua based 2 lmod module system. Rather than fall into the rabbit hole of brew we will leverage the existing system to add our new libraries. Not finding any good collections of these composable environments, and having failed once before to install Nix as a user without admin access, I decided to start my own collection of Lmod recipies. The rest of this post details the installation proceedure to be carried out in conjunction with the hzHPC_lmod repo.
Setting Up
These are reproduced from the repo for completeness.
git clone https://github.com/kobus-v-schoor/dotgit.git
mkdir -p ~/.local/bin
cp -r dotgit/bin/dotgit* ~/.local/bin
cat dotgit/bin/bash_completion >> ~/.bash_completion
rm -rf dotgit
I actually strongly suggest using a target from my Dotfiles in conjunction with this, but it isn’t really required, so:
~/.local/bin/dotgit restore hzhpc
Note that because of the suggested separation, I have not opted to setup a shell or even ensure that there are scripts here to help keep module
in your path. Those are in my Dotfiles.
If, you opt to not use these dotfiles
, then do not run the ml load
commands.
LMod Libraries
Note that: garpur already has
lmod
and a module for GNUgcc
9.2.0
The scripts in this post will also be part of the repo, but keep in mind that these are not meant to be robust ways to install anything, and every command should be run by hand because things will probably break badly.
GMP
myprefix=$HOME/.hpc/gcc/gmp/6.2.0
export PATH
wget https://gmplib.org/download/gmp/gmp-6.2.0.tar.xz
tar xfv gmp-6.2.0.tar.xz
cd gmp-6.2.0
./configure --prefix=$myprefix \
--enable-cxx \
--docdir=$myprefix/doc/gmp-6.1.2
make -j$(nproc)
make install
MPFR
myprefix=$HOME/.hpc/gcc/mpfr/4.1.0
export PATH
wget https://www.mpfr.org/mpfr-current/mpfr-4.1.0.tar.xz
tar xfv mpfr-4.1.0.tar.xz
cd mpfr-4.1.0
./configure --prefix=$myprefix \
--enable-thread-safe \
--with-gmp=$HOME/.hpc/gcc/gmp/6.2.0 \
--docdir=$myprefix/doc/mpfr-4.1.0
make -j$(nproc)
make install
MPC
myprefix=$HOME/.hpc/gcc/mpc/1.2.0
export PATH
wget https://ftp.gnu.org/gnu/mpc/mpc-1.2.0.tar.gz
tar xfv mpc-1.2.0.tar.gz
cd mpc-1.2.0
./configure --prefix=$myprefix \
--with-gmp=$HOME/.hpc/gcc/gmp/6.2.0 \
--with-mpfr=$HOME/.hpc/gcc/mpfr/4.1.0 \
--docdir=$myprefix/doc/mpc-1.2.0
make -j$(nproc)
make install
GCC 9.2.0
mkdir -p ~/tmpHPC
cd $HOME/tmpHPC
myprefix=$HOME/.hpc/gcc/9.2.0
export PATH
export LIBRARY_PATH=/usr/lib64/:$LIBRARY_PATH
wget https://ftp.gnu.org/gnu/gcc/gcc-9.2.0/gcc-9.2.0.tar.xz
tar xfv gcc-9.2.0.tar.xz
cd gcc-9.2.0
case $(uname -m) in
x86_64)
sed -e '/m64=/s/lib64/lib/' \
-i.orig gcc/config/i386/t-linux64
;;
esac
mkdir -p build &&
cd build &&
SED=sed \
../configure --prefix=$myprefix \
--enable-languages=c,c++,fortran \
--disable-multilib \
--with-gmp=$HOME/.hpc/gcc/gmp/6.2.0 \
--with-mpfr=$HOME/.hpc/gcc/mpfr/4.1.0 \
--with-mpc=$HOME/.hpc/gcc/mpc/1.2.0 \
--disable-bootstrap \
--with-system-zlib
make -j$(nproc)
ml load gcc/9.2.0
Autotools
Following the standard approach outlined in the GNU Autotools FAQ:
mkdir -p ~/tmpHPC
cd $HOME/tmpHPC
myprefix=$HOME/.hpc/autotools
export PATH
wget http://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.gz
wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
wget http://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.gz
wget http://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.gz
gzip -dc m4-1.4.18.tar.gz | tar xvf -
gzip -dc autoconf-2.69.tar.gz | tar xvf -
gzip -dc automake-1.16.2.tar.gz | tar xvf -
gzip -dc libtool-2.4.6.tar.gz | tar xvf -
cd m4-1.4.18
./configure -C --prefix=$myprefix/m4/1.4.18 && make -j$(nproc) && make install
cd ../autoconf-2.69
./configure -C --prefix=$myprefix/autoconf/2.69 && make -j$(nproc) && make install
cd ../automake-1.16.2
./configure -C --prefix=$myprefix/automake/1.16.2 && make -j$(nproc) && make install
cd ../libtool-2.4.6
./configure -C --prefix=$myprefix/libtool/2.4.6 && make -j$(nproc) && make install
ml load autotools/autotools
We also need the archive.
myprefix=$HOME/.hpc/autotools
git clone git://git.sv.gnu.org/autoconf-archive.git
cd autoconf-archive
Combined with the lmod manual gives rise to the following definiton (roughly the same for each one):
local home = os.getenv("HOME")
local version = myModuleVersion()
local pkgName = myModuleName()
local pkg = pathJoin(home,".hpc",pkgName,version,"bin")
prepend_path("PATH", pkg)
We will no longer bother with the module definitions for the rest of this post, as they are handled and documented in the repo.
Perl
This is essentially the setup from the main docs.
# Get Perl
curl -L http://xrl.us/installperlnix | bash
# Use Perl
ml use perl/5.28.0
cpanm ExtUtils::MakeMaker # For git
ml load perl/5.28.0
Git
This is very similar to the previous approach. However, since by default the system perl
was being picked up, some slight changes have been made.
myprefix=$HOME/.hpc/git/2.9.5
PATH=$myprefix/bin:$PATH
export PATH
wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.9.5.tar.gz
gzip -dc git-2.9.5.tar.gz | tar xvf -
cd git-2.9.5
./configure --with-perl=$(which perl) -C --prefix=$myprefix
make -j $(nproc)
make install
ml load git/2.9.5
Caveat
Also, for TRAMP, we would prefer having a more constant path, so we can set up a symlink:
mkdir ~/.hpc/bin
ln ~/.hpc/git/2.9.5/bin/git ~/.hpc/bin/git
Boost
The boost website is utterly incomprehensible. As is the documentation. Also, fun fact, the move from svn
makes things worse. Thankfully, a quick dive into the slightly better Github wiki led to this nugget:
git clone --recursive https://github.com/boostorg/boost.git
cd boost
git checkout tags/boost-1.73.0 # or whatever branch you want to use
./bootstrap.sh
./b2 headers
This means we’re almost done!
./b2
./b2 install --prefix=$HOME/.hpc/boost/boost-1.73.0
ml load boost/boost-1.73.0
Pkg-Config
myprefix=$HOME/.hpc/pkg-config/0.29.2
wget https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
gzip -dc pkg-config-0.29.2.tar.gz | tar xvf -
cd pkg-config-0.29.2
./configure --prefix=$myprefix --with-internal-glib --disable-host-tool --docdir=$myprefix/share/doc/pkg-config-0.29.2
mkdir $myprefix/lib
make -j $(nproc)
make install
ml load pkg-config/0.29.2
Zlib
myprefix=$HOME/.hpc/zlib/1.2.11
wget http://zlib.net/zlib-1.2.11.tar.gz
gzip -dc zlib-1.2.11.tar.gz | tar xvf -
cd zlib-1.2.11
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load zlib/1.2.11
XZ Utils
myprefix=$HOME/.hpc/xz/5.2.5
wget https://tukaani.org/xz/xz-5.2.5.tar.gz
gzip -dc xz-5.2.5.tar.gz | tar xvf -
cd xz-5.2.5
./configure --prefix=$myprefix --enable-threads=yes
make -j $(nproc)
make install
ml load xz/5.2.5
OpenSSL
myprefix=$HOME/.hpc/openssl/1.1.1d
wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz
gzip -dc openssl-1.1.1d.tar.gz | tar xvf -
cd openssl-1.1.1d
./config --prefix=$myprefix --openssldir=$myprefix/etc/ssl shared zlib-dynamic
make -j $(nproc)
make install
ml load openssl/1.1.1d
Cmake
myprefix=$HOME/.hpc/cmake/3.18.1
wget https://github.com/Kitware/CMake/releases/download/v3.18.1/cmake-3.18.1.tar.gz
gzip -dc cmake-3.18.1.tar.gz | tar xvf -
cd cmake-3.18.1
./bootstrap --prefix=$myprefix
make -j $(nproc)
make install
ml load cmake/3.18.1
GNU-Make
myprefix=$HOME/.hpc/make/4.3
wget http://ftp.gnu.org/gnu/make/make-4.3.tar.gz
gzip -dc make-4.3.tar.gz | tar xvf -
cd make-4.3
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load make/4.3
Brotli
myprefix=$HOME/.hpc/brotli/1.0.1
git clone https://github.com/bagder/libbrotli
cd libbrotli
./autogen.sh
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load brotli/1.0.1
ncurses
We will need to manually ensure the paths for pkg-config
are in a feasible location.
myprefix=$HOME/.hpc/ncurses/6.2
wget https://invisible-mirror.net/archives/ncurses/ncurses-6.2.tar.gz
gzip -dc ncurses-6.2.tar.gz | tar xvf -
cd ncurses-6.2
./configure --prefix=$myprefix --enable-widec --enable-pc-files --with-shared
make -j $(nproc)
make install
mkdir pkgconfig
cp misc/formw.pc misc/menuw.pc misc/ncurses++w.pc misc/ncursesw.pc misc/panelw.pc pkgconfig/
mv pkgconfig $myprefix/lib/
ml load ncurses/6.2
texinfo
myprefix=$HOME/.hpc/texinfo/6.7
wget http://ftp.gnu.org/gnu/texinfo/texinfo-6.7.tar.gz
gzip -dc texinfo-6.7.tar.gz | tar xvf -
cd texinfo-6.7
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load texinfo/6.7
gperf
myprefix=$HOME/.hpc/gperf/3.1
wget http://ftp.gnu.org/gnu/gperf/gperf-3.1.tar.gz
gzip -dc gperf-3.1.tar.gz | tar xvf -
cd gperf-3.1
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load gperf/3.1
libseccomp
There is a bug, which requires modifying src/system.c
to change __NR_seccomp
to _nr_seccomp
.
myprefix=$HOME/.hpc/libseccomp/2.5.0
git clone https://github.com/seccomp/libseccomp
cd libseccomp
git checkout tags/v2.5.0
./autogen.sh
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load libseccomp/2.5.0
Alternatively, it is easier to work with an older version.
myprefix=$HOME/.hpc/libseccomp/2.4.4
wget https://github.com/seccomp/libseccomp/releases/download/v2.4.4/libseccomp-2.4.4.tar.gz
tar xfv libseccomp-2.4.4.tar.gz
cd libseccomp-2.4.4
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load libseccomp/2.4.4
BDWGC
myprefix=$HOME/.hpc/bdwgc/8.0.4
wget https://github.com/ivmai/bdwgc/releases/download/v8.0.4/gc-8.0.4.tar.gz
gzip -dc gc-8.0.4.tar.gz | tar xvf -
cd gc-8.0.4
./configure --prefix=$myprefix --enable-cplusplus
make -j $(nproc)
make install
ml load bdwgc/8.0.4
pcre
We will prep both pcre2
and pcre
.
myprefix=$HOME/.hpc/pcre2/10.35 wget https://ftp.pcre.org/pub/pcre/pcre2-10.35.tar.gz gzip -dc pcre2-10.35.tar.gz | tar xvf - cd pcre2-10.35 ./configure –prefix=$myprefix \
–enable-pcre2-16 \
–enable-pcre2-32 \
–enable-pcre2grep-libz make -j $(nproc) make install ml load pcre2/10.35
myprefix=$HOME/.hpc/pcre/8.44 wget https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz gzip -dc pcre-8.44.tar.gz | tar xvf - cd pcre-8.44 ./configure –prefix=$myprefix \
–enable-pcre-16 \
–enable-pcre-32 \
–enable-pcregrep-libz make -j $(nproc) make install ml load pcre/8.44
bison
myprefix=$HOME/.hpc/bison/3.7.1
wget http://ftp.gnu.org/gnu/bison/bison-3.7.1.tar.gz
gzip -dc bison-3.7.1.tar.gz | tar xvf -
cd bison-3.7.1
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load bison/3.7.1
flex
myprefix=$HOME/.hpc/flex/2.6.4
wget https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.gz
gzip -dc flex-2.6.4.tar.gz | tar xvf -
cd flex-2.6.4
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load flex/2.6.4
jq
myprefix=$HOME/.hpc/jq/1.6
git clone https://github.com/stedolan/jq.git
cd jq
git submodule update --init
git checkout tags/jq-1.6
autoreconf -fi
./configure --prefix=$myprefix --with-oniguruma=builtin
make -j $(nproc)
make install
ml load jq/1.6
bzip2
Needed to manually configure it as shown here
myprefix=$HOME/.hpc/bzip2/1.0.8
wget https://www.sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz
gzip -dc bzip2-1.0.8.tar.gz | tar xvf -
cd bzip2-1.0.8
make -f Makefile-libbz2_so
ln -sf libbz2.so.1.0 libbz2.so
mkdir -p $myprefix/include
mkdir -p $myprefix/lib
cp -avf bzlib.h $myprefix/include
cp -avf libbz2.so* $myprefix/lib
make install PREFIX=$myprefix
ml load bzip2/1.0.8
sqlite
myprefix=$HOME/.hpc/sqlite/3.32.3
wget https://www.sqlite.org/2020/sqlite-autoconf-3320300.tar.gz
gzip -dc sqlite-autoconf-3320300.tar.gz | tar xvf -
cd sqlite-autoconf-3320300
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load sqlite/3.32.3
editline
myprefix=$HOME/.hpc/editline/1.17.1
wget https://github.com/troglobit/editline/releases/download/1.17.1/editline-1.17.1.tar.gz
gzip -dc editline-1.17.1.tar.gz | tar xvf -
cd editline-1.17.1
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load editline/1.17.1
Miniconda
We don’t need this very much, but it is still useful for some edge cases, mainly revolving around jupyter
infrastructure.
cd $HOME
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod +x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh
# Do not allow it to mess up the shell rc files
eval "$($HOME/miniconda3/bin/conda shell.zsh hook)"
Note that we will prefer the manual evaluation since it can be handled in the lmod
file.
Applications
Libraries and git
aside, there are some tools we might want to have.
ag
The silver searcher, along with rg
is very useful to have.
myprefix=$HOME/.hpc/the_silver_searcher/2.2.0
wget https://geoff.greer.fm/ag/releases/the_silver_searcher-2.2.0.tar.gz
gzip -dc the_silver_searcher-2.2.0.tar.gz | tar xvf -
cd the_silver_searcher-2.2.0
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load the_silver_searcher/2.2.0
Neovim
myprefix=$HOME/.hpc/nvim/0.5.0
wget https://github.com/neovim/neovim/releases/download/nightly/nvim.appimage
chmod +x nvim.appimage
./nvim.appimage --appimage-extract
mkdir -p $myprefix
mv squashfs-root/usr/* $myprefix
ml load nvim/0.5.0
Tmux
myprefix=$HOME/.hpc/tmux/3.1b
wget https://github.com/tmux/tmux/releases/download/3.1b/tmux-3.1b-x86_64.AppImage
chmod +x tmux-3.1b-x86_64.AppImage
rm -rf squashfs-root
./tmux-3.1b-x86_64.AppImage --appimage-extract
mkdir -p $myprefix
mv squashfs-root/usr/bin squashfs-root/usr/lib squashfs-root/usr/share $myprefix
ml load tmux/3.1b
Zsh
More of an update than a requirement.
myprefix=$HOME/.hpc/zsh/5.8
wget https://github.com/zsh-users/zsh/archive/zsh-5.8.tar.gz
gzip -dc zsh-5.8.tar.gz | tar xvf -
cd zsh-zsh-5.8
./configure --prefix=$myprefix
make -j $(nproc)
make install
ml load zsh/5.8
Conclusion
Having composed a bunch of these, I will of course try to somehow get nix
up and running so it can bootstrap itself and allow me to work in peace. I might also eventually create shell scripts to automate updating these, but hopefully I can set up nix
and not re-create package manager logic in lua
.