14 minutes
Written: 2020-08-09 02:29 +0000
HPC Dotfiles and LMod
Background
My move away from the powerful, but unimaginatively named HPC clusters of IITK 1 brought me in close contact with the Lua based 2 lmod module system. Rather than fall into the rabbit hole of brew we will leverage the existing system to add our new libraries. Not finding any good collections of these composable environments, and having failed once before to install Nix as a user without admin access, I decided to start my own collection of Lmod recipies. The rest of this post details the installation proceedure to be carried out in conjunction with the hzHPC_lmod repo.
Setting Up
These are reproduced from the repo for completeness.
1git clone https://github.com/kobus-v-schoor/dotgit.git
2mkdir -p ~/.local/bin
3cp -r dotgit/old/bin/dotgit* ~/.local/bin
4cat dotgit/old/bin/bash_completion >> ~/.bash_completion
5rm -rf dotgit
I actually strongly suggest using a target from my Dotfiles in conjunction with this, but it isn’t really required, so:
1~/.local/bin/dotgit restore hzhpc
Note that because of the suggested separation, I have not opted to setup a shell or even ensure that there are scripts here to help keep module in your path. Those are in my Dotfiles.
If, you opt to not use these dotfiles, then do not run the ml load commands.
Basic Setup
We will first load an environment with basic packages (autotools). This is a stop-gap solution.
MicroMamba
We will set up a micromamba installation to get git and other base utilities.
1mkdir -p ~/.local/bin
2export PATH=$HOME/.local/bin:$PATH
3wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba
4mv bin/micromamba ~/.local/bin
5rm -rf bin
6micromamba shell init -s bash -p $HOME/micromamba
7. ~/.bashrc
Now we can use mamba to get a temporary interactive environment
1# Spack requirements
2PKGS="git patch curl make tar gzip unzip bzip2 xz zstd subversion lsof m4"
3mkdir ~/.mamba_venvs
4micromamba create -p ~/.mamba_venvs/intBase $PKGS
5micromamba activate ~/.mamba_venvs/intBase
For subsequent logins we can simply run:
1export PATH=$HOME/.local/bin:$PATH
2. ~/.bashrc
3micromamba activate ~/.mamba_venvs/intBase
Configuration
A simple .condarc will suffice for the above.
1channels:
2 - conda-forge
3 - defaults
4channel_priority: disabled
Warning
Note that once the base packages have been installed…
We must unload our micromamba environment!!
1micromamba deactivate
It is recommended to use a minimal set of micromamba packages.
LMod Libraries
Note that: garpur and elja already has
lmod
The scripts in this post will also be part of the repo, but keep in mind that these are not meant to be robust ways to install anything, and every command should be run by hand because things will probably break badly. We keep all sources in $HOME/tmphpc and install everything relative to $HOME/.hpc. Paths are managed by the lmod system.
1export hpcroot=$HOME/tmphpc
2mkdir -p $hpcroot
Combining our paths with the lmod manual gives rise to the following definiton (roughly the same for each one):
1local home = os.getenv("HOME")
2local version = myModuleVersion()
3local pkgName = myModuleName()
4local pkg = pathJoin(home,".hpc",pkgName,version,"bin")
5prepend_path("PATH", pkg)
We will no longer bother with the module definitions for the rest of this post, as they are handled and documented in the repo.
1# Setting up hzhpc modules
2cd tmphpc
3git clone https://github.com/kobus-v-schoor/dotgit.git
4mkdir -p ~/.local/bin
5cp -r dotgit/old/bin/dotgit* ~/.local/bin
6cat dotgit/bin/bash_completion >> ~/.bash_completion
7rm -rf dotgit
8~/.local/bin/dotgit restore hzhpc
Patch
This is a basic utility we need before getting to anything else.
1cd $hpcroot
2myprefix=$HOME/.hpc/patch/2.7.2
3wget http://ftp.gnu.org/gnu/patch/patch-2.7.2.tar.gz
4gzip -dc patch-2.7.2.tar.gz | tar xvf -
5cd patch-2.7.2
6./configure --prefix=$myprefix
7make -j$(nproc) && make install
8ml load patch
Help2man
1cd $hpcroot
2myprefix=$HOME/.hpc/help2man/1.48.3
3wget https://gnuftp.uib.no/help2man/help2man-1.48.3.tar.xz
4tar xfv help2man-1.48.3.tar.xz
5cd help2man-1.48.3
6./configure --prefix=$myprefix
7make -j$(nproc)
8make install
9ml load help2man/1.48.3
Perl
This is essentially the setup from the main docs.
1# Get Perl
2curl -L http://xrl.us/installperlnix | bash
3cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
4cpanm ExtUtils::MakeMaker --force # For git
5cpanm Thread::Queue # For automake
6ml load perl/5.28.0
Autotools
M4
This requires special considerations for glibc greater than 2.28 (true for compilers like gcc8 and above).
1cd $HOME/tmphpc
2myprefix=$HOME/.hpc/autotools
3wget http://ftp.gnu.org/gnu/m4/m4-1.4.18.tar.gz
4gzip -dc m4-1.4.18.tar.gz | tar xvf -
5cd m4-1.4.18
6# From http://git.openembedded.org/openembedded-core/commit/meta/recipes-devtools/m4/m4/m4-1.4.18-glibc-change-work-around.patch
7# From https://askubuntu.com/a/1112101/690387
8wget http://git.openembedded.org/openembedded-core/plain/meta/recipes-devtools/m4/m4/m4-1.4.18-glibc-change-work-around.patch
9patch -p 1 < m4-1.4.18-glibc-change-work-around.patch
10./configure -C --prefix=$myprefix/m4/1.4.18 && make -j$(nproc) && make install
Patch contents
The patch is needed since glibc 2.28 and above have changed the nesting levels. The patch is effectively backported from the upstream repositories. Some more context is here.
1update for glibc libio.h removal in 2.28+
2
3see
4https://src.fedoraproject.org/rpms/m4/c/814d592134fad36df757f9a61422d164ea2c6c9b?branch=master
5
6Upstream-Status: Backport [https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=4af4a4a718]
7Signed-off-by: Khem Raj <raj.khem@gmail.com>
8
9Index: m4-1.4.18/lib/fflush.c
10===================================================================
11--- m4-1.4.18.orig/lib/fflush.c
12+++ m4-1.4.18/lib/fflush.c
13@@ -33,7 +33,7 @@
14 #undef fflush
15
16
17-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
18+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
19
20 /* Clear the stream's ungetc buffer, preserving the value of ftello (fp). */
21 static void
22@@ -72,7 +72,7 @@ clear_ungetc_buffer (FILE *fp)
23
24 #endif
25
26-#if ! (defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */)
27+#if ! (defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */)
28
29 # if (defined __sferror || defined __DragonFly__ || defined __ANDROID__) && defined __SNPT
30 /* FreeBSD, NetBSD, OpenBSD, DragonFly, Mac OS X, Cygwin, Android */
31@@ -148,7 +148,7 @@ rpl_fflush (FILE *stream)
32 if (stream == NULL || ! freading (stream))
33 return fflush (stream);
34
35-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
36+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
37
38 clear_ungetc_buffer_preserving_position (stream);
39
40Index: m4-1.4.18/lib/fpending.c
41===================================================================
42--- m4-1.4.18.orig/lib/fpending.c
43+++ m4-1.4.18/lib/fpending.c
44@@ -32,7 +32,7 @@ __fpending (FILE *fp)
45 /* Most systems provide FILE as a struct and the necessary bitmask in
46 <stdio.h>, because they need it for implementing getc() and putc() as
47 fast macros. */
48-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
49+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
50 return fp->_IO_write_ptr - fp->_IO_write_base;
51 #elif defined __sferror || defined __DragonFly__ || defined __ANDROID__
52 /* FreeBSD, NetBSD, OpenBSD, DragonFly, Mac OS X, Cygwin, Android */
53Index: m4-1.4.18/lib/fpurge.c
54===================================================================
55--- m4-1.4.18.orig/lib/fpurge.c
56+++ m4-1.4.18/lib/fpurge.c
57@@ -62,7 +62,7 @@ fpurge (FILE *fp)
58 /* Most systems provide FILE as a struct and the necessary bitmask in
59 <stdio.h>, because they need it for implementing getc() and putc() as
60 fast macros. */
61-# if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
62+# if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
63 fp->_IO_read_end = fp->_IO_read_ptr;
64 fp->_IO_write_ptr = fp->_IO_write_base;
65 /* Avoid memory leak when there is an active ungetc buffer. */
66Index: m4-1.4.18/lib/freadahead.c
67===================================================================
68--- m4-1.4.18.orig/lib/freadahead.c
69+++ m4-1.4.18/lib/freadahead.c
70@@ -25,7 +25,7 @@
71 size_t
72 freadahead (FILE *fp)
73 {
74-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
75+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
76 if (fp->_IO_write_ptr > fp->_IO_write_base)
77 return 0;
78 return (fp->_IO_read_end - fp->_IO_read_ptr)
79Index: m4-1.4.18/lib/freading.c
80===================================================================
81--- m4-1.4.18.orig/lib/freading.c
82+++ m4-1.4.18/lib/freading.c
83@@ -31,7 +31,7 @@ freading (FILE *fp)
84 /* Most systems provide FILE as a struct and the necessary bitmask in
85 <stdio.h>, because they need it for implementing getc() and putc() as
86 fast macros. */
87-# if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
88+# if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
89 return ((fp->_flags & _IO_NO_WRITES) != 0
90 || ((fp->_flags & (_IO_NO_READS | _IO_CURRENTLY_PUTTING)) == 0
91 && fp->_IO_read_base != NULL));
92Index: m4-1.4.18/lib/fseeko.c
93===================================================================
94--- m4-1.4.18.orig/lib/fseeko.c
95+++ m4-1.4.18/lib/fseeko.c
96@@ -47,7 +47,7 @@ fseeko (FILE *fp, off_t offset, int when
97 #endif
98
99 /* These tests are based on fpurge.c. */
100-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
101+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
102 if (fp->_IO_read_end == fp->_IO_read_ptr
103 && fp->_IO_write_ptr == fp->_IO_write_base
104 && fp->_IO_save_base == NULL)
105@@ -123,7 +123,7 @@ fseeko (FILE *fp, off_t offset, int when
106 return -1;
107 }
108
109-#if defined _IO_ftrylockfile || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
110+#if defined _IO_EOF_SEEN || __GNU_LIBRARY__ == 1 /* GNU libc, BeOS, Haiku, Linux libc5 */
111 fp->_flags &= ~_IO_EOF_SEEN;
112 fp->_offset = pos;
113 #elif defined __sferror || defined __DragonFly__ || defined __ANDROID__
114Index: m4-1.4.18/lib/stdio-impl.h
115===================================================================
116--- m4-1.4.18.orig/lib/stdio-impl.h
117+++ m4-1.4.18/lib/stdio-impl.h
118@@ -18,6 +18,12 @@
119 the same implementation of stdio extension API, except that some fields
120 have different naming conventions, or their access requires some casts. */
121
122+/* Glibc 2.28 made _IO_IN_BACKUP private. For now, work around this
123+ problem by defining it ourselves. FIXME: Do not rely on glibc
124+ internals. */
125+#if !defined _IO_IN_BACKUP && defined _IO_EOF_SEEN
126+# define _IO_IN_BACKUP 0x100
127+#endif
128
129 /* BSD stdio derived implementations. */
Auto{Conf,Make} and Libtool
This follows the standard approach outlined in the GNU Autotools FAQ:
1cd $hpcroot
2myprefix=$HOME/.hpc/autotools
3export PATH
4wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.69.tar.gz
5wget http://ftp.gnu.org/gnu/automake/automake-1.16.2.tar.gz
6wget http://ftp.gnu.org/gnu/libtool/libtool-2.4.6.tar.gz
7wget http://ftp.gnu.org/gnu/gettext/gettext-0.20.tar.gz
8wget https://gnuftp.uib.no/autoconf-archive/autoconf-archive-2021.02.19.tar.xz
9tar xfv autoconf-archive-2021.02.19.tar.xz
10gzip -dc autoconf-2.69.tar.gz | tar xvf -
11gzip -dc automake-1.16.2.tar.gz | tar xvf -
12gzip -dc libtool-2.4.6.tar.gz | tar xvf -
13gzip -dc gettext-0.20.tar.gz | tar xvf -
14cd autoconf-2.69
15./configure -C --prefix=$myprefix/autoconf/2.69 && make -j$(nproc) && make install
16cd ../automake-1.16.2
17./configure -C --prefix=$myprefix/automake/1.16.2 --docdir=$myprefix/automake/1.16.2/share/doc/automake-1.16.2 && make -j$(nproc) && make install
18cd ../autoconf-archive-2021.02.19
19./configure -C --prefix=$myprefix/automake/1.16.2
20cd ../libtool-2.4.6
21./configure -C --disable-static --prefix=$myprefix/libtool/2.4.6 && make -j$(nproc) && make install
22cd ../gettext-0.20
23./configure -C --prefix=$myprefix/gettext/0.20 && make -j$(nproc) && make install
24ml load autotools/autotools
GMP
1cd $hpcroot
2myprefix=$HOME/.hpc/gcc/gmp/6.2.0
3export PATH
4wget https://gmplib.org/download/gmp/gmp-6.2.0.tar.xz
5tar xfv gmp-6.2.0.tar.xz
6cd gmp-6.2.0
7./configure --prefix=$myprefix \
8 --enable-cxx \
9 --docdir=$myprefix/doc/gmp-6.2.0
10make -j$(nproc)
11make install
12ml load gcc/gmp/6.2.0
MPFR
1cd $hpcroot
2myprefix=$HOME/.hpc/gcc/mpfr/4.1.0
3export PATH
4wget https://www.mpfr.org/mpfr-current/mpfr-4.1.0.tar.xz
5tar xfv mpfr-4.1.0.tar.xz
6cd mpfr-4.1.0
7./configure --prefix=$myprefix \
8 --enable-thread-safe \
9 --with-gmp=$HOME/.hpc/gcc/gmp/6.2.0 \
10 --docdir=$myprefix/doc/mpfr-4.1.0
11make -j$(nproc)
12make install
13ml load gcc/mpfr/4.1.0
MPC
1cd $hpcroot
2myprefix=$HOME/.hpc/gcc/mpc/1.2.0
3export PATH
4wget https://ftp.gnu.org/gnu/mpc/mpc-1.2.0.tar.gz
5tar xfv mpc-1.2.0.tar.gz
6cd mpc-1.2.0
7./configure --prefix=$myprefix \
8 --with-gmp=$HOME/.hpc/gcc/gmp/6.2.0 \
9 --with-mpfr=$HOME/.hpc/gcc/mpfr/4.1.0 \
10 --docdir=$myprefix/doc/mpc-1.2.0
11make -j$(nproc)
12make install
13ml load gcc/mpc/1.2.0
GCC 9.2.0
1cd $hpcroot
2ml load gcc/gmp gcc/mpfr gcc/mpc
3myprefix=$HOME/.hpc/gcc/9.2.0
4wget https://ftp.gnu.org/gnu/gcc/gcc-9.2.0/gcc-9.2.0.tar.xz
5tar xfv gcc-9.2.0.tar.xz
6cd gcc-9.2.0
7case $(uname -m) in
8 x86_64)
9 sed -e '/m64=/s/lib64/lib/' \
10 -i.orig gcc/config/i386/t-linux64
11 ;;
12esac
13mkdir -p build &&
14cd build &&
15
16SED=sed \
17../configure --prefix=$myprefix \
18 --enable-languages=c,c++,fortran \
19 --disable-multilib \
20 --with-gmp=$HOME/.hpc/gcc/gmp/6.2.0 \
21 --with-mpfr=$HOME/.hpc/gcc/mpfr/4.1.0 \
22 --with-mpc=$HOME/.hpc/gcc/mpc/1.2.0 \
23 --disable-bootstrap \
24 --with-system-zlib
25export PATH
26unset LIBRARY_PATH
27export LIBRARY_PATH=/usr/lib64/
28mkdir -p -- .deps
29make -j$(nproc)
30make install
31ml load gcc/9.2.0
Pkg-Config
1cd $hpcroot
2myprefix=$HOME/.hpc/pkg-config/0.29.2
3wget https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
4gzip -dc pkg-config-0.29.2.tar.gz | tar xvf -
5cd pkg-config-0.29.2
6./configure --prefix=$myprefix --with-internal-glib --disable-host-tool --docdir=$myprefix/share/doc/pkg-config-0.29.2
7mkdir $myprefix/lib
8make -j $(nproc)
9make install
10ml load pkg-config/0.29.2
Zlib
1cd $hpcroot
2myprefix=$HOME/.hpc/zlib/1.2.11
3wget http://zlib.net/zlib-1.2.11.tar.gz
4gzip -dc zlib-1.2.11.tar.gz | tar xvf -
5cd zlib-1.2.11
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load zlib/1.2.11
XZ Utils
1cd $hpcroot
2myprefix=$HOME/.hpc/xz/5.2.5
3wget https://tukaani.org/xz/xz-5.2.5.tar.gz
4gzip -dc xz-5.2.5.tar.gz | tar xvf -
5cd xz-5.2.5
6./configure --prefix=$myprefix --enable-threads=yes
7make -j $(nproc)
8make install
9ml load xz/5.2.5
OpenSSL
1cd $hpcroot
2myprefix=$HOME/.hpc/openssl/1.1.1d
3wget https://www.openssl.org/source/openssl-1.1.1d.tar.gz
4gzip -dc openssl-1.1.1d.tar.gz | tar xvf -
5cd openssl-1.1.1d
6./config --prefix=$myprefix --openssldir=$myprefix/etc/ssl shared zlib-dynamic
7make -j $(nproc)
8make install
9ml load openssl/1.1.1d
cURL
1cd $hpcroot
2myprefix=$HOME/.hpc/curl/7.76.0
3wget https://curl.haxx.se/download/curl-7.76.0.tar.xz
4tar xfv curl-7.76.0.tar.xz
5cd curl-7.76.0
6./configure --prefix=$myprefix \
7 --disable-static \
8 --enable-threaded-resolver \
9 --with-openssl
10make -j $(nproc)
11make install
12ml load curl/7.76.0
Git
This is very similar to the previous approach. However, since by default the system perl was being picked up, some slight changes have been made.
1cd $hpcroot
2myprefix=$HOME/.hpc/git/2.9.5
3PATH=$myprefix/bin:$PATH
4export PATH
5wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.9.5.tar.gz
6gzip -dc git-2.9.5.tar.gz | tar xvf -
7cd git-2.9.5
8./configure --with-perl=$(which perl) --with-curl=$(which curl) -C --prefix=$myprefix
9make -j $(nproc)
10make install
11ml load git/2.9.5
12ml unload openssl # System will suffice
13ml unload curl # System will suffice
Caveat
Also, for TRAMP, we would prefer having a more constant path, so we can set up a symlink:
1mkdir ~/.hpc/bin
2ln ~/.hpc/git/2.9.5/bin/git ~/.hpc/bin/git
Boost
The boost website is utterly incomprehensible. As is the documentation. Also, fun fact, the move from svn makes things worse. This is best installed with the standard compiler present, since the B2 engine detection seems to be very hit and miss. Thankfully, a quick dive into the slightly better Github wiki led to this nugget:
1cd $hpcroot
2git clone --recursive https://github.com/boostorg/boost.git
3cd boost
4git checkout tags/boost-1.76.0 # or whatever branch you want to use
5./bootstrap.sh
6./b2 headers
This means we’re almost done!
1./b2
2./b2 install --prefix=$HOME/.hpc/boost/boost-1.76.0
3ml load boost/boost-1.76.0
Cmake
1cd $hpcroot
2myprefix=$HOME/.hpc/cmake/3.20.1
3wget https://github.com/Kitware/CMake/releases/download/v3.20.1/cmake-3.20.1.tar.gz
4gzip -dc cmake-3.20.1.tar.gz | tar xvf -
5cd cmake-3.20.1
6CXXFLAGS="-std=c++11" CC=$(which gcc) CXX=$(which g++) ./bootstrap --prefix=$myprefix --system-libs
7make -j $(nproc)
8make install
9ml load cmake/3.20.1
GNU-Make
1cd $hpcroot
2myprefix=$HOME/.hpc/make/4.3
3wget http://ftp.gnu.org/gnu/make/make-4.3.tar.gz
4gzip -dc make-4.3.tar.gz | tar xvf -
5cd make-4.3
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load make/4.3
Brotli
1cd $hpcroot
2myprefix=$HOME/.hpc/brotli/1.0.1
3git clone https://github.com/bagder/libbrotli
4cd libbrotli
5libtoolize
6aclocal
7autoheader
8./autogen.sh
9./configure --prefix=$myprefix
10make -j $(nproc)
11make install
12ml load brotli/1.0.1
ncurses
We will need to manually ensure the paths for pkg-config are in a feasible location.
1cd $hpcroot
2myprefix=$HOME/.hpc/ncurses/6.2
3wget https://invisible-mirror.net/archives/ncurses/ncurses-6.2.tar.gz
4gzip -dc ncurses-6.2.tar.gz | tar xvf -
5cd ncurses-6.2
6./configure --prefix=$myprefix --enable-widec --enable-pc-files --with-shared
7make -j $(nproc)
8make install
9mkdir pkgconfig
10cp misc/formw.pc misc/menuw.pc misc/ncurses++w.pc misc/ncursesw.pc misc/panelw.pc pkgconfig/
11mv pkgconfig $myprefix/lib/
12ml load ncurses/6.2
texinfo
1cd $hpcroot
2myprefix=$HOME/.hpc/texinfo/6.7
3wget http://ftp.gnu.org/gnu/texinfo/texinfo-6.7.tar.gz
4gzip -dc texinfo-6.7.tar.gz | tar xvf -
5cd texinfo-6.7
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load texinfo/6.7
gperf
1cd $hpcroot
2myprefix=$HOME/.hpc/gperf/3.1
3wget http://ftp.gnu.org/gnu/gperf/gperf-3.1.tar.gz
4gzip -dc gperf-3.1.tar.gz | tar xvf -
5cd gperf-3.1
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load gperf/3.1
libseccomp
There is a bug, which requires modifying src/system.c to change __NR_seccomp to _nr_seccomp.
1cd $hpcroot
2myprefix=$HOME/.hpc/libseccomp/2.5.0
3git clone https://github.com/seccomp/libseccomp
4cd libseccomp
5git checkout tags/v2.5.0
6./autogen.sh
7./configure --prefix=$myprefix
8make -j $(nproc)
9make install
10ml load libseccomp/2.5.0
Alternatively, it is easier to work with an older version.
1cd $hpcroot
2myprefix=$HOME/.hpc/libseccomp/2.4.4
3wget https://github.com/seccomp/libseccomp/releases/download/v2.4.4/libseccomp-2.4.4.tar.gz
4tar xfv libseccomp-2.4.4.tar.gz
5cd libseccomp-2.4.4
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load libseccomp/2.4.4
BDWGC
1cd $hpcroot
2myprefix=$HOME/.hpc/bdwgc/8.0.4
3wget https://github.com/ivmai/bdwgc/releases/download/v8.0.4/gc-8.0.4.tar.gz
4gzip -dc gc-8.0.4.tar.gz | tar xvf -
5cd gc-8.0.4
6./configure --prefix=$myprefix --enable-cplusplus
7make -j $(nproc)
8make install
9ml load bdwgc/8.0.4
pcre
We will prep both pcre2 and pcre.
1cd $hpcroot
2myprefix=$HOME/.hpc/pcre2/10.35
3wget https://ftp.pcre.org/pub/pcre/pcre2-10.35.tar.gz
4gzip -dc pcre2-10.35.tar.gz | tar xvf -
5cd pcre2-10.35
6./configure --prefix=$myprefix \
7 --enable-pcre2-16 \
8 --enable-pcre2-32 \
9 --enable-pcre2grep-libz
10make -j $(nproc)
11make install
12ml load pcre2/10.35
1cd $hpcroot
2myprefix=$HOME/.hpc/pcre/8.44
3wget https://ftp.pcre.org/pub/pcre/pcre-8.44.tar.gz
4gzip -dc pcre-8.44.tar.gz | tar xvf -
5cd pcre-8.44
6./configure --prefix=$myprefix \
7 --enable-pcre-16 \
8 --enable-pcre-32 \
9 --enable-pcregrep-libz
10make -j $(nproc)
11make install
12ml load pcre/8.44
bison
1cd $hpcroot
2myprefix=$HOME/.hpc/bison/3.7.1
3wget http://ftp.gnu.org/gnu/bison/bison-3.7.1.tar.gz
4gzip -dc bison-3.7.1.tar.gz | tar xvf -
5cd bison-3.7.1
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load bison/3.7.1
flex
1cd $hpcroot
2myprefix=$HOME/.hpc/flex/2.6.4
3wget https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.gz
4gzip -dc flex-2.6.4.tar.gz | tar xvf -
5cd flex-2.6.4
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load flex/2.6.4
jq
We first need the oniguruma regular expression library.
1cd $hpcroot
2myprefix=$HOME/.hpc/oniguruma/6.9.7.1
3git clone git@github.com:kkos/oniguruma.git
4cd oniguruma
5git checkout tags/v6.9.7.1
6libtoolize
7autoreconf -i
8./configure --prefix=$myprefix
9make -j $(nproc)
10make install
11ml load oniguruma/6.9.7.1
We will actually use the binary.
1cd $HOME/.local/bin
2wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64
3chmod +x jq-linux64
4mv jq-linux64 jq
bzip2
Needed to manually configure it as shown here
1cd $hpcroot
2myprefix=$HOME/.hpc/bzip2/1.0.8
3wget https://www.sourceware.org/pub/bzip2/bzip2-1.0.8.tar.gz
4gzip -dc bzip2-1.0.8.tar.gz | tar xvf -
5cd bzip2-1.0.8
6make -f Makefile-libbz2_so
7ln -sf libbz2.so.1.0 libbz2.so
8mkdir -p $myprefix/include
9mkdir -p $myprefix/lib
10cp -avf bzlib.h $myprefix/include
11cp -avf libbz2.so* $myprefix/lib
12make install PREFIX=$myprefix
13ml load bzip2/1.0.8
Actually there are a set of patches for 1.0.6 which include pkg-config support so we will use those as well.
1cd $hpcroot
2myprefix=$HOME/.hpc/bzip2/1.0.6
3wget ftp://sourceware.org/pub/bzip2/bzip2-1.0.6.tar.gz
4tar xfz bzip2-1.0.6.tar.gz
5cd bzip2-1.0.6
6# patches.list: https://gist.github.com/steakknife/0ee85c93495ab9f9cff5e21ee12fb25b
7wget https://gist.githubusercontent.com/steakknife/946f6ee331512a269145b293cbe898cc/raw/bzip2-1.0.6-install_docs-1.patch
8wget https://gist.githubusercontent.com/steakknife/eceda09cae0cdb4900bcd9e479bab9be/raw/bzip2recover-CVE-2016-3189.patch
9wget https://gist.githubusercontent.com/steakknife/42feaa223adb4dd7c5c85f288794973c/raw/bzip2-man-page-location.patch
10wget https://gist.githubusercontent.com/steakknife/94f8aa4bfa79a3f896a660bf4e973f72/raw/bzip2-shared-make-install.patch
11wget https://gist.githubusercontent.com/steakknife/4faee8a657db9402cbeb579279156e84/raw/bzip2-pkg-config.patch
12patch -u < bzip2-1.0.6-install_docs-1.patch
13patch -u < bzip2recover-CVE-2016-3189.patch
14patch -u < bzip2-man-page-location.patch
15patch -u < bzip2-shared-make-install.patch
16patch -u < bzip2-pkg-config.patch
17make
18make install PREFIX=$myprefix
19ml load bzip2/1.0.6
sqlite
1cd $hpcroot
2myprefix=$HOME/.hpc/sqlite/3.32.3
3wget https://www.sqlite.org/2020/sqlite-autoconf-3320300.tar.gz
4gzip -dc sqlite-autoconf-3320300.tar.gz | tar xvf -
5cd sqlite-autoconf-3320300
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load sqlite/3.32.3
editline
1cd $hpcroot
2myprefix=$HOME/.hpc/editline/1.17.1
3wget https://github.com/troglobit/editline/releases/download/1.17.1/editline-1.17.1.tar.gz
4gzip -dc editline-1.17.1.tar.gz | tar xvf -
5cd editline-1.17.1
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load editline/1.17.1
Miniconda
We don’t need this very much, but it is still useful for some edge cases, mainly revolving around jupyter infrastructure.
1cd $HOME
2wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
3chmod +x Miniconda3-latest-Linux-x86_64.sh
4./Miniconda3-latest-Linux-x86_64.sh
5# Do not allow it to mess up the shell rc files
6eval "$($HOME/miniconda3/bin/conda shell.zsh hook)"
Note that we will prefer the manual evaluation since it can be handled in the lmod file.
Applications
Libraries and git aside, there are some tools we might want to have.
ag
The silver searcher, along with rg is very useful to have.
1cd $hpcroot
2myprefix=$HOME/.hpc/the_silver_searcher/2.2.0
3wget https://geoff.greer.fm/ag/releases/the_silver_searcher-2.2.0.tar.gz
4gzip -dc the_silver_searcher-2.2.0.tar.gz | tar xvf -
5cd the_silver_searcher-2.2.0
6./configure --prefix=$myprefix
7make -j $(nproc)
8make install
9ml load the_silver_searcher/2.2.0
Neovim
1cd $hpcroot
2myprefix=$HOME/.hpc/nvim/0.5.0
3wget https://github.com/neovim/neovim/releases/download/nightly/nvim.appimage
4chmod +x nvim.appimage
5./nvim.appimage --appimage-extract
6mkdir -p $myprefix
7mv squashfs-root/usr/* $myprefix
8ml load nvim/0.5.0
Tmux
1cd $hpcroot
2myprefix=$HOME/.hpc/tmux/3.1b
3wget https://github.com/tmux/tmux/releases/download/3.1b/tmux-3.1b-x86_64.AppImage
4chmod +x tmux-3.1b-x86_64.AppImage
5rm -rf squashfs-root
6./tmux-3.1b-x86_64.AppImage --appimage-extract
7mkdir -p $myprefix
8mv squashfs-root/usr/bin squashfs-root/usr/lib squashfs-root/usr/share $myprefix
9ml load tmux/3.1b
Zsh
More of an update than a requirement.
1cd $hpcroot
2myprefix=$HOME/.hpc/zsh/5.8
3wget https://github.com/zsh-users/zsh/archive/zsh-5.8.tar.gz
4gzip -dc zsh-5.8.tar.gz | tar xvf -
5cd zsh-zsh-5.8
6autoreconf -fiv
7./configure --prefix=$myprefix
8make -j $(nproc)
9make install
10ml load zsh/5.8
Conclusion
Having composed a bunch of these, I will of course try to somehow get nix up and running so it can bootstrap itself and allow me to work in peace. I might also eventually create shell scripts to automate updating these, but hopefully I can set up nix and not re-create package manager logic in lua.
Read other posts
Comments