Steffen says: better use spack load for variant support! - build hashes may change in the future. The following table lists both spack load arguments and the equivalent module load name.

Caveat: spack load (still) loads multiple modules of the same name. This is in contradiction to the manual. Check for such occurrences, and remove the “older” one. (This happened after updating the git copy.) For now, specify version or hash. This needs some investigation, and subsequent cleanup.

(pip appears to be missing, investigate) (Steffen again: there's py-pip)

(SO:) after loading the swig module, swig still points to /usr/bin/swigsry, fixed - spack load swig didn't complain that the module wasn't there?!? List updated.

(S:) Check whether there are duplicate modules (different compiler, different variants): module list. If there are collisions, spack unload … and add specifiers (%gcc@7.3.0, variant) where necessary (and please update the instructions!)

This guide relies on all external, non-python dependencies to be supplied as modules via spack. I always recommend installing all python things via pip in a virtual environment. Note that this still uses Python 2.7 unfortunately, since Python 3 is not supported fully yet.

To get started:

. /home/SPACK/share/spack/setup-env.sh

Load things to install the python virtualenv:

module purge
spack load python @2.7.15                 # module load python-2.7.15-gcc-7.3.0-2azrjn6
spack load py-setuptools                  # module load py-setuptools-35.0.2-gcc-7.3.0-ef474iq
spack load py-packaging                   # module load py-packaging-16.8-gcc-7.3.0-mgow2tn
spack load py-pyparsing                   # module load py-pyparsing-2.2.0-gcc-7.3.0-e3vim7d
spack load py-appdirs                     # module load py-appdirs-1.4.3-gcc-7.3.0-6qkuajr
spack load py-six                         # module load py-six-1.10.0-gcc-7.3.0-4jd5suz
spack load py-pip                         # module load py-pip-9.0.1-6uwjdi6

Install virtualenv:

pip install virtualenv --user

Create your virtualenv:

virtualenv test_install

Now source it:

source test_install/bin/activate

Purge the environment (I found this necessary or virtualenv doesn't work with pip correctly):

module purge

Load dependencies

spack load gcc                            # module load gcc-7.3.0-gcc-7.3.0-pqzvlyr
spack load libpng
spack load openmpi                        # module load openmpi-3.0.0-gcc-7.3.0-7qkyepv
spack load hdf5 @1.8.19 +hl +mpi          # module load hdf5-1.8.19-gcc-7.3.0-3xwjknd or 
#spack load hdf5 @1.10.1 +hl +mpi         # module load hdf5-1.10.1-gcc-7.3.0-gwojm7q
spack load gsl @2.4                       # module load gsl-2.4-gcc-7.3.0-yobvdek
spack load fftw @3.3.7                    # module load fftw-3.3.7-gcc-7.3.0-6u4ghoh
spack load cfitsio                        # module load cfitsio-3.420-gcc-7.3.0-ti6tfnv
spack load chealpix                       # module load chealpix-3.30.0-gcc-7.3.0-ylinsra
spack load metaio                         # module load metaio-8.5.1-gcc-7.3.0-mlvv3s2
spack load libframe                       # module load libframe-8.30-gcc-7.3.0-iy6ze7r
spack load libxml2                        # module load libxml2-2.9.4-gcc-7.3.0-etonamb
spack load swig                           # module load swig-3.0.12-gcc-7.3.0-xsb6vfv

Install all the python goodies:

pip install --upgrade pip
pip install numpy scipy matplotlib h5py cython ipython healpy shapely

Clone LALSuite (somewhere you like, I recomment inside your virtualenv, i.e. test_install/src) :

git clone git@git.ligo.org:lscsoft/lalsuite.git

Boot it up:

cd lalsuite
./00boot

Configure (you may or may not want to point the prefix to the virtualenv, up to you). This will take ~10 mins, have a coffee:

./configure --prefix=/home/sossokine/test_install --enable-swig-python --enable-mpi --enable-lal-all

Make it. It goes a lot faster if you do this on 16 cores, but please don't do this on the head node:

srun -N 1 --pty bash
make -j 16

Make install:

make -j 16 install

Exit the interactive node, since it doesn't have access to the internet:

exit

Get pylal:

cd src
git clone git@git.ligo.org:lscsoft/pylal.git

Install it:

source ~/test_install/etc/lalsuite-user-env.sh 
cd pylal
python setup.py install

Install pycbc

pip install pycbc

In addition to LALSuite itself, several supporting packages are needed:

• LDG allows us to clone the development version of LALSuite and provides access to LIGO clusters via gsissh, gsiscp
• LALSuite needs installations of libframe, glue, metaio, swig, fftw3, hdf5

This is the list of modules that (I think) I used during installation (updated 07/03/17 by Sergei):

gcc/4.9.3
gsl/gcc-4.9.3/2.1
openmpi/gcc-4.9.3/1.10.2
opa-psm2/minerva
swig/gcc-4.9.3/3.0.12
hdf5/gcc-4.9.3/1.8.18

Started from these instructions: https://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/installclient-source.html

I downloaded a binary Linux tarball for globus from http://toolkit.globus.org/ftppub/gt6/installers/linux/globus_toolkit-6.0.1478289945-x86_64-unknown-linux-gnu-Build-303.tar.gz Untared and installed into ~/sw/globus

I downloaded http://software.grid.iu.edu/pacman/cadist/1.58IGTFNEW/osg-certificates-1.58IGTFNEW.tar.gz

tar xvf osg-certificates-1.58IGTFNEW.tar.gz -C ~/sw/globus/share/

I copied two scripts (ecp-cookie-init, ligo-proxy-init) from my Mac installation of LDG to ~/sw/globus/bin/

I added this to my .bash_profile:

export PATH=$PATH:/home/mpuer/sw/globus/bin/
source ~/sw/globus/share/globus-user-env.sh

With LDG installed we're ready to perform a one-time setup:

ecp-cookie-init LIGO.ORG https://versions.ligo.org/git albert.einstein
git config --global http.cookiefile /tmp/ecpcookie.u`id -u`

and can clone lalsuite:

git clone https://versions.ligo.org/git/lalsuite.git

Some packages we need are hosted on a LIGO server:

LSCSOFT_SRCURL=http://software.ligo.org/lscsoft/source
wget $LSCSOFT_SRCURL/glue-1.53.0.tar.gz
wget $LSCSOFT_SRCURL/libframe-8.30.tar.gz
wget $LSCSOFT_SRCURL/metaio-8.4.0.tar.gz

I build everything with gcc. (I ran into some problems linking against icc compiled packages).

For most packages I simply do:

./configure --prefix=/home/mpuer/sw/ && make -j install

and I only provide the address from which I downloaded the package.

For glue:

python setup.py install --prefix=/home/mpuer/sw

To install fftw3 a bit more tweaking is needed. The shared is necessary for pycbc

wget http://www.fftw.org/fftw-3.3.5.tar.gz
tar xzvf fftw-3.3.5.tar.gz && cd fftw-3.3.5
# double precision
./configure --prefix=/home/mpuer/sw --with-pic
# Edit the Makefile and add -fPIC to CFLAGS
make -j install
# single precision
make distclean
./configure --prefix=/home/mpuer/sw --enable-single --with-pic
# Edit the Makefile and add -fPIC to CFLAGS
make -j install
make distclean
./configure --prefix=/home/mpuer/sw --with-pic --enable-shared
# Edit the Makefile and add -fPIC to CFLAGS
make -j install
# single precision
make distclean
./configure --prefix=/home/mpuer/sw --enable-single --with-pic --enable-shared
# Edit the Makefile and add -fPIC to CFLAGS
make -j install
	

The swig binary will be in ~/sw/bin: wget http://downloads.sourceforge.net/swig/swig-3.0.10.tar.gz export PATH=$PATH:/home/mpuer/sw/bin/ I built HDF5 with gcc. I see there is hdf5/gcc-4.9.3/1.8.16 now, so the latter should work. wget https://support.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.17.tar.gz

Point to pkgconfig files:

export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:~/sw/lib/pkgconfig

Here I disable some packages that I don't always need and build from the top level:

export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:~/sw/lib/pkgconfig

./00boot && ./configure --prefix=/home/mpuer/master/ --disable-laldetchar --disable-lalpulsar --disable-lalburst --disable-lalinspiral --enable-swig-python && make -j install

Sergei: In my experience I had to get rid of all the –disable and use –enable-mpi to get lalinference to build correctly including the MCMC sampler.

For this I rely on Vivien's scripts:

build.sh, setup-lscsoftrc.sh

A copy of these can be found in /home/mpuer/src.

pip install --user virtualenv
virtualenv venv_lalinference_post --no-site-packages
source venv_lalinference_post/bin/activate
easy_install pip
pip install ipython
pip install scipy
pip install numpy
pip install matplotlib
pip install healpy
pip install h5py
pip install corner

We also need basemap and geoslib for the skymaps:

git clone https://github.com/libgeos/libgeos.git
./autogen.sh
./configure --prefix=/home/mpuer/sw --with-pic --enable-python
make -j install

source venv_lalinference_post/bin/activate
export GEOS_DIR=/home/mpuer/sw
pip install https://github.com/matplotlib/basemap/archive/master.zip

We neeed chealpix and cfitsoi for post-processing:

get cfitsio from https://heasarc.gsfc.nasa.gov/fitsio/
./configure --prefix=/home/mpuer/sw
make -j install

get chealpic from https://sourceforge.net/projects/healpix/?source=typ_redirect
./configure --prefix=/home/mpuer/sw --with-pic
make -j install

Rebuild lalinference into venv:

./00boot
./configure --prefix=/home/mpuer/venv_lalinference_post/ --disable-laldetchar --enable-swig-python --enable-lalinference
make -j install

• make sure that lalinference runs properly:
  • post-processing:
    • plot of NR injection: Segfault fixed in https://bugs.ligo.org/redmine/issues/4683
    • : name of NR HDF5 file is not passed to cbcBayesPostProc.py, so shows up as NULL.