====== Installing LALSuite on Minerva (the SPACK way) ======

<note warning> Under construction, these instructions don't work fully yet.</note>


===== Misc notes, to become obsolete =====

//Steffen says: better use ''spack load'' for variant support! - build hashes may change in the future.//
//The following table lists both ''spack load'' arguments and the equivalent ''module load'' name.//

**Caveat: ''spack load'' (still) loads multiple modules of the same name. This is in contradiction to the manual.
Check for such occurrences, and remove the "older" one. (This happened after updating the git copy.)**
//For now, specify version or hash. This needs some investigation, and subsequent cleanup.//


  
<del>(pip appears to be missing, investigate)</del> //(Steffen again: there's py-pip)//

<del>(SO:) after loading the swig module, swig still points to /usr/bin/swig</del>sry, fixed - ''spack load swig'' didn't complain that the module wasn't there?!? List updated.

(S:) Check whether there are duplicate modules (different compiler, different variants): ''module list''.
If there are collisions, ''spack unload ...'' and add specifiers (''%gcc@7.3.0'', variant) where necessary
(and please update the instructions!)

===== Instructions =====


This guide relies on all external, non-python dependencies to be supplied as modules via spack.
I always recommend installing all python things via pip in a virtual environment. Note that
this still uses Python 2.7 unfortunately, since Python 3 is not supported fully yet.

To get started:

  . /home/SPACK/share/spack/setup-env.sh

Load things to install the python virtualenv:
  module purge
  spack load python @2.7.15                 # module load python-2.7.15-gcc-7.3.0-2azrjn6
  spack load py-setuptools                  # module load py-setuptools-35.0.2-gcc-7.3.0-ef474iq
  spack load py-packaging                   # module load py-packaging-16.8-gcc-7.3.0-mgow2tn
  spack load py-pyparsing                   # module load py-pyparsing-2.2.0-gcc-7.3.0-e3vim7d
  spack load py-appdirs                     # module load py-appdirs-1.4.3-gcc-7.3.0-6qkuajr
  spack load py-six                         # module load py-six-1.10.0-gcc-7.3.0-4jd5suz
  spack load py-pip                         # module load py-pip-9.0.1-6uwjdi6

Install virtualenv:
  pip install virtualenv --user


  	
Create your virtualenv:
  virtualenv test_install

Now source it:
  source test_install/bin/activate

Purge the environment (I found this necessary or virtualenv doesn't work with pip correctly): 
  module purge

Load dependencies
  spack load gcc                            # module load gcc-7.3.0-gcc-7.3.0-pqzvlyr
  spack load libpng
  spack load openmpi                        # module load openmpi-3.0.0-gcc-7.3.0-7qkyepv
  spack load hdf5 @1.8.19 +hl +mpi          # module load hdf5-1.8.19-gcc-7.3.0-3xwjknd or 
  #spack load hdf5 @1.10.1 +hl +mpi         # module load hdf5-1.10.1-gcc-7.3.0-gwojm7q
  spack load gsl @2.4                       # module load gsl-2.4-gcc-7.3.0-yobvdek
  spack load fftw @3.3.7                    # module load fftw-3.3.7-gcc-7.3.0-6u4ghoh
  spack load cfitsio                        # module load cfitsio-3.420-gcc-7.3.0-ti6tfnv
  spack load chealpix                       # module load chealpix-3.30.0-gcc-7.3.0-ylinsra
  spack load metaio                         # module load metaio-8.5.1-gcc-7.3.0-mlvv3s2
  spack load libframe                       # module load libframe-8.30-gcc-7.3.0-iy6ze7r
  spack load libxml2                        # module load libxml2-2.9.4-gcc-7.3.0-etonamb
  spack load swig                           # module load swig-3.0.12-gcc-7.3.0-xsb6vfv
  
Install all the python goodies:
  pip install --upgrade pip
  pip install numpy scipy matplotlib h5py cython ipython healpy shapely
  

  
Clone LALSuite (somewhere you like, I recomment inside your virtualenv, i.e. test_install/src) :
  git clone git@git.ligo.org:lscsoft/lalsuite.git
  
Boot it up:
  cd lalsuite
  ./00boot
  
Configure (you may or may not want to point the prefix to the virtualenv, up to you). This will take ~10 mins, have a coffee:
  ./configure --prefix=/home/sossokine/test_install --enable-swig-python --enable-mpi --enable-lal-all
Make it. It goes a lot faster if you do this on 16 cores, but please don't do this on the head node:
  srun -N 1 --pty bash
  make -j 16
Make install:
  make -j 16 install

Exit the interactive node, since it doesn't have access to the internet:
  exit

Get pylal:
  cd src
  git clone git@git.ligo.org:lscsoft/pylal.git
Install it:
  source ~/test_install/etc/lalsuite-user-env.sh 
  cd pylal
  python setup.py install

Install pycbc
  pip install pycbc


  
  	
====== Michael's notes on installing LALSuite on the Minerva cluster (the Modules way) ======

In addition to LALSuite itself, several supporting packages are needed:
  * LDG allows us to clone the development version of LALSuite and provides access to LIGO clusters via gsissh, gsiscp
  * LALSuite needs installations of libframe, glue, metaio, swig, fftw3, hdf5

===== Modules =====

This is the list of modules that (I think) I used during installation (updated 07/03/17 by Sergei):

  gcc/4.9.3
  gsl/gcc-4.9.3/2.1
  openmpi/gcc-4.9.3/1.10.2
  opa-psm2/minerva
  swig/gcc-4.9.3/3.0.12
  hdf5/gcc-4.9.3/1.8.18
  
===== LDG =====

Started from these instructions: https://www.lsc-group.phys.uwm.edu/lscdatagrid/doc/installclient-source.html

I downloaded a binary Linux tarball for globus from http://toolkit.globus.org/ftppub/gt6/installers/linux/globus_toolkit-6.0.1478289945-x86_64-unknown-linux-gnu-Build-303.tar.gz
Untared and installed into ~/sw/globus

I downloaded http://software.grid.iu.edu/pacman/cadist/1.58IGTFNEW/osg-certificates-1.58IGTFNEW.tar.gz

  tar xvf osg-certificates-1.58IGTFNEW.tar.gz -C ~/sw/globus/share/

I copied two scripts (ecp-cookie-init, ligo-proxy-init) from my Mac installation of LDG to ~/sw/globus/bin/

I added this to my .bash_profile:

  export PATH=$PATH:/home/mpuer/sw/globus/bin/
  source ~/sw/globus/share/globus-user-env.sh

==== Cloning LALSuite  ====

With LDG installed we're ready to perform a one-time setup:

  ecp-cookie-init LIGO.ORG https://versions.ligo.org/git albert.einstein
  git config --global http.cookiefile /tmp/ecpcookie.u`id -u`

and can clone lalsuite:

  git clone https://versions.ligo.org/git/lalsuite.git

===== Supporting packages =====

Some packages we need are hosted on a LIGO server:

  LSCSOFT_SRCURL=http://software.ligo.org/lscsoft/source
  wget $LSCSOFT_SRCURL/glue-1.53.0.tar.gz
  wget $LSCSOFT_SRCURL/libframe-8.30.tar.gz
  wget $LSCSOFT_SRCURL/metaio-8.4.0.tar.gz

I build everything with gcc. (I ran into some problems linking against icc compiled packages).

For most packages I simply do:

  ./configure --prefix=/home/mpuer/sw/ && make -j install

and I only provide the address from which I downloaded the package.

For glue: 

  python setup.py install --prefix=/home/mpuer/sw

To install fftw3 a bit more tweaking is needed. The shared is necessary for pycbc

  wget http://www.fftw.org/fftw-3.3.5.tar.gz
  tar xzvf fftw-3.3.5.tar.gz && cd fftw-3.3.5
  # double precision
  ./configure --prefix=/home/mpuer/sw --with-pic
  # Edit the Makefile and add -fPIC to CFLAGS
  make -j install
  # single precision
  make distclean
  ./configure --prefix=/home/mpuer/sw --enable-single --with-pic
  # Edit the Makefile and add -fPIC to CFLAGS
  make -j install
  make distclean
  ./configure --prefix=/home/mpuer/sw --with-pic --enable-shared
  # Edit the Makefile and add -fPIC to CFLAGS
  make -j install
  # single precision
  make distclean
  ./configure --prefix=/home/mpuer/sw --enable-single --with-pic --enable-shared
  # Edit the Makefile and add -fPIC to CFLAGS
  make -j install
  	
<del>The swig binary will be in ~/sw/bin:
  wget http://downloads.sourceforge.net/swig/swig-3.0.10.tar.gz
  export PATH=$PATH:/home/mpuer/sw/bin/

I built HDF5 with gcc. I see there is hdf5/gcc-4.9.3/1.8.16 now, so the latter should work.
  wget https://support.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.17.tar.gz</del>

Point to pkgconfig files:
  export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:~/sw/lib/pkgconfig


===== Simple LALSuite install =====

Here I disable some packages that I don't always need and build from the top level:

  export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:~/sw/lib/pkgconfig

  ./00boot && ./configure --prefix=/home/mpuer/master/ --disable-laldetchar --disable-lalpulsar --disable-lalburst --disable-lalinspiral --enable-swig-python && make -j install
  
Sergei: In my experience I had to get rid of all the --disable and use --enable-mpi to get lalinference to build correctly including the MCMC sampler.

===== Full LALsuite install =====

For this I rely on Vivien's scripts: 

  build.sh, setup-lscsoftrc.sh

A copy of these can be found in /home/mpuer/src.


===== Python venv for lalinference post-processing =====

  pip install --user virtualenv
  virtualenv venv_lalinference_post --no-site-packages
  source venv_lalinference_post/bin/activate
  easy_install pip
  pip install ipython
  pip install scipy
  pip install numpy
  pip install matplotlib
  pip install healpy
  pip install h5py
  pip install corner


We also need basemap and geoslib for the skymaps:

  git clone https://github.com/libgeos/libgeos.git
  ./autogen.sh
  ./configure --prefix=/home/mpuer/sw --with-pic --enable-python
  make -j install
  
  source venv_lalinference_post/bin/activate
  export GEOS_DIR=/home/mpuer/sw
  pip install https://github.com/matplotlib/basemap/archive/master.zip

We neeed chealpix and cfitsoi for post-processing:

  get cfitsio from https://heasarc.gsfc.nasa.gov/fitsio/
  ./configure --prefix=/home/mpuer/sw
  make -j install

  get chealpic from https://sourceforge.net/projects/healpix/?source=typ_redirect
  ./configure --prefix=/home/mpuer/sw --with-pic
  make -j install

Rebuild lalinference into venv:

  ./00boot
  ./configure --prefix=/home/mpuer/venv_lalinference_post/ --disable-laldetchar --enable-swig-python --enable-lalinference
  make -j install
  

===== Tests =====

  * make sure that lalinference runs properly: 
    * post-processing:
      * plot of NR injection: Segfault fixed in https://bugs.ligo.org/redmine/issues/4683
      * FIXME: name of NR HDF5 file is not passed to cbcBayesPostProc.py, so shows up as NULL.