Astrophysical and Cosmological Relativity Group

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Using SPACK for virtual environments ======

<note warning>
//**Caveat:**// There seems to be some confusion related to the term **"module"**.
Indeed, there are two slightly separate contexts:
  * Environment modules (that modify ''$PATH'' and other environment variables, to make executables available, provide access to header files and libraries)
    * used with the ''module'' (''module av'', ''module list'', ''module load'', ...) and ''spack'' (''spack find'', ''spack load'', ...) shell commands
    * installed in system-wide (even cluster-wide) available locations (''/trinity/shared/modulefiles/'', ''/home/SPACK*/'')
  * Python modules (only visible to the matching version of ''python'')
    * used with ''import'' in a Python script
    * installed system-wide (''python*/*-packages/'' below ''/usr/lib/'') or individually by users (below ''~/.local/lib/'', //venv//''/lib/'')
Please don't mix them up!
</note>

There have been recent requests for (Python) modules like ''astroquery.utils.tap.core'' - which will not be fulfilled on a system level,
but there's a way to set up a virtual environment ("//venv//") to get them installed on an individual basis.

//The following steals the basic ideas from [[clusters:minerva:lalsuite_spack]] (thanks, Serguei, for paving the way and documenting it).//

The command sequence
  * uses SPACK to use the latest ''python'' and ''pip''
  * creates a virtual environment (//venv//) under ''pip'' control
  * activates the //venv//
  * installs ''astroquery'' with all dependencies
  * finally leaves the //venv//:

Initialization, only to be done once:

Add the newest SPACK repository to ''$PATH'' etc.:
  [stefgru@login02 ~]$ . /home/SPACK2021/share/spack/setup-env.sh

(**Use the latest SPACK repository!** The text below was written at SPACK2019 times though.)

Load the ''pip'' (SPACK) module //with all its dependencies// (the **-r** option is no longer required):
  [stefgru@login02 ~]$ spack load  py-pip
Install the ''virtualenv'' (Python) module:
  [stefgru@login02 ~]$ pip install virtualenv --user
  [...]
  Successfully installed virtualenv-16.7.7
Create a new //venv//:
  [stefgru@login02 ~]$ virtualenv astroquery-test
  [...]
  done.

<note warning>
On Hypatia (and Minerva now too), ''spack load  py-virtualenv'', then just run the ''virtualenv'' command (no ''pip'' magic needed).

What may also work, using SPACK's ''python''(3) and the system's ''virtualenv'': ''spack load  python; virtualenv --python=`which python` ...'' (almost completely untested and not recommended any longer!) 
</note>

To initially populate the new //venv//:

Activate the //venv//:
  [stefgru@login02 ~]$ . astroquery-test/bin/activate
Note that the prompt changes, to indicate you're inside the //venv// now!
  (astroquery-test) [stefgru@login02 ~]$ module purge
This is **important**. If you forget this, you'll get unexpected behaviour. **Check with ''module list'' that no system or SPACK modules are loaded!**
  (astroquery-test) [stefgru@login02 ~]$ module list
  No Modulefiles Currently Loaded.
As a safety measure, bump ''pip'' and ''setuptools'' to the latest version:
  (astroquery-test) [stefgru@login02 ~]$ pip install --upgrade pip
  Requirement already up-to-date: pip in ./astroquery-test/lib/python3.7/site-packages (19.3.1)
  (astroquery-test) [stefgru@login02 ~]$ pip install --upgrade setuptools
  Requirement already up-to-date: setuptools in ./astroquery-test/lib/python3.7/site-packages (41.6.0)
Now install the real payload:
  (astroquery-test) [stefgru@login02 ~]$ pip install astroquery
  Collecting astroquery
  [...]
  Installing collected packages: numpy, astropy, chardet, certifi, idna, urllib3, requests, jeepney, six, pycparser, cffi, cryptography, secretstorage, entrypoints, keyring, soupsieve, beautifulsoup4, webencodings, html5lib, astroquery
  Successfully installed astropy-3.2.3 astroquery-0.3.10 beautifulsoup4-4.8.1 certifi-2019.9.11 cffi-1.13.2 chardet-3.0.4 cryptography-2.8 entrypoints-0.3 html5lib-1.0.1 idna-2.8 jeepney-0.4.1 keyring-19.2.0 numpy-1.17.3 pycparser-2.19 requests-2.22.0 secretstorage-3.1.1 six-1.13.0 soupsieve-1.9.5 urllib3-1.25.6 webencodings-0.5.1
It's time for a test now:
  (astroquery-test) [stefgru@login02 ~]$ python
  Python 3.7.4 (default, Nov  6 2019, 17:06:47)
  [GCC 9.2.0] on linux
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import astroquery.utils.tap.core
  >>> exit()
The (Python) module is there, obviously. We're done for the setup part:
  (astroquery-test) [stefgru@login02 ~]$ deactivate
  [stefgru@login02 ~]$

To use the //venv//:
  [stefgru@login02 ~]$ . astroquery-test/bin/activate
  (astroquery-test) [stefgru@login02 ~]$ module purge
  (astroquery-test) [stefgru@login02 ~]$ module list
  No Modulefiles Currently Loaded.
  (astroquery-test) [stefgru@login02 ~]$ python -c "import astroquery.utils.tap.core" # you want to do real work here...
  (astroquery-test) [stefgru@login02 ~]$ [...]
  [...]
  (astroquery-test) [stefgru@login02 ~]$ deactivate

There's no need to go through the initialization sequence anymore as the //venv// "remembers" the ''python'' and ''pip'' that were used at its creation time!

I think that this is the minimum to get ''astroquery'' installed - all the steps are required!