Using SPACK for virtual environments

Caveat: There seems to be some confusion related to the term “module”. Indeed, there are two slightly separate contexts:

Environment modules (that modify $PATH and other environment variables, to make executables available, provide access to header files and libraries)
- used with the module (module av, module list, module load, …) and spack (spack find, spack load, …) shell commands
- installed in system-wide (even cluster-wide) available locations (/trinity/shared/modulefiles/, /home/SPACK*/)
Python modules (only visible to the matching version of python)
- used with import in a Python script
- installed system-wide (python*/*-packages/ below /usr/lib/) or individually by users (below ~/.local/lib/, venv/lib/)

Please don't mix them up!

There have been recent requests for (Python) modules like astroquery.utils.tap.core - which will not be fulfilled on a system level, but there's a way to set up a virtual environment (“venv”) to get them installed on an individual basis.

The following steals the basic ideas from lalsuite_spack (thanks, Serguei, for paving the way and documenting it).

The command sequence

uses SPACK to use the latest python and pip
creates a virtual environment (venv) under pip control
activates the venv
installs astroquery with all dependencies
finally leaves the venv:

Initialization, only to be done once:

Add the newest SPACK repository to $PATH etc.:

[stefgru@login02 ~]$ . /home/SPACK2021/share/spack/setup-env.sh

(Use the latest SPACK repository! The text below was written at SPACK2019 times though.)

Load the pip (SPACK) module with all its dependencies (the -r option is no longer required):

[stefgru@login02 ~]$ spack load  py-pip

Install the virtualenv (Python) module:

[stefgru@login02 ~]$ pip install virtualenv --user
[...]
Successfully installed virtualenv-16.7.7

Create a new venv:

[stefgru@login02 ~]$ virtualenv astroquery-test
[...]
done.

On Hypatia (and Minerva now too), spack load py-virtualenv, then just run the virtualenv command (no pip magic needed).

What may also work, using SPACK's python(3) and the system's virtualenv: spack load python; virtualenv –python=`which python` … (almost completely untested and not recommended any longer!)

To initially populate the new venv:

Activate the venv:

[stefgru@login02 ~]$ . astroquery-test/bin/activate

Note that the prompt changes, to indicate you're inside the venv now!

(astroquery-test) [stefgru@login02 ~]$ module purge

This is important. If you forget this, you'll get unexpected behaviour. Check with module list that no system or SPACK modules are loaded!

(astroquery-test) [stefgru@login02 ~]$ module list
No Modulefiles Currently Loaded.

As a safety measure, bump pip and setuptools to the latest version:

(astroquery-test) [stefgru@login02 ~]$ pip install --upgrade pip
Requirement already up-to-date: pip in ./astroquery-test/lib/python3.7/site-packages (19.3.1)
(astroquery-test) [stefgru@login02 ~]$ pip install --upgrade setuptools
Requirement already up-to-date: setuptools in ./astroquery-test/lib/python3.7/site-packages (41.6.0)

Now install the real payload:

(astroquery-test) [stefgru@login02 ~]$ pip install astroquery
Collecting astroquery
[...]
Installing collected packages: numpy, astropy, chardet, certifi, idna, urllib3, requests, jeepney, six, pycparser, cffi, cryptography, secretstorage, entrypoints, keyring, soupsieve, beautifulsoup4, webencodings, html5lib, astroquery
Successfully installed astropy-3.2.3 astroquery-0.3.10 beautifulsoup4-4.8.1 certifi-2019.9.11 cffi-1.13.2 chardet-3.0.4 cryptography-2.8 entrypoints-0.3 html5lib-1.0.1 idna-2.8 jeepney-0.4.1 keyring-19.2.0 numpy-1.17.3 pycparser-2.19 requests-2.22.0 secretstorage-3.1.1 six-1.13.0 soupsieve-1.9.5 urllib3-1.25.6 webencodings-0.5.1

It's time for a test now:

(astroquery-test) [stefgru@login02 ~]$ python
Python 3.7.4 (default, Nov  6 2019, 17:06:47)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import astroquery.utils.tap.core
>>> exit()

The (Python) module is there, obviously. We're done for the setup part:

(astroquery-test) [stefgru@login02 ~]$ deactivate
[stefgru@login02 ~]$

To use the venv:

[stefgru@login02 ~]$ . astroquery-test/bin/activate
(astroquery-test) [stefgru@login02 ~]$ module purge
(astroquery-test) [stefgru@login02 ~]$ module list
No Modulefiles Currently Loaded.
(astroquery-test) [stefgru@login02 ~]$ python -c "import astroquery.utils.tap.core" # you want to do real work here...
(astroquery-test) [stefgru@login02 ~]$ [...]
[...]
(astroquery-test) [stefgru@login02 ~]$ deactivate

There's no need to go through the initialization sequence anymore as the venv “remembers” the python and pip that were used at its creation time!

I think that this is the minimum to get astroquery installed - all the steps are required!