====== Installing spectre ======

You can refer to {{ https://spectre-code.org/installation.html | the spectre installation instructions }}. Here is what Guillermo has tried.

Git clone on your home (sub)directory:

<code>
git clone git@github.com:sxs-collaboration/spectre.git
</code>

In the spectre home directory create a python environment (Note that the name 'env' is ignored by git, so its convenient to name the environment like this)

<code>
cd [spectre_home]/
python -m venv env
</code>

Activate the environment

<code>
source env/bin/activate
</code>

And pip install the required python packages summarized in the support/ subdirectory:

<code>
pip install -r support/Python/requirements.txt -r support/Python/dev_requirements.txt
</code>

Load the necessary modules (Can put this in a script to avoid doing it everytime):

<code>
module purge
module load gcc/11
module load openmpi/4
module load boost/1.79
module load gsl/1.16
module load cmake/3.22
module load hdf5-serial/1.12.2
module load anaconda/3/2021.11
</code>

Guillermo has already installed {{ https://charm.readthedocs.io/en/latest/quickstart.html | Charm }} and the remaining modules in an {{https://spack-tutorial.readthedocs.io/en/latest/tutorial_basics.html | Spack }} environment (Please ask if you don't have access):

<code>
source /u/guilara/repos/spack/share/spack/setup-env.sh
spack env activate env3_spectre
</code>

Define the charm paths and spectre home

<code>
export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc
export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin
export SPECTRE_HOME=/u/[your_username]/path/to/spectre
</code>

Activate the python environment (if you haven't already)

<code>
source $SPECTRE_HOME/env/bin/activate
</code>

===== Configure spectre =====

In the spectre home directory make a build directory

<code>
cd $SPECTRE_HOME
mkdir build
</code>

Run cmake:

<code>
cd $SPECTRE_HOME/build/

cmake -D CMAKE_C_COMPILER=gcc -D CMAKE_CXX_COMPILER=g++ -D CMAKE_Fortran_COMPILER=gfortran -D CHARM_ROOT=$CHARM_ROOT -D CMAKE_BUILD_TYPE=Release -D DEBUG_SYMBOLS=OFF -D BUILD_SHARED_LIBS=ON -D MEMORY_ALLOCATOR=JEMALLOC -D BUILD_PYTHON_BINDINGS=ON -D Python_EXECUTABLE=$SPECTRE_HOME/env/bin/python -Wno-dev "$@" $SPECTRE_HOME
</code>

===== Build executable =====


Build and executable

<code>
cd $SPECTRE_HOME/build/
make -j16 [Executable_name]
</code>

For example, to make the unit tests:

<code>
make -j16 unit-tests
</code>

And to run them:

<code>
ctest -R Unit -j16
</code>

====== Submit a job ======

Submit a job:

<code>
sbatch submit_script.sh
</code>

Here is an example submit scirpt (submit_script.sh):

<code>
#!/bin/bash -l
# Standard output and error:
#SBATCH -o ./tjob.out.%j
#SBATCH -e ./tjob.err.%j
# Initial working directory:
#SBATCH -D ./
# Job Name:
#SBATCH -J testrun
# Number of MPI Tasks:
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
# Do not enable hyperthreading
#SBATCH --ntasks-per-core=1
# Memory usage [MB] of the job is required, e.g. 2200 MB per cpu:
#SBATCH --mem=158400
# for OpenMP:
#SBATCH --cpus-per-task=72
#
#SBATCH --mail-type=all
#SBATCH --mail-user=[you_username]@aei.mpg.de
#
# Wall clock limit:
#SBATCH --time=23:59:00

#
# Load compiler and MPI modules with explicit version specifications,
# consistently with the versions used to build the executable.
module purge
module load gcc/11
module load openmpi/4
module load boost/1.79
# module load doxygen/1.9.5
module load gsl/1.16
module load cmake/3.22
module load hdf5-serial/1.12.2
module load anaconda/3/2021.11

# Load Spack environment
source /u/guilara/repos/spack/share/spack/setup-env.sh
spack env activate env3_spectre

# Define Charm paths
export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc
export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin

# Spectre directories
export SPECTRE_HOME=/u/[your_username]/path/to/spectre
export SPECTRE_BUILD_DIR=${SPECTRE_HOME}/build/

# Specify the run directory. Be sure to make this directory first
export SPECTRE_RUN_DIR=/urania/ptmp/${USER}/spectre/Test/TestKerr

# Load python environment
source $SPECTRE_HOME/env/bin/activate

# Print directories
echo "Spectre home directory: ${SPECTRE_HOME}"
echo "Build directory: ${SPECTRE_BUILD_DIR}"
echo "Run directory: ${SPECTRE_RUN_DIR}"

# Choose the executable and input file to run
export SPECTRE_EXECUTABLE=${SPECTRE_BUILD_DIR}/bin/EvolveGhSingleBlackHole
export SPECTRE_INPUT_FILE=${SPECTRE_RUN_DIR}/KerrSchild.yaml

cd ${SPECTRE_RUN_DIR}

# Set desired permissions for files created with this script
umask 0022

# Set the path to include the build directory's bin directory
export PATH=${SPECTRE_BUILD_DIR}/bin:$PATH

# One thread for communication
CHARM_PPN=$(expr ${SLURM_CPUS_PER_TASK} - 1)
CHARM_PN=$((SLURM_NPROCS - SLURM_NNODES))
echo "Slurm tasks: ${SLURM_NTASKS}"
echo "Slurm cpus per task: ${SLURM_CPUS_PER_TASK}"
echo "Charm ppn: ${CHARM_PPN}"

# Print loaded modules
module list
spack find

# Run the program:
srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71  > tjob.out

# If you are starting from a checkpoint
# srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71 \
#	+restart SpectreCheckpoint000001 > tjob.out
</code>


For more examples refer to the {{https://docs.mpcdf.mpg.de/doc/computing/cobra-user-guide.html | Cobra}} and {{ https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html | Raven }} user guides or to the SubmitScripts subdirectory in spectre/support/