User Tools

Site Tools


clusters:urania:spectre

Installing spectre

You can refer to the spectre installation instructions . Here is what Guillermo has tried.

Git clone on your home (sub)directory:

git clone git@github.com:sxs-collaboration/spectre.git

In the spectre home directory create a python environment (Note that the name 'env' is ignored by git, so its convenient to name the environment like this)

cd [spectre_home]/
python -m venv env

Activate the environment

source env/bin/activate

And pip install the required python packages summarized in the support/ subdirectory:

pip install -r support/Python/requirements.txt -r support/Python/dev_requirements.txt

Load the necessary modules (Can put this in a script to avoid doing it everytime):

module purge
module load gcc/11
module load openmpi/4
module load boost/1.79
module load gsl/1.16
module load cmake/3.22
module load hdf5-serial/1.12.2
module load anaconda/3/2021.11

Guillermo has already installed Charm and the remaining modules in an Spack environment (Please ask if you don't have access):

source /u/guilara/repos/spack/share/spack/setup-env.sh
spack env activate env3_spectre

Define the charm paths and spectre home

export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc
export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin
export SPECTRE_HOME=/u/[your_username]/path/to/spectre

Activate the python environment (if you haven't already)

source $SPECTRE_HOME/env/bin/activate

Configure spectre

In the spectre home directory make a build directory

cd $SPECTRE_HOME
mkdir build

Run cmake:

cd $SPECTRE_HOME/build/

cmake -D CMAKE_C_COMPILER=gcc -D CMAKE_CXX_COMPILER=g++ -D CMAKE_Fortran_COMPILER=gfortran -D CHARM_ROOT=$CHARM_ROOT -D CMAKE_BUILD_TYPE=Release -D DEBUG_SYMBOLS=OFF -D BUILD_SHARED_LIBS=ON -D MEMORY_ALLOCATOR=JEMALLOC -D BUILD_PYTHON_BINDINGS=ON -D Python_EXECUTABLE=$SPECTRE_HOME/env/bin/python -Wno-dev "$@" $SPECTRE_HOME

Build executable

Build and executable

cd $SPECTRE_HOME/build/
make -j16 [Executable_name]

For example, to make the unit tests:

make -j16 unit-tests

And to run them:

ctest -R Unit -j16

Submit a job

Submit a job:

sbatch submit_script.sh

Here is an example submit scirpt (submit_script.sh):

#!/bin/bash -l
# Standard output and error:
#SBATCH -o ./tjob.out.%j
#SBATCH -e ./tjob.err.%j
# Initial working directory:
#SBATCH -D ./
# Job Name:
#SBATCH -J testrun
# Number of MPI Tasks:
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
# Do not enable hyperthreading
#SBATCH --ntasks-per-core=1
# Memory usage [MB] of the job is required, e.g. 2200 MB per cpu:
#SBATCH --mem=158400
# for OpenMP:
#SBATCH --cpus-per-task=72
#
#SBATCH --mail-type=all
#SBATCH --mail-user=[you_username]@aei.mpg.de
#
# Wall clock limit:
#SBATCH --time=23:59:00

#
# Load compiler and MPI modules with explicit version specifications,
# consistently with the versions used to build the executable.
module purge
module load gcc/11
module load openmpi/4
module load boost/1.79
# module load doxygen/1.9.5
module load gsl/1.16
module load cmake/3.22
module load hdf5-serial/1.12.2
module load anaconda/3/2021.11

# Load Spack environment
source /u/guilara/repos/spack/share/spack/setup-env.sh
spack env activate env3_spectre

# Define Charm paths
export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc
export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin

# Spectre directories
export SPECTRE_HOME=/u/[your_username]/path/to/spectre
export SPECTRE_BUILD_DIR=${SPECTRE_HOME}/build/

# Specify the run directory. Be sure to make this directory first
export SPECTRE_RUN_DIR=/urania/ptmp/${USER}/spectre/Test/TestKerr

# Load python environment
source $SPECTRE_HOME/env/bin/activate

# Print directories
echo "Spectre home directory: ${SPECTRE_HOME}"
echo "Build directory: ${SPECTRE_BUILD_DIR}"
echo "Run directory: ${SPECTRE_RUN_DIR}"

# Choose the executable and input file to run
export SPECTRE_EXECUTABLE=${SPECTRE_BUILD_DIR}/bin/EvolveGhSingleBlackHole
export SPECTRE_INPUT_FILE=${SPECTRE_RUN_DIR}/KerrSchild.yaml

cd ${SPECTRE_RUN_DIR}

# Set desired permissions for files created with this script
umask 0022

# Set the path to include the build directory's bin directory
export PATH=${SPECTRE_BUILD_DIR}/bin:$PATH

# One thread for communication
CHARM_PPN=$(expr ${SLURM_CPUS_PER_TASK} - 1)
CHARM_PN=$((SLURM_NPROCS - SLURM_NNODES))
echo "Slurm tasks: ${SLURM_NTASKS}"
echo "Slurm cpus per task: ${SLURM_CPUS_PER_TASK}"
echo "Charm ppn: ${CHARM_PPN}"

# Print loaded modules
module list
spack find

# Run the program:
srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71  > tjob.out

# If you are starting from a checkpoint
# srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71 \
#	+restart SpectreCheckpoint000001 > tjob.out

For more examples refer to the Cobra and Raven user guides or to the SubmitScripts subdirectory in spectre/support/

clusters/urania/spectre.txt · Last modified: 2023/12/06 11:23 by Steffen Grunewald