Table of Contents
Installing spectre
You can refer to the spectre installation instructions . Here is what Guillermo has tried.
Git clone on your home (sub)directory:
git clone git@github.com:sxs-collaboration/spectre.git
In the spectre home directory create a python environment (Note that the name 'env' is ignored by git, so its convenient to name the environment like this)
cd [spectre_home]/ python -m venv env
Activate the environment
source env/bin/activate
And pip install the required python packages summarized in the support/ subdirectory:
pip install -r support/Python/requirements.txt -r support/Python/dev_requirements.txt
Load the necessary modules (Can put this in a script to avoid doing it everytime):
module purge module load gcc/11 module load openmpi/4 module load boost/1.79 module load gsl/1.16 module load cmake/3.22 module load hdf5-serial/1.12.2 module load anaconda/3/2021.11
Guillermo has already installed Charm and the remaining modules in an Spack environment (Please ask if you don't have access):
source /u/guilara/repos/spack/share/spack/setup-env.sh spack env activate env3_spectre
Define the charm paths and spectre home
export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin export SPECTRE_HOME=/u/[your_username]/path/to/spectre
Activate the python environment (if you haven't already)
source $SPECTRE_HOME/env/bin/activate
Configure spectre
In the spectre home directory make a build directory
cd $SPECTRE_HOME mkdir build
Run cmake:
cd $SPECTRE_HOME/build/ cmake -D CMAKE_C_COMPILER=gcc -D CMAKE_CXX_COMPILER=g++ -D CMAKE_Fortran_COMPILER=gfortran -D CHARM_ROOT=$CHARM_ROOT -D CMAKE_BUILD_TYPE=Release -D DEBUG_SYMBOLS=OFF -D BUILD_SHARED_LIBS=ON -D MEMORY_ALLOCATOR=JEMALLOC -D BUILD_PYTHON_BINDINGS=ON -D Python_EXECUTABLE=$SPECTRE_HOME/env/bin/python -Wno-dev "$@" $SPECTRE_HOME
Build executable
Build and executable
cd $SPECTRE_HOME/build/ make -j16 [Executable_name]
For example, to make the unit tests:
make -j16 unit-tests
And to run them:
ctest -R Unit -j16
Submit a job
Submit a job:
sbatch submit_script.sh
Here is an example submit scirpt (submit_script.sh):
#!/bin/bash -l # Standard output and error: #SBATCH -o ./tjob.out.%j #SBATCH -e ./tjob.err.%j # Initial working directory: #SBATCH -D ./ # Job Name: #SBATCH -J testrun # Number of MPI Tasks: #SBATCH --nodes=1 #SBATCH --ntasks-per-node=1 # Do not enable hyperthreading #SBATCH --ntasks-per-core=1 # Memory usage [MB] of the job is required, e.g. 2200 MB per cpu: #SBATCH --mem=158400 # for OpenMP: #SBATCH --cpus-per-task=72 # #SBATCH --mail-type=all #SBATCH --mail-user=[you_username]@aei.mpg.de # # Wall clock limit: #SBATCH --time=23:59:00 # # Load compiler and MPI modules with explicit version specifications, # consistently with the versions used to build the executable. module purge module load gcc/11 module load openmpi/4 module load boost/1.79 # module load doxygen/1.9.5 module load gsl/1.16 module load cmake/3.22 module load hdf5-serial/1.12.2 module load anaconda/3/2021.11 # Load Spack environment source /u/guilara/repos/spack/share/spack/setup-env.sh spack env activate env3_spectre # Define Charm paths export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin # Spectre directories export SPECTRE_HOME=/u/[your_username]/path/to/spectre export SPECTRE_BUILD_DIR=${SPECTRE_HOME}/build/ # Specify the run directory. Be sure to make this directory first export SPECTRE_RUN_DIR=/urania/ptmp/${USER}/spectre/Test/TestKerr # Load python environment source $SPECTRE_HOME/env/bin/activate # Print directories echo "Spectre home directory: ${SPECTRE_HOME}" echo "Build directory: ${SPECTRE_BUILD_DIR}" echo "Run directory: ${SPECTRE_RUN_DIR}" # Choose the executable and input file to run export SPECTRE_EXECUTABLE=${SPECTRE_BUILD_DIR}/bin/EvolveGhSingleBlackHole export SPECTRE_INPUT_FILE=${SPECTRE_RUN_DIR}/KerrSchild.yaml cd ${SPECTRE_RUN_DIR} # Set desired permissions for files created with this script umask 0022 # Set the path to include the build directory's bin directory export PATH=${SPECTRE_BUILD_DIR}/bin:$PATH # One thread for communication CHARM_PPN=$(expr ${SLURM_CPUS_PER_TASK} - 1) CHARM_PN=$((SLURM_NPROCS - SLURM_NNODES)) echo "Slurm tasks: ${SLURM_NTASKS}" echo "Slurm cpus per task: ${SLURM_CPUS_PER_TASK}" echo "Charm ppn: ${CHARM_PPN}" # Print loaded modules module list spack find # Run the program: srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71 > tjob.out # If you are starting from a checkpoint # srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71 \ # +restart SpectreCheckpoint000001 > tjob.out
For more examples refer to the Cobra and Raven user guides or to the SubmitScripts subdirectory in spectre/support/