Differences

This shows you the differences between two versions of the page.

--- clusters:urania:spectre [2023/12/06 11:23] – removed - external edit (Unknown date) 127.0.0.1
+++ clusters:urania:spectre [2023/12/06 11:23] (current) – ↷ Page moved from clusters_internal:urania:spectre to clusters:urania:spectre Steffen Grunewald
@@ Line 1: / Line 1: @@
+====== Installing spectre ======
+You can refer to {{ https://spectre-code.org/installation.html | the spectre installation instructions }}. Here is what Guillermo has tried.
+Git clone on your home (sub)directory:
+<code>
+git clone git@github.com:sxs-collaboration/spectre.git
+</code>
+In the spectre home directory create a python environment (Note that the name 'env' is ignored by git, so its convenient to name the environment like this)
+<code>
+cd [spectre_home]/
+python -m venv env
+</code>
+Activate the environment
+<code>
+source env/bin/activate
+</code>
+And pip install the required python packages summarized in the support/ subdirectory:
+<code>
+pip install -r support/Python/requirements.txt -r support/Python/dev_requirements.txt
+</code>
+Load the necessary modules (Can put this in a script to avoid doing it everytime):
+<code>
+module purge
+module load gcc/11
+module load openmpi/4
+module load boost/1.79
+module load gsl/1.16
+module load cmake/3.22
+module load hdf5-serial/1.12.2
+module load anaconda/3/2021.11
+</code>
+Guillermo has already installed {{ https://charm.readthedocs.io/en/latest/quickstart.html | Charm }} and the remaining modules in an {{https://spack-tutorial.readthedocs.io/en/latest/tutorial_basics.html | Spack }} environment (Please ask if you don't have access):
+<code>
+source /u/guilara/repos/spack/share/spack/setup-env.sh
+spack env activate env3_spectre
+</code>
+Define the charm paths and spectre home
+<code>
+export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc
+export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin
+export SPECTRE_HOME=/u/[your_username]/path/to/spectre
+</code>
+Activate the python environment (if you haven't already)
+<code>
+source $SPECTRE_HOME/env/bin/activate
+</code>
+===== Configure spectre =====
+In the spectre home directory make a build directory
+<code>
+cd $SPECTRE_HOME
+mkdir build
+</code>
+Run cmake:
+<code>
+cd $SPECTRE_HOME/build/
+cmake -D CMAKE_C_COMPILER=gcc -D CMAKE_CXX_COMPILER=g++ -D CMAKE_Fortran_COMPILER=gfortran -D CHARM_ROOT=$CHARM_ROOT -D CMAKE_BUILD_TYPE=Release -D DEBUG_SYMBOLS=OFF -D BUILD_SHARED_LIBS=ON -D MEMORY_ALLOCATOR=JEMALLOC -D BUILD_PYTHON_BINDINGS=ON -D Python_EXECUTABLE=$SPECTRE_HOME/env/bin/python -Wno-dev "$@" $SPECTRE_HOME
+</code>
+===== Build executable =====
+Build and executable
+<code>
+cd $SPECTRE_HOME/build/
+make -j16 [Executable_name]
+</code>
+For example, to make the unit tests:
+<code>
+make -j16 unit-tests
+</code>
+And to run them:
+<code>
+ctest -R Unit -j16
+</code>
+====== Submit a job ======
+Submit a job:
+<code>
+sbatch submit_script.sh
+</code>
+Here is an example submit scirpt (submit_script.sh):
+<code>
+#!/bin/bash -l
+# Standard output and error:
+#SBATCH -o ./tjob.out.%j
+#SBATCH -e ./tjob.err.%j
+# Initial working directory:
+#SBATCH -D ./
+# Job Name:
+#SBATCH -J testrun
+# Number of MPI Tasks:
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=1
+# Do not enable hyperthreading
+#SBATCH --ntasks-per-core=1
+# Memory usage [MB] of the job is required, e.g. 2200 MB per cpu:
+#SBATCH --mem=158400
+# for OpenMP:
+#SBATCH --cpus-per-task=72
+#
+#SBATCH --mail-type=all
+#SBATCH --mail-user=[you_username]@aei.mpg.de
+#
+# Wall clock limit:
+#SBATCH --time=23:59:00
+#
+# Load compiler and MPI modules with explicit version specifications,
+# consistently with the versions used to build the executable.
+module purge
+module load gcc/11
+module load openmpi/4
+module load boost/1.79
+# module load doxygen/1.9.5
+module load gsl/1.16
+module load cmake/3.22
+module load hdf5-serial/1.12.2
+module load anaconda/3/2021.11
+# Load Spack environment
+source /u/guilara/repos/spack/share/spack/setup-env.sh
+spack env activate env3_spectre
+# Define Charm paths
+export CHARM_ROOT=/u/guilara/charm/mpi-linux-x86_64-smp-gcc
+export PATH=$PATH:/u/guilara/charm/mpi-linux-x86_64-smp-gcc/bin
+# Spectre directories
+export SPECTRE_HOME=/u/[your_username]/path/to/spectre
+export SPECTRE_BUILD_DIR=${SPECTRE_HOME}/build/
+# Specify the run directory. Be sure to make this directory first
+export SPECTRE_RUN_DIR=/urania/ptmp/${USER}/spectre/Test/TestKerr
+# Load python environment
+source $SPECTRE_HOME/env/bin/activate
+# Print directories
+echo "Spectre home directory: ${SPECTRE_HOME}"
+echo "Build directory: ${SPECTRE_BUILD_DIR}"
+echo "Run directory: ${SPECTRE_RUN_DIR}"
+# Choose the executable and input file to run
+export SPECTRE_EXECUTABLE=${SPECTRE_BUILD_DIR}/bin/EvolveGhSingleBlackHole
+export SPECTRE_INPUT_FILE=${SPECTRE_RUN_DIR}/KerrSchild.yaml
+cd ${SPECTRE_RUN_DIR}
+# Set desired permissions for files created with this script
+umask 0022
+# Set the path to include the build directory's bin directory
+export PATH=${SPECTRE_BUILD_DIR}/bin:$PATH
+# One thread for communication
+CHARM_PPN=$(expr ${SLURM_CPUS_PER_TASK} - 1)
+CHARM_PN=$((SLURM_NPROCS - SLURM_NNODES))
+echo "Slurm tasks: ${SLURM_NTASKS}"
+echo "Slurm cpus per task: ${SLURM_CPUS_PER_TASK}"
+echo "Charm ppn: ${CHARM_PPN}"
+# Print loaded modules
+module list
+spack find
+# Run the program:
+srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71  > tjob.out
+# If you are starting from a checkpoint
+# srun ${SPECTRE_EXECUTABLE} ++ppn ${CHARM_PPN} --input-file ${SPECTRE_INPUT_FILE} +pemap 0-34,36-70 +commap 35,71 \
+#	+restart SpectreCheckpoint000001 > tjob.out
+</code>
+For more examples refer to the {{https://docs.mpcdf.mpg.de/doc/computing/cobra-user-guide.html | Cobra}} and {{ https://docs.mpcdf.mpg.de/doc/computing/raven-user-guide.html | Raven }} user guides or to the SubmitScripts subdirectory in spectre/support/