Astrophysical and Cosmological Relativity Group

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== OpenMPI on Hypatia =====

MPI tasks may be easier to handle in a Slurm setup. Use the [[:hypatia_internal:bilby_farm:start|BilbyFarm]] for that.

**Caution, experimental, not extremely well tested as of now**

(Instructions provided by Patrick Brem //for Vulcan//, thank you!)

**NEW:** You may use ''/usr/local/bin/openmpiscript'' which has been adapted, and will be maintained.
 
The submit file ''submit_mpi.sub'' should look something like this:

<blockquote>
  universe = parallel
  executable = /usr/local/bin/openmpiscript
  initialdir = /home/user/workingdir
  arguments = /home/user/workingdir/executable arg1 arg2 etc
  machine_count = 8
  log = log
  output = out.$(NODE)
  error = err.$(NODE)
  getenv = true
  should_transfer_files = yes
  when_to_transfer_output = on_exit
  queue
</blockquote>


(Patrick Brem:) So far I have had bad experiences using relative paths in the executable, so to be safe use absolute paths there, since the child processes find themselves in $HOME initially even though setting an explicit initialdir above. Feel free to experiment there.

(Steffen Grunewald:) It's good practice also for other reasons to use full paths, also (and especially) for **initialdir**.
The ''arguments'' (of the ''openmpiscript'' executable) will be used to complete a ''mpirun'' invocation.
The ''machine_count'' is the number of **cores** (like Datura, unlike Minerva) MPI nodes get mapped on. Not always will you get full machines. //(Kyohei, I was wrong.)//

If you have suggestions how to get 4 cores per machine please add them here!

  * Request 4 cores per machine, quadruple the resulting machinefile


<blockquote>
  condor_submit submit_mpi.sub
</blockquote>


(last modified by Patrick Brem on 16 Jul 2014 / Steffen Grunewald on 17 Oct 2018)