OpenMPI on Hypatia
MPI tasks may be easier to handle in a Slurm setup. Use the BilbyFarm for that.
Caution, experimental, not extremely well tested as of now
(Instructions provided by Patrick Brem for Vulcan, thank you!)
NEW: You may use /usr/local/bin/openmpiscript which has been adapted, and will be maintained.
The submit file submit_mpi.sub should look something like this:
universe = parallel executable = /usr/local/bin/openmpiscript initialdir = /home/user/workingdir arguments = /home/user/workingdir/executable arg1 arg2 etc machine_count = 8 log = log output = out.$(NODE) error = err.$(NODE) getenv = true should_transfer_files = yes when_to_transfer_output = on_exit queue
(Patrick Brem:) So far I have had bad experiences using relative paths in the executable, so to be safe use absolute paths there, since the child processes find themselves in $HOME initially even though setting an explicit initialdir above. Feel free to experiment there.
(Steffen Grunewald:) It's good practice also for other reasons to use full paths, also (and especially) for initialdir.
The arguments (of the openmpiscript executable) will be used to complete a mpirun invocation.
The machine_count is the number of cores (like Datura, unlike Minerva) MPI nodes get mapped on. Not always will you get full machines. (Kyohei, I was wrong.)
If you have suggestions how to get 4 cores per machine please add them here!
- Request 4 cores per machine, quadruple the resulting machinefile
condor_submit submit_mpi.sub
(last modified by Patrick Brem on 16 Jul 2014 / Steffen Grunewald on 17 Oct 2018)