Using Open MPI
From UMaine Supercomputer
Contents |
Introduction
As of July 5th, 2006, we are planning on including Open MPI in the next system upgrade of Bender, Fawlty and Kearney. Open MPI represents the collaboration of developers from current MPI implementations (see References below). In preliminary testing we have seen significant improvement in our benchmarking tests by merely switching from mpich-1.2.6 to Open MPI. There a few slight changes when using Open MPI which we detail below.
Compile
Open MPI is installed in /usr/local/ompi-<compiler>. For OS X we use the IBM XL compilers, for Linux we use GNU gcc/gfortran 4.1.0. The executables to be used are the same as before, i.e. mpicc, mpif90, mpif77 etc.
Run
Open MPI includes its own implementation of mpiexec which you must use instead of the one installed in /usr/local/mpiexec. At this point in time, /usr/local/ompi-<compiler>/bin/mpiexec does not pick up the number of nodes granted by Torque, so you need to use the -np flag. Here's an example:
#!/bin/bash #PBS -q default #PBS -l nodes=4:ppn=2:myrinet #PBS -l walltime=5:00:00 #PBS -o out.darwin.8 #PBS -e err.darwin.8 #PBS -V Mpich-1.2.7, using Ethernet usr/local/mpiexec/bin/mpiexec -comm=mpich-l # Mpich-1.2.7, using Myrinet usr/local/mpiexec/bin/mpiexec -comm=gm ./xhpl # Open MPI, using Ethernet usr/local/ompi-xl/bin/mpiexec -np 8 -mca btl self,tcp ./xhpl # Open MPI, using Myrinet usr/local/ompi-xl/bin/mpiexec -np 8 -mca btl self,gm ./xhpl
You can tune Open MPI in a large number of ways at runtime using their mpiexec implementation. For more information, see the Open MPI website.
Errata
- For linux you may need to add -path /usr/local/ompi-gnu/bin to the mpiexec arguments if you get errors complaining about not being able to find orted.
