Using MPI

Running MPI software

To use software that uses MPI, you must first load the appropriate compiler. Most people will use GCC and OpenMPI. If you have software that requires an MPICH compatible MPI, please send us mail at vacchelp@uvm.edu.

Run generic OpenMPI job

It is always good practice to remove all loaded modules prior to configuring your environment for MPI. Using the default version of GCC and of OpenMPI as examples, you load the modules with

$ module purge
$ module load gcc
$ module load openmpi

Note that the default openmpi module does not support Infiniband communication and will run on any node. See below for changes needed to use Infiniband.

Most MPI programs require that you use mpirun (or mpiexec) to start them. We have a precompiled test program that you can use as a test: /gpfs1/sw/examples/mpi/integration.

Copy the example first_job.sbat file from our examples directory, and review the Slurm directives and the command.

$ cp /gpfs1/sw/examples/mpi/first_mpi_job.sbat ~/
$ sbatch first_mpi_job.sbat

MPI enables us to request more than one node with #SBATCH --nodes=2 and use both of them. The #SBATCH --ntasks-per-node=2 determines how many copies of the MPI program will run. In MPI-speak, those are called ranks. Finally, most MPI programs will only use one CPU per rank. So, these are the three key Slurm directives for an MPI job.

MPI jobs can be run on a single node; you do NOT have to use more than one node. Only request more than one node if you need more CPUs than are available on a single node.

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=1

The sbatch command will print a job number, like

Submitted batch job 1729

and the job will produce a file called slurm-1729.out (your job number will be different).

Interactive programs and Slurm jobs

If your software will prompt for input, do NOT submit a job without making sure that all information is provided on the command line. If the software prompts, the job will run the full length of requested time waiting for you to type input and never do anything.

Run MPI job with Infiniband

To run an MPI job on a node or nodes that support Infiniband, you must specify that you want particular nodes and you must load an OpenMPI module that supports Infiniband.

1. Change first_mpi_job.sbat so that the line

#SBATCH --constraint=ib1

appears as the first #SBATCH line. Also in that file, you need to load an openmpi module that loads an OpenMPI with Infiniband support. The version number will have an appended -ib to indicate Infiniband support.

module load openmpi/5.0.5-ib

Note that MPI software that needs Infiniband must be compiled with that support included, and it must be compiled on a node that has Infinband. To compile such software, you must start a job with one of the ib constraints and compile it there. See Compiling with MPI for more information.