Example Submission Scripts

From UMaine Supercomputer

Intro

These are a few examples of job submission scripts

Case 1:

This is a script to submit a myrinet job on Kearney2

#!/bin/bash
#PBS -l nodes=24:ppn=2:myrinet
#PBS -l walltime=24:00:00
#PBS -l arch=x86-32
#PBS -q default
cd /home/john/cluster_compare/hpl/kearney2/Myrinet-openmpi
/usr/local/ompi-gnu/bin/mpiexec --mca btl gm,self -np 48  ./xhpl

Now what each line means:

#!/bin/bash                      --> interpret commands (except PBS directives) using the bash shell
#PBS -l nodes=24:ppn=2:myrinet   --> requesting 24 nodes with:
                                     two processors per node
                                     myrinet  (we could have requested GigE instead)
#PBS -l walltime=24:00:00        -->  requesting a maximum run time of 24 hours
#PBS -l arch=x86-32              -->  x86 32 bit architecture (currently the Pentium IIIs)
#PBS -q default                  -->  use the default queue
cd /home/john/...                -->  cd to this directory before running
/usr/local/ompi-gnu...           -->  /usr/local/ompi-gnu/bin/mpiexec is the version of mpiexec we use with open mpi
                                    --mca btl gm,self -np 48  this tells open mpi to use gm (Myrinet) to communicate
                                                                  and to run 48 processes...normally np = nodes*ppn
                                    ./xhpl  this is the program to run... in this case it is in the directory we
                                            just changed to, but if you use the full path, it can be anywhere

Notes:

1. With openmpi, it is necessary to specify the number of processes to run along with the number of nodes and processors per node. This specification may seem redundant, and normally is.
2. Please always set the maximum wall time to the time you really need with a small buffer.
3. You should ALWAYS include 'self' in communication...otherwise your program will crash in weird and hard-to-debug ways

Case 2

Now the same job, but with GigE and 8 rather than 24 nodes:

#!/bin/bash
#PBS -l nodes=8:ppn=2
#PBS -l walltime=24:00:00
#PBS -l arch=x86-32
#PBS -q default
cd /home/john/cluster_compare/hpl/kearney2/GiGE
/usr/local/ompi-gnu/bin/mpiexec -np 16 --mca btl tcp,self  ./xhpl

Notes:

1. The number of nodes is adjusted, and myrinet is not specified. The system will first look for GiGE nodes, and if none are available will try myrinet nodes.
2. The directory is different, but that is just a user preference not a requirement
3. The number or processes on the mpiexec line changes, and we specify tcp rather than gm for the communication
4. self communication is still specified