Job Submission and Scheduling (PBS Scripts)

Many MSI systems use job queues to efficiently and fairly manage when computations are executed.  When computational jobs are submitted to a job queue they wait in the queue in line until the appropriate computational resources are available.

The queuing system which MSI uses is called PBS, which stands for Portable Batch System.  To submit a job to a PBS queue users create PBS job scripts.  PBS job scripts contain information on the resources requested for the calculation, as well as the commands for executing the calculation.

PBS Script Format

A PBS job script is a small text file containing information about what resources a job requires, including time, number of nodes, and memory.   The PBS script also contains the commands needed to begin executing the desired computation.  A sample PBS job script is shown below.

#!/bin/bash -l
#PBS -l walltime=8:00:00,nodes=3:ppn=8,pmem=1000mb
#PBS -m abe

cd ~/program_directory
module load intel
module load ompi/intel
mpirun -np 24 program_name < inputfile > outputfile

The first line in the PBS script defines which type of shell the script will be read with (how the system will read the file).  It is recommended to make the first line #!/bin/bash -l   Commands for the PBS queing system begin with #PBS.  The second line in the above sample script contains the PBS resource request.   The sample job will require 8 hours, 3 nodes each with 8 processor cores per node (ppn), and 1000 megabytes of RAM per processor core (pmem).  The resource request must contain appropriate values; if the requested time, processors, or memory are not suitable for the hardware the job will not be able to run.

The two lines containing #PBS -m abe , and #PBS -M , are both commands having to do with sending message emails to the user.  The first of these lines instructs the PBS system to send a message email when the job aborts, begins, or ends.  The second command specifies the email address to be used.  Using the message emails is recommended because the reason for a job failure can often be determined using information in the emails.

The rest of the sample PBS script contains the commands which will be executed to begin the calculation.  A PBS script should contain the appropriate change directory commands to get to the job execution location (the script will start in the user home directory).  A PBS script also needs to contain module load commands for any software modules that the calculation might need.  The last lines of a PBS script contain commands used to execute the calculation.  In the above example the final line contains an execution command to start a program which uses MPI communication to run on 24 processor cores.

Submitting Job Scripts

Once a job script is made it is submitted using the qsub command:

qsub -q quename scriptname

Here quename is the name of the queue being submitted to, and scriptname is the name of the job script.  The -q quename portion of the command may be ommitted, in which case the job would be submitted to whichever queue is set as the default.  Alternatively, the queue specification can be placed inside the job script (see below).

To view all of the jobs submitted by a particular user use the command:

qstat -u username

This command will display the status of the specified jobs, and the associated job ID numbers. The command qstat by itself will show all jobs on the system.

To cancel a submitted job use the command:

qdel jobIDnumber

Here jobIDnumber should be replaced with the appropriate job ID number determined by using the qstat command.

PBS Script Commands

Below is a table summarizing some commands that can be used inside PBS job scripts.  The first two commands (interpreter specification, and resource request) are required, while the other commands are optional.  Each of the below PBS commands is meant to go on a single line within a PBS script.

PBS command Effect
#!/bin/bash -l Specifies how the PBS file should be read (by the bash interpreter). A statement like this is required to be the first line of a PBS script.
#PBS -l walltime=2:00:00,nodes=1:ppn=8,pmem=2500mb The resource request (required). The resource request must specify the job walltime (hours:minutes:seconds), number of nodes, and processor cores per node (ppn). It is recommended to specify either the required memory per processor core (pmem), or the required total memory (mem).
#PBS -m abe Makes the PBS system send message emails when the job aborts, begins, or ends.
#PBS -M Specifies the email address that should be used when the PBS system sends message emails.
#PBS -N jobname Specifies a name for the job that will appear in the job queue.
#PBS -o output_filename Directs the job standard output to be placed in the named file.
#PBS -e error_filename Directs the job error output to be placed in the named file.
#PBS -q queue_name Specifies that the job should be run in the named queue.

Combining Calculations Using PBS Scripts (pbsdsh)

Warning: Jobs using more than 16 nodes occasionally hang when using pbsdsh.

PBS scripts provide a way to combine smaller calculations into single jobs.  One of the most effective ways to do this is using the pbsdsh command.  The pbsdsh command is used to have specific cores execute shell scripts within a multi-core job.  An example job script using pbsdsh is shown below:

#!/bin/bash -l
#PBS -l walltime=8:00:00,nodes=2:ppn=8,pmem=1000mb
#PBS -m abe

cd ~/job_directory
pbsdsh -n 0 ~/jobdirectory/ &
pbsdsh -n 1 ~/jobdirectory/ &
pbsdsh -n 2 ~/jobdirectory/ &
pbsdsh -n 3 ~/jobdirectory/ &
pbsdsh -n 4 ~/jobdirectory/ &
pbsdsh -n 5 ~/jobdirectory/ &
pbsdsh -n 6 ~/jobdirectory/ &
pbsdsh -n 7 ~/jobdirectory/ &
pbsdsh -n 8 ~/jobdirectory/ &
pbsdsh -n 9 ~/jobdirectory/ &
pbsdsh -n 10 ~/jobdirectory/ &
pbsdsh -n 11 ~/jobdirectory/ &
pbsdsh -n 12 ~/jobdirectory/ &
pbsdsh -n 13 ~/jobdirectory/ &
pbsdsh -n 14 ~/jobdirectory/ &
pbsdsh -n 15 ~/jobdirectory/ &


This job script requests two nodes, each having 8 cores (for 16 cores total), and then uses pbsdsh commands to have each core execute a separate shell script.  Within a PBS job each core is given a label number beginning with 0.  pbsdsh can be used to have a core with a specific label number execute a specific script.  The line pbsdsh -n 3 ~/jobdirectory/  makes the core labeled 3 execute the script located at ~/jobdirectory/  Usually it is necessary to give the full path to the shell script when using the pbsdsh command (~ is interpreted as the user home directory).

When using this method of combining calculations each shell (.sh) script should only attempt to perform a serial computation using one core.  Each shell script will begin execution from the user's home directory, and each shell script will need to include any appropriate change directory (cd) commands from that location.  Additionally, each shell script will begin execution with only the default software modules loaded, so each shell script needs to contain the module load commands for any further software modules required for its calculations.

If the shell scripts that are to be executed have names in a numerical progression it is possible to use a pbsdsh loop.  For example, the below loop is equivalent to the above list of pbsdsh commands:

for proc in `seq 0 15`
pbsdsh -n $proc ~/jobdirectory/script$ &