Calhoun (SGI Altix XE 1300 Linux Cluster) Quickstart Guide

This guide will provide you with the basic information needed to get up and running on Calhoun. For detailed operations information, see the User Resources section for this resource. If you run into issues using Calhoun, please check out the FAQ.

Login Procedure

Like most MSI resources, access to Calhoun requires that you connect through login.msi.umn.edu or nx.msi.umn.edu. Please see MSI's interactive connections FAQ, and use calhoun.msi.umn.edu as the hostname.

Compilers

The following tables summarize the compilers, MPI implementations, and software installed on Calhoun. In the subsequent commands and scripts, we have used Intel compilers for Fortran or C programs. To use other compilers, follow the same instructions and replace the module files and the executables based on the table below.

Serial code and OpenMP:

Compiler Commands Module
GNU gcc, g++, gfortran gcc
Intel icc, icpc, ifort intel

MPI:

Software Commands Module
OpenMPI mpicc, mpiCC, mpicxx, mpif77, mpif90 ompi/gnu, ompi/intel, ompi/pathscale

Compiling Program Codes:

Serial Codes

Example commands for compiling codes with the Intel compilers are shown below.

Fortran

module load intel

ifort -o test -O3 mycode.f

C

module load intel

icc -o test -O3 mycode.c

Users can select different compiling options for optimizing the performance. Please see the man page (e.g., use the commands man ifort or man icc) for the available options.

MPI codes

There are multiple MPI implementations on Calhoun. Modules for all of them are mutually exclusive. That is, you can only have one MPI implementation loaded into your environment at any given time. The recommended MPI for use on Calhoun is OpenMPI.

Fortran

module load intel
module load ompi/intel
mpif90 -o test mycode.f

C

module load intel
module load ompi/intel
mpicc -o test mycode.c

OpenMP codes

It is important to use the option: -openmp

Fortran

module load intel
ifort -o test -O3 -openmp mycode.f

More Examples

Working examples of how to build a variety of applications, including those using libraries such as MKL, are posted on Calhoun in the directory /soft/examples. Each subdirectory contains an example of how to build and run a test case illustrating the use of a particular library or technique.

How to submit batch jobs to the queue using PBS

We use the PBS queueing system to ensure the machine is being used fairly and to its full potential. To submit jobs to the queue, all you need to do is create a PBS script file and use the command qsub to submit. For information on the available queues, see queue information on the main policy page.  Calhoun nodes have 16 GB of memory, and jobs should ask for 15GB of memory per node (1875MB per processor core) or less.

Serial jobs

While single-core serial jobs are allowed to run on Calhoun, it is still recommended that each job fully occupy a node's 8 cores. There are also few limits on the number of jobs that can be submitted to Calhoun. This means that huge numbers of jobs could be queued on Calhoun at any time. Fairshare will ensure jobs from users in groups that have lightly used Calhoun recently will be run within a reasonable time window. Please feel free to contact user support for assistance in running these kinds of jobs (e-mail: help@msi.umn.edu or call 612-626-0802).

Multiple copies of a simulation using different inputs can be collected into one queue job which uses an entire node; see the example below.

The following is a PBS script for a job that will run on 1 node using all 8 cores for 8 different tasks. Each of the cores needs 1875 mb memory. Save the script as 'myscript.pbs'

#!/bin/bash -l
#PBS -l walltime=01:00:00,pmem=1875mb,nodes=1:ppn=8
#PBS -m abe
module load intel
pbsdsh -n 0 /scratch2/username/task1 &
pbsdsh -n 1 /scratch2/username/task2 &
pbsdsh -n 2 /scratch2/username/task3 &
pbsdsh -n 3 /scratch2/username/task4 &
pbsdsh -n 4 /scratch2/username/task5 &
pbsdsh -n 5 /scratch2/username/task6 &
pbsdsh -n 6 /scratch2/username/task7 &
pbsdsh -n 7 /scratch2/username/task8 &
wait

Example of PBS script to submit a parallel MPI batch job

Below is an example script for running an MPI job using OpenMPI and the Intel compilers:

#!/bin/bash -l
#PBS -l walltime=1:00:00,mem=14gb,nodes=4:ppn=8
#PBS -m abe
module load intel
module load ompi/intel
/usr/bin/time mpirun -np 32 ./a.out

Below is a script for a 16-processor MPI job that will use a total of 4 GB of memory and run up to 10 hours on 2 nodes using 8 processors per node:

#!/bin/bash -l
#PBS -l walltime=10:00:00,mem=4gb,nodes=2:ppn=8
#PBS -m abe
cd ~/TEST/mytest_pbs
module load intel
module load ompi/intel
/usr/bin/time mpirun -np 16 ./a.out

Below is a PBS script for a 16-processor MPI job that will run up to 10 hours on 2 nodes using 8 processors per node. This time the script is using "pmem=500mb" to request 500 MB of memory per task:

#!/bin/bash -l
#PBS -l walltime=10:00:00,pmem=500mb,nodes=2:ppn=8
#PBS -m abe
cd ~/TEST/mytest_pbs
module load intel
module load ompi/intel
/usr/bin/time mpirun -np 16 ./a.out

Additional examples of submission scripts, as well as examples on how to use various versions of MPI, can be found in the User Resources section for this resource.

PBS Commands

Submitting a batch job:

qsub myscript.pbs

Submitting a batch job to the long queue (for extended walltimes):

qsub -q long myscript.pbs

Monitoring queue status:

showq

qstat -f

Monitoring job's status:

checkjob jobnumer

Canceling a job:

qdel jobnumber

Submitting a job for an interactive session:

qsub -I myscript.pbs

Note, your command prompt will hang until the interactive session starts. When your session starts, you will be logged into a compute node.

Interactively run MPI programs with:

mpirun ...

Logging out of the compute node ends an interactive session.