Calhoun (SGI Altix XE 1300 Linux Cluster) User Resources

This page includes detailed information on using the HPC resource Calhoun.  It aims to answer your most common questions that are beyond the depth of the Quickstart guide. If you have questions after consulting this page, please send an email to technical support or call 612-626-0802.

Science and Math Libraries

Below is some information about how to use some common scientific computation libraries on Calhoun.

FFTW - Fast Fourier Transform Libraries

The FFTW libraries are located within directories at /soft/fftw/. Below are some sample commands for compiling a program using the Intel C compiler and the FFTW version 3 libraries:

module load intel
module load fftw
icc mycode.c -lfftw3

Intel MKL - Math Kernel Libraries (Includes LAPACK and ScaLAPACK Linear Algebra Libraries)

The Intel MKL libraries provide fast mathematical algorithms, including linear algebra algorithms. The MKL libraries are located in directories at /soft/intel/mkl. Below are sample commands for compiling a program using the Intel C compiler and the MKL linear algebra (lapack) libraries:

module load intel 
module load mkl 
icc mycode.c -lmkl_lapack

The MKL libraries also include ScaLAPACK parallel linear algebra libraries. These libraries are for use with programs doing parallel computation using MPI message passing. Below are sample commands for compiling a program using the Intel C compiler, OpenMPI message passing, and the ScaLAPACK linear algebra libraries.

module load intel
module load ompi/intel
module load mkl
mpicc mycode.c -lmkl_scalapack_ilp64

MPI - Message Passing Interface for Parallel Computation

Calhoun relies primarily on the OpenMPI version of MPI for parallel communication.

OpenMPI

There are different versions of OpenMPI installed, some of which are for use with specific compilers. To see all available OpenMPI versions use the command:

module avail ompi

The default OpenMPI version is for use with the Intel compilers. To load the default version use the command:

module load ompi/intel

Most of the commands for compiling programs using MPI begin with the characters "mpi". Below is an example command for compiling an MPI functional program written in C++:

mpic++ testprogram.cpp

The command to run an OpenMPI program is mpirun. A sample command for running a program named a.out on 8 processors using OpenMPI is shown below:

mpirun -np 8 ./a.out

Jobs of significant length must use the PBS job scheduling queue to run; see the example PBS job script toward the end of this page. Jobs not in the PBS job scheduling queue are automatically stopped after a short period of time.

By default, OpenMPI will use shared memory for processors residing on the same node. That is, all of your job's processes that have a node in common will share the same memory space. To force OpenMPI to only communicate over Infiniband, and thus force it to have seperate memory for each process, you must change your mpirun options:

mpirun -mca btl openib -np 8 ./a.out

For some applications, this can result in increased performance, but the application's memory consumption will be much larger.

Performance tuning

To improve the performance and stability for very large jobs, add the following to your .bashrc file:

ulimit -s unlimited
ulimit -n 4096

This will maximize the stack size and to provide control over the resources available.

Common Intel Compiler flags

Recommended options:

-ipo -O3 -no-prec-div (or just use the option: -fast)

Example PBS Script File

Below is an example PBS queue script for an 8-processor single node OpenMPI job that will run up to 10 hours and use up to 4 gigabytes of RAM:

#!/bin/bash -l
#PBS -l walltime=10:00:00,mem=4gb,nodes=1:ppn=8
#PBS -m abe
cd ~/TEST/mytest_pbs
module load intel
mpirun -np 8 ./a.out

The script should be saved as a file. To submit a PBS script to the queue use the command:

qsub scriptname

Scratch Spaces

Local scratch is available on 86 of the compute nodes. To use local scratch, you need to request the scratch feature in your submission script, e.g.:

#PBS -l nodes=4:scratch:ppn=8

Scratch directories are not backed up. All files in the scratch directories that have not been modified for 14 days will be deleted. The scratch local to each node is at /scratch.

This scratch space is in addition to the shared scratch spaces referenced elsewhere.