ItascaSB Quickstart Guide

Introduction

ItascaSB consists of 51 Sandy-Bridge nodes (blades). Each node contains 2 eight-core Sandy bridge E5-2670 processor chips (2.6 GHz), and one of three possible amounts of memory: 64GB, 128GB and 256GB.  Based on the memory size, ItascaSB nodes are configured into three queues named as sb, sb128 and sb256 respectively (Table 1).  

There are 35 nodes with 64GB of memory (sb queue), 8 nodes with 128GB of memory (sb128 queue) and 8 nodes with 256GB of memory (sb256 queue).  The 64GB nodes each include 112GB of local scratch space. The 128GB and 256GB nodes each include 534GB of local scratch space.  Larger memory and larger local disk nodes target users with greater demands of per-core or node memory, and users  who need large local scratch drives for Input/Output operations.

In contrast to previous Nehalem and Westmere processors, the Sandy Bridge processors utilize the Advanced Vector Extensions (AVX) instruction set.  AVX allows Single Instruction Multiple Data (SIMD) operations on Intel processors, which dramatically improves throughput of floating-point operations if the code is well vectorizable.

MSI home directories and the Lustre filesystem (/lustre) are accessible from the Sandy Bridge nodes.  The nodes are connected through Itasca's infiniband fabric.

Use of ItascaSB

Users can access ItascaSB nodes interactively by typing one of the following commands on an Itasca login node.

qsub -I -q sb -l walltime=2:30:00,nodes=1:ppn=16
qsub -I -q sb128 -l walltime=3:30:00,nodes=1:ppn=16
qsub -I -q sb256 -l walltime=1:30:00,nodes=1:ppn=16

ItascaSB nodes have the same software modules available as Itasca.

Intel Optimization Flags

For programs compiled with the Intel compilers users should recompile their codes adding the -xAVX flag to the existing Intel compiler options used on Itasca so that the executables can take advantage of the Advanced Vector Extensions to improve application performance on the Sandy Bridge nodes. Please note that the executables built with -xAVX flag  will not run on the original Itasca nodes. Further information on how to use the Intel compilers to compile and optimize code using the AVX instruction set can be found on the Intel website.

GNU Optimization Flags

For programs compiled with the GNU compilers, the flag -mavx enables the Advanced Vector Extension (AVX) instructions.  Alternatively compiling with the pair of flags -march=corei7-avx  and -mtune=corei7-avx will also enable AVX, and will optimize the program using all available instructions.  The pair of flags also enable some AVX instructions not available under -mavx.  Binaries built with AVX extensions will not run on the original Itasca nodes.  Further information on GNU compiler options is available on the GCC website.

Batch jobs on ItascaSB

Three queues are set up for users to run batch jobs according to the memory needs. Users can use one of the following commands to submit jobs:

qsub -q sb job_script.pbs
qsub -q sb128 job_script.pbs
qsub -q sb256 job_script.pbs

where sb, sb128 and sb256 are for the nodes having memory of 64 GB, 128 GB and 256 GB respectively.  The job script format and the syntax for submitting jobs to the Sandy Bridge nodes are the same as used on Itasca, with further details available in Itasca's quick start guide.

Table 1: Queue and Throttling Policies
Queue name Number of Nodes
(16 cores/node)
Wallclock Limit
(#PBS -l walltime=)
Total Memory Limit
(#PBS -l mem=)
Per-core Memory Limit
(#PBS -l pmem=)
Local Scratch
(GB/node)
mem64
(-q sb)
35 nodes (560 cores) 48 hours 2215gb 3750mb 112 GB
mem128
(-q sb128)
8 nodes (128 cores) 96 hours 1016gb 7500mb 534 GB
mem256
(-q sb256)
8 nodes (128 cores) 96 hours 2032gb 15000mb 534 GB