Itasca (HP ProLiant BL280c G6 Linux Cluster)

Itasca

Itasca is an HP Linux cluster with 1,091 HP ProLiant BL280c G6 blade servers, each with two-socket, quad-core 2.8 GHz Intel Xeon X5560 "Nehalem EP" processors sharing 24 GB of system memory, with a 40-gigabit QDR InfiniBand (IB) interconnect. In total, Itasca consists of 8,728 compute cores and 24 TB of main memory.

 

User Guide

For a quick introduction to working on Itasca, see the quick start guide.

Hardware and Configuration

Itasca has 2,186 Intel Xeon X5560 "Nehalem EP"-class processors. Each processor has four cores, 8 MB cache, and communicates with memory on a QuickPath Interconnect (QPI) interface.

  • 1,134 compute nodes
  • 2 interactive nodes
  • 5 server nodes
  • 8,744 total cores
  • 31.3 TB total main memory
  • Suitable for: large MPI jobs
     

Each node:

  • Processors: Two quad-core 2.8 GHz Intel Xeon X5560 "Nehalem EP"-class processors
  • Memory: 24 GB main memory

ItascaSB

ItascaSB  is an expansion made by adding 51 "Sandy Bridge" blades to Itasca system. The "Sandy Bridge" processor architecture allows for higher computation rates for some calculations. Detailed information is provided at ItascaSB page.

Jay

Jay is a special high-end analysis and visualization system installed especially to enhance the effectiveness of Itasca, and for efficient transfers of large datasets. It has fast processors coupled to large memory, and uses the Itasca Lustre scratch system for high-performance storage. The system is connected to Itasca's IP and IB network and to the campus backbone using high-bandwidth networks for faster data transfer. Access is expected to be primarily through advanced reservation.

Network

All of the systems within Itasca are interconnected with a 40-gigabit InfiniBand fabric in a dual-tree 2:1 blocking fashion used for interprocess communication (IPC). The InfiniBand fabric is a high-bandwidth, low-latency network, the intent of which is to accommodate high-speed communication for large MPI jobs. The nodes are also interconnected with two 1-gigabit ethernet networks for administration and file access, respectively.

Home Directories and Disks

Itasca home directories are as described on the MSI Home Directories page and are shared with Cascade. MSI central project spaces are available on the Itasca interactive nodes by request.

Scratch Spaces

There are two kinds of scratch space on Itasca: a large scratch partition that is shared across all compute and interactive nodes, and local scratch space on each compute node.

The large and shared scratch partition (/lustre) has 538 TB of Lustre file system storage. This is useful for voluminous input and output, including checkpoint/restart files. There is no disk quota on the shared scratch file system. However, a scratch clean up process runs daily on the system and all files in the scratch file system that have not been modified for 30 days will be deleted. Users may request exceptions to the scratch clean up by sending email to help@msi.umn.edu.

The local scratch on each compute node (/scratch) is not visible on any other compute node, nor is it visible on the interactive nodes. It is only useful as temporary space for an application to write to and read back during a single job, and is not available after the job ends.

Queues and Throttling Policies

Given Itasca's size and desired usage, compute nodes on this system are not shared. This means that serial jobs (anything less than 8 cores, as there are 8 cores per node) are not allowed; these jobs can be run on other MSI systems. Itasca allows up to 2 running and 3 idle jobs per user. Node totals per queue do not add up to total system cores, as some queues overlap. The memory of 22 GB per node is the maximum that a job can request.

Queues visible on the system but not listed below are unavailable for general use.

Itasca Queue Policies

Default Queue

(-q batch)

Development Queue

(-q devel)

Long Queue

(-q long)

Jay (Visualization)

(-q jay)

1,086 nodes or 8,688 cores

 

32 nodes or 256 cores

 

28 nodes or 224 cores

 

1 node or 8 cores

24hr run-time

2hr run-time

48hr run-time

24hr run-time

For the Sandy Bridge node queue policies see the ItascaSB webpage.