Koronis SGI Constellation

KoronisNOTE: Koronis was decommissioned on March 17, 2014. Information about retrieving and transferring Koronis data can be found here. Please contact help@msi.umn.edu with any questions.

Koronis is a constellation of SGI systems, including foremost an Altix UV 1000 server with 1,140 compute cores (190 6-core Intel Xeon X7542 "Westmere” processors at 2.66 GHz) and 2.96 TiB of globally-addressable shared memory in a single system image.

Additional compute resources in Koronis include two UV100 systems each with 66 compute cores (11 6-core Intel Xeon X7542 processors) and 352 GiB of RAM for development -- one includes four Nvidia Tesla S2050 cards; a UV10 system with 24 cores (4 6-core Intel Xeon X7542) and 256 GiB of RAM and an Nvidia Quadro FX1800 display adaptor; and three CY1103 systems each with 12 cores (2 6-core Intel Xeon X5670s at 2.93 GHz), 48 GiB of RAM and Nvidia Quadro FX5800 adapters dedicated to visualization, including SGI's remote visualization software.

All systems in this constellation run SUSE Linux Enterprise Server 11 SP1 with SGI Performance Suite.

The salient feature of Koronis' UV1000 system is its large-scale, cache-coherent Non-Uniform Memory Architecture (ccNUMA). It allows for efficient processing of jobs requiring large amounts of shared memory exceeding what is available on other MSI HPC resources. Although codes running on the system can be programmed using a message passing model, communication among the processes will occur via shared memory. OpenMP or other threaded codes should run well on this resource.

 

User Guide

For a quick introduction to working on Koronis, see the quick start guide.

Hardware Summary

  • Compute
    • UV1000: 1140 cores
    • UV100: 66 cores
    • UV100: 66 cores plus 4 Nvidia Tesla 2050 "Fermi" cards
  • Graphics
    • UV10: 24 cores plus Nvidia Quadro FX1800 card
    • CY1103: three systems each with 12 cores plus Nvidia Quadro FX5800 cards

Network

Within each ccNUMA system - the UV1000 and UV100s - is an SGI NUMALink v5 interconnect. The UV1000 NUMALink is in a 2-D torus topology. All of the systems within Koronis are also interconnected with two 1-gigabit ethernet networks for administration, as well as a 10-gigabit network for higher-bandwidth traffic in and out of the constellation.

Overall Storage

All Koronis systems can access a complete data storage and management solution, including 760 TB of storage (136 TB usable high-speed FiberChannel providing scratch space for running jobs, and 524 TB usable SATA for longer-term storage) with an automated tape library for automated data migration and archival.

Home Directories

Currently, 16 TB of total file space has been allocated for user home directories. The default quota for home directories is 50 GB. Please email help@msi.umn.edu if additional space is required.

Scratch Spaces

Currently, there is 136 TB of file space allocated for scratch files in local /scratchfilesystems on the compute systems. There are no quotas on the /scratchfilesystems.

Koronis-only Project Spaces

Koronis adds a new Koronis-only project space that comes via multiple mount points all beginning with /cxfs. Currently, there are roughly 500 TB of file space allocated to the CXFS project spaces that can be used to hold large, persistent data needed only on Koronis. Files in CXFS project spaces are subject to automatic migration to tape depending on age of the file and recent use. Users will still see the files in their directories and can operate on them normally, but if a file has been migrated to tape, it will take extra time to pull the file from tape. 

Special Requirements for Use of Koronis

Resource allocation for this system is handled separately from other MSI HPC resources to comply with requirements by NIH, the funding agency. Specifically, principal investigators that are granted access to this resource must currently be NIH principal investigators, or must be in the process of trying to obtain funding from NIH. See the Koronis Access page for details.

Queues

Koronis Queue Policies

Default Queue

 

UV1000

-q uv1000

Long Queue

-q long

Restrictions

up to 66 cores

up to 24 hr run-time

up to 1140 cores

up to 24 hr run-time

up to 144 cores

up to 96 hr run-time

For all jobs, cores must be requested in increments of 6. The memory assigned to each job depends on the number of cores specified. For UV1000 and long queues, each core has 2,66 GB.  For the default queue, it is 3.6 gb.