Agate, the Minnesota Supercomputing Institute’s newest supercomputer, has been selected. It is a Hewlett Packard Enterprise (HPE) cluster. Delivery is expected to begin in mid-October 2021, with the cluster available to Minnesota researchers by the end of 2021.

The new cluster’s name, Agate, was selected by MSI staff from a pool of names submitted by 133 people. The Lake Superior agate is Minnesota’s State Gemstone.

The Agate cluster will join existing systems in MSI's data center at the University of Minnesota's Walter Library: the HPE/AMD Mangi cluster (2019) and HPE/Intel Mesabi cluster (2015).

See the OVPR's Inquiry blog post about Agate.

See Nvidia's story about high-performance computing systems in academia, including Agate.

System Specifications and Timeline

System Specifications

The theoretical performance of the new HPE-built cluster is about 7 quadrillion floating point operations per second (7 petaflops), of which 2 petaflops are provided by the cluster's general-purpose processors, and 5 petaflops by the GPU subsystem. "We worked closely with HPE to develop a highly efficient and flexible system …" said Graham Allan, Associate Director for Operations at MSI. "The Agate cluster will provide over 7 times the performance of the existing Mesabi cluster, while consuming the same amount of power and occupying less floor space."

The Agate cluster will contain a total of 416 nodes, including HPE Apollo 2000 XL225 Gen10+ CPU-only systems, and Apollo 6500 Gen10+ GPU servers, all using Direct Liquid Cooling, coupled with a set of large-memory HPE DL385 Gen10+ nodes. In total the cluster will contain 770 AMD Milan 7763 64-core processors, and 264 NVIDIA A100 Tensor Core GPUs, all connected with Mellanox HDR-100 InfiniBand network.

One unique feature of the new cluster is provision of an additional 80 Nvidia A40 single-precision GPUs, to further support interactive applications with server-side GPU rendering, and interactive machine-learning workloads via command line and JupyterHub.

"The new system is designed to position our faculty at the leading-edge of high performance computing capabilities and to minimize obstacles to accelerating data intensive research," said Jim Wilgenbusch, Director of Research Computing

Between now and delivery of the new cluster, MSI will also be upgrading the capacity and performance of its cluster storage systems, starting with updating the existing Panasas parallel storage system to next-generation Panasas Ultra hardware. This will deliver approximately 10 PB of new capacity, representing a net increase of 5 PB, in order to better support a variety of data-intensive research initiatives. Alongside the Panasas storage upgrades, the capacity of MSI's ceph-based tier 2 storage, built using HPE hardware, will also be increased by 5 PB.


We anticipate delivery of the Agate system to begin in October 2021, with initial production capability by the end of the year.

Researcher and Collaborator Reactions

Brian Peterson, Chief Operating Officer, Panasas: It's exciting to be a part of the research being fueled by Agate. We look forward to continuing our partnership with the university for years to come.

John Josephakis, Global Vice President of Sales and Business Development for HPC/Supercomputing, NVIDIA: NVIDIA enables the most performance-intensive supercomputers in the world and our collaboration with the Minnesota Supercomputing Institute continues to drive pioneering scientific research forward. The NVIDIA accelerated platform pushes the limits of advanced research computing, supporting the Institute in their mission to provide scientists and researchers with unrivaled, next-generation resources to tackle the hardest problems in the world.

GEMS (Genetic, Environmental, Management, and Socioeconomic) leadership: We are very excited to have access to the successor to Mesabi at MSI. GEMS is a major nexus of geospatial data and analytics for use in de-risking the agri-food sectors. The spatio-temporal analysis of the risks associated with crop pest occurrence and dispersal across agricultural landscapes is incredibly compute intensive. The new compute capabilities at MSI will enable us to simulate many more plant pests, different geographies, and different predicted weather scenarios, potentially in near-real time.

Assistant Professor Steven Friedenberg (College of Veterinary Medicine), MSI PI: We perform a significant amount of whole genome sequencing of companion animals. We also study genetic variants and rearrangements on many of the key genes in the immune system. Because our datasets are so large and technically complex, we face challenges on a daily basis in terms of how fast we can process and analyze our data, and in terms of how much space we have to store our analyses. We are extremely excited about the opportunity to take advantage of MSI’s new system, which will dramatically reduce our run times, expand our ability to study different diseases in more animals, and essentially make discoveries at a much faster pace.

Associate Professor Suzanne McGaugh (College of Biological Sciences), MSI PI: The expanded capabilities of the Minnesota Supercomputing Institute will give my lab a competitive advantage in genomic analyses. High-performance storage space and computing power, especially RAM, are absolutely critical to our analyses and we will work much more efficiently with the additions provided by this new resource. 

Associate Professor Peter Morrell (College of Food, Agricultural, and Natural Resource Sciences), MSI PI: We are identifying structural variants important to climate adaptation in barley. We collect very long DNA sequence reads. The raw data collected is a measure of electrical current (known as squiggles) as DNA strands are drawn through a protein nanopore. The squiggles are converted into DNA nucleotide sequence and to identify modifications to DNA, but the process is very computationally intensive. Graphical processing units (GPUs) installed on the Mangi compute cluster dramatically accelerate base calling. The updates to current supercomputing resources at MSI will permit additional computationally intensive analysis of nanopore sequence and will allow us to fully exploit the information latent in the data.  

Associate Professor Sapna Sarupria (College of Science and Engineering), MSI PI: Using GPUs for molecular dynamics (MD) simulations has become standard practice now. GPUs have accelerated MD simulations and continue to enable us to study systems that were previously beyond the grasp of computational molecular science. We now routinely study large proteins to design better technology for using protein-based diagnostics. The access to the GPU-based cluster at MSI will further enable such research. For example, in a recent project, our goal is to simulate virus particles to design better stabilization technologies for them so that we can avoid the requirement of extremely low temperatures to store vaccines. This can be game-changing for the accessibility of vaccines. Such simulations cannot be ima gined without access to several GPUs. I am thrilled that MSI is continuing to grow its computational power and investing in GPU-based clusters.

Professor J. Ilja Siepmann (College of Science and Engineering), MSI PI: The Siepmann group develops computational tools and applies them to aid our understanding of complex chemical systems and to discover materials for energy-efficient chemical separations. [At present, chemical separations account for more than 10% of energy usage in the USA. Thus, finding energy-efficient separation processes play a pivotal role toward a sustainable future.] Due to the large sizes of 3D data and neural network structure, the increased amount of video memory on NVIDIA Ampere GPUs will accelerate the training of large machine learning models for 3D energy grids by 2 to 4 folds. The Multi-Instance GPU technology provided by the Ampere architecture can also improve the GPU utilization of smaller machine learning models in nanoporous material discovery, such as predicting the continuous state space surface of multicomponent adsorption.