MSI Beta is a way for MSI’s staff and users to explore new technologies without hindering ongoing research. It includes a number of new and experimental computing technologies.
An article about MSI Beta appeared in the Office of the Vice President for Research Inquiry blog.
Current MSI Beta Projects
Users interested in a specific MSI Beta project should contact firstname.lastname@example.org.
Transferring Files to Second Tier Storage Using Globus
MSI has a Globus endpoint on the MSI Second Tier storage system. This endpoint allows MSI users to transfer data to and from MSI Second Tier storage. MSI's Second Tier storage system is currently an implementation of the Ceph distributed filesystem. Globus has recently released a Beta version of the Globus endpoint software that supports Ceph filesystems. As an early adopter, MSI has created an endpoint for accessing Second Tier storage, using the Globus endpoint Beta software that supports Ceph transfers.
Jupyter Notebooks Service
MSI has opened notebooks.msi.umn.edu to all users as a new MSI Beta service. This interactive computing environment requires only a web browser, and enables data analysis and visualization on our HPC resources in a shareable, reproducible notebook format. Notebooks currently supports Python 2.7, Python 3.4, and R 3.2. MSI Python tutorial materials provide a useful overview. Please note that off-campus users must connect through the University VPN.
Research Cloud for Protected Data (a.k.a. Stratus)
Catering to the needs of modern users with non-traditional HPC workflows, MSI is building a local research compute cloud environment called Stratus. Stratus will enable users to operate within their own self-service virtual machines on the spectrum between a low-level bare-metal HPC and a fully virtualized, shared-tenancy mode. Stratus is powered by the Mitaka version of the OpenStack cloud platform, and is backed by Ceph storage. The Stratus compute hardware currently consists of 20x HPE Proliant XL230a compute nodes, each with 2x Intel E5-2680v4 (14-core) CPUs, 256 GB of RAM, and 10 GigE networking to the outside world. An array of HPE Apollo 4200 servers, connected to compute nodes via redundant 40 GigE switches, complements the cluster with 200 TB of Ceph block device storage. The initial interation of Stratus will feature three functionalities not available on other MSI systems:
- Users will be able to create virtual machines and remote desktops on demand.
- Long-running jobs (i.e., > 30 days) will be enabled by leveraging live-migration during maintenance windows.
- Virtual machines will support Docker containers for self-service software management (e.g., to deploy local versions of Galaxy or JupyterHub).
The Hadoop Cluster is a storage solution solving the issues around accessing and working with very large datasets. MSI staff are also exploring its ability to run frameworks for machine learning.
Hadoop HDFS and MapReduce are installed on the cluster. HDFS is a distributed filesystem. Each node in the Hadoop cluster holds some of its data in HDFS.