How do I use Globus to transfer data to Second Tier Storage at MSI?

As a Beta service, MSI has a Globus endpoint on the MSI Second Tier storage system, with endpoint name umnmsi#tier2. This endpoint allows MSI users to transfer data to and from MSI Second Tier storage.

MSI's Second Tier storage system is currently an implementation of the Ceph distributed filesystem. Globus has recently released a Beta version of the Globus endpoint software that supports Ceph filesystems. As an early adopter, MSI has created an endpoint for accessing Second Tier storage, using the Globus endpoint Beta software that supports Ceph transfers.

To use Globus to access MSI Second Tier storage, follow the same process for a Globus transfer outlined in our Globus documentation, but select endpoint umnmsi#tier2 from the Transfer Panel. After selecting umnmsi#tier2, you will be asked to log in using your MSI username and password (which may be the same as your UMN username and password, if your MSI and UMN accounts are unified).

After logging in to the umnmsi#tier2 endpoint, it is possible to view and transfer files and directories as with any other Globus endpoint. To move data between MSI Primary and Secondary storage, transfers may be performed between the umnmsi#home and umnmsi#tier2 endpoints.  For transfers of long duration, it can be helpful to request a longer credential lifetime at login, by clicking on the Advanced link below the password field, and specifying a longer credential lifetime (the units are in hours, with a maximum value of 168).

The Globus endpoint software supporting transfers to Ceph filesystems has appeared stable, though in its current implementation the umnmsi#tier2 endpoint has generally delivered lower transfer performance than MSI's standard Globus endpoint. It has been observed that higher transfer rates are obtained for fewer large files, than for many small files. Users may wish to consider combining small files into composite archive files for faster transfer, though this is up to the user. We will be working to improve the endpoint performance, and working with Globus as they improve the Globus Ceph endpoint software.

As with other Globus endpoints, it is possible to enable read-only sharing of a file or directory (Ceph bucket) via the Globus sharing mechanism. For more information on Globus sharing, and other Globus endpoint features, see the primary Globus documentation.


Warnings:

General warnings regarding Globus transfers apply to the Second Tier storage endpoint. Please consult the primary Globus documentation to understand the warnings and limitations.

Within Second Tier storage, data must exist within at least one directory (Ceph bucket). Data transfers to umnmsi#tier2 should attempt to place data within at least one directory, or the transfers will fail.

The Globus sharing mechanism is completely independent of Ceph file sharing mechanisms. Changing the shared status of data via the Globus interface has no effect on Ceph sharing permissions, and vice versa. As with other endpoints, Globus shared directories remain shared until sharing is removed using the Globus interface. Any material within a Globus shared directory (including newly added material) is shared via Globus. For security, it is recommended that users disable sharing when their need for sharing has ended.

As an MSI Beta service, the umnmsi#tier2 endpoint may experience more frequent maintenance and adjustment than MSI primary services, as we work to improve the service performance and stability.