How do I use Globus to transfer data to Second Tier Storage at MSI?

MSI has a Globus endpoint on the MSI Second Tier storage system, with endpoint name umnmsi#tier2. This endpoint allows MSI users to transfer data to and from MSI Second Tier storage. 

MSI Second Tier storage exists on Ceph shared filesystem, and Globus has released a Ceph connector, which enables Globus transfers to and from Ceph filesystems.

To use Globus to access MSI Second Tier storage, follow the same process for a Globus transfer outlined in our Globus documentation, but select endpoint umnmsi#tier2 from the Transfer Panel. After selecting umnmsi#tier2, you will be asked to log in using your MSI username and password (which may be the same as your UMN username and password, if your MSI and UMN accounts are unified).

After logging in to the umnmsi#tier2 endpoint, it is possible to view and transfer files and directories as with any other Globus endpoint. To move data between MSI Primary and Secondary storage, transfers may be performed between the umnmsi#home and umnmsi#tier2 endpoints.  For transfers of long duration, it can be helpful to request a longer credential lifetime at login, by clicking on the Advanced link below the password field, and specifying a longer credential lifetime (the units are in hours, with a maximum value of 168).

The Globus endpoint software supporting transfers to Ceph filesystems has appeared stable, though in its current implementation the umnmsi#tier2 endpoint has generally delivered lower transfer performance than MSI's standard Globus endpoint. It has been observed that higher transfer rates are obtained for fewer large files, than for many small files. Users may wish to consider combining small files into composite archive files for faster transfer, though this is up to the user.

As with other Globus endpoints, it is possible to enable read-only sharing of a file or directory (Ceph bucket) via the Globus sharing mechanism. For more information on Globus sharing, and other Globus endpoint features, see the primary Globus documentation.


Warnings:

General warnings regarding Globus transfers apply to the Second Tier storage endpoint. Please consult the primary Globus documentation to understand the warnings and limitations.

Within Second Tier storage, data must exist within at least one directory (Ceph bucket). Data transfers to umnmsi#tier2 should attempt to place data within at least one directory, or the transfers will fail.

The Globus sharing mechanism is completely independent of Ceph file sharing mechanisms. Changing the shared status of data via the Globus interface has no effect on Ceph sharing permissions, and vice versa. As with other endpoints, Globus shared directories remain shared until sharing is removed using the Globus interface. Any material within a Globus shared directory (including newly added material) is shared via Globus. For security, it is recommended that users disable sharing when their need for sharing has ended.