How do I use Globus to transfer data at MSI?
The University of Minnesota participates in the Globus data transfer service. Globus is a service that allows for fast transfer of data between computer systems, using a convenient web interface. Globus data transfer speeds are typically significantly faster than data transfer speeds using other methods. Globus also contains data sharing capability, which allows users to share data with collaborators at UMN or other institutions.
MSI has a Globus endpoint on the primary MSI filesystem, with endpoint name UMN MSI Home. This endpoint allows MSI users to transfer data to and from MSI home, group shared, and global scratch directories, using Globus.
MSI also has a Globus endpoint accessing our Second Tier Storage system, with endpoint name UMN MSI Tier2. For more information about accessing Second Tier Storage using Globus, see this documentation.
This page describes how to set up and use Globus to transfer data to and from MSI systems. For advanced documentation, see the Globus How To webpage, the Globus FAQ webpage, and the Globus documentation site.
Transferring Data With Globus
In order to use Globus to transfer data between MSI and another system, follow these simple steps:
1. Navigate to https://www.globus.org/ and click the Login button in the upper right. When prompted for an organization, select the University of Minnesota. Complete the login using your UMN username and password.
2. From the Globus menu, select Transfer Files. This will lead to a webpage showing two panels which can be used to select Globus endpoints.
3. To select the MSI home directory endpoint, click in one of the Endpoint fields, and type UMN MSI Home and select it from the search results.
4. Use the transfer panels to select files or directories, and transfer them between any two Globus endpoints using the left and right arrows in the center of the interface. Following a completed transfer, the Globus system will send a notification email message. Active and pending transfers can be viewed or canceled under the Activity link. If you would like to set up a Globus endpoint on your local computer, see the relevant section below.
Warnings:
Globus automatically retries failed transfers over a period of 24 hours following an initial transfer request. If a transfer request contains an error such that it will never succeed, it is beneficial to delete the request using the activity panel, to prevent continued failed transfer attempts.
Users are advised against using one endpoint as both source and destination for a single transfer, as this slows the endpoint, and more effective techniques for moving data within MSI are available.
Transfer Restrictions
As with other transfer methods, users may only transfer data for which they have normal filesystem read access, and may only place new data in directories for which they have normal filesystem write access. Globus is bound by filesystem permissions in the usual way.
Additionally, a user may only perform Globus transfers involving data within their home directory and other locations such as their group shared directory ~/../shared and their group public directory ~/../public. Other directories such as /scratch.global are available as well. If a path is unavailable and you would like it to be made available, contact the helpdesk.
On MSI systems Globus will deny transfers involving home directory hidden, aka ‘dot’, files.
Setting Up a Globus Endpoint on a Local Machine
After logging in to https://www.globus.org/ select Endpoints, and click on the link for Add Globus Connect Personal Endpoint. This will lead to the page shown below, where it is possible to generate a Globus key, and to download the Globus Connect Personal client.
Follow the steps on the page to create a name for your local endpoint and generate a Globus key. Next, follow the directions to download the Globus Connect Personal client appropriate for your local operating system. Install the Globus Connect Personal client by running the installation program, and when prompted, paste in the Globus key which was generated for your personal endpoint.
While the Globus Connect Personal client is running on your local machine, your local machine contains a Globus endpoint. The Globus endpoint on your local machine may be selected by name from the Endpoint field in the Transfer Files panels. You may now use Globus to transfer files between your local machine and other Globus endpoints.
When the Globus Connect Personal client on your local machine is stopped, the local endpoint will stop.
Globus transfers involving a personal endpoint are likely to be slower than transfers between institutional endpoints.
Sharing Data With Globus, Guest Collections
The upgrade and switch from Globus 4 to Globus 5 means users will need to switch any existing shared endpoints that are still in use from the previous umnmsi#home to the UMN MSI Home endpoint.
Globus may be used to share data with other Globus users, at UMN or other institutions. Sharing is performed on a directory-by-directory basis, and users may choose which directories they would like to share with collaborators. MSI currently allows users to enable read-only Globus sharing of certain directories. Sharing is subject to the same transfer restrictions outlined in the Transfer Restrictions section above. To enable sharing of a directory, select the directory from within a transfer panel, and click on the "Share" option in the upper right of the transfer panel. The user will be brought to another interface and must select “Add a Guest Collection”.
The user must then consent for the Globus web app to manage collections on their behalf by clicking Continue. The user will then need to fill out a form with details for their new Guest Collection.
After creating this Guest Collection, the user may add permissions to share the collection with other Globus users using the “Add Permissions - Share With” button. This brings up an interface for specifying the share details.
The owner may manage or delete the share from the Endpoints link on the left of the Globus interface.
More specific information regarding Globus sharing is available in the Globus documentation: How to Share Data Using Globus.
Warning:
Globus shared directories remain shared until sharing is removed using the Globus interface. Any material within a Globus shared directory (including newly added material) is shared via Globus. For security, it is recommended that users disable sharing when their need for sharing has ended.
Globus write-sharing is disabled on MSI systems by default, and attempts to write to MSI directories via sharing will fail. If you believe you have a need for Globus write-sharing within your collaboration, please contact help@msi.umn.edu.