How do I use Globus to transfer data at MSI?

The University of Minnesota participates in the Globus data transfer service. Globus is a service that allows for fast transfer of data between computer systems, using a convenient web interface. Globus data transfer speeds are typically significantly faster than data transfer speeds using other methods. Globus also contains data sharing capability, which allows users to share data with collaborators at UMN or other institutions.

MSI has a Globus endpoint on the primary MSI filesystem, with endpoint name umnmsi#home. This endpoint allows MSI users to transfer data to and from MSI home, group shared, and global scratch directories, using Globus.

MSI also has a Globus endpoint accessing our Second Tier Storage system, with endpoint name umnmsi#tier2.  For more information about accessing Second Tier Storage using Globus, see this documentation.

This page describes how to set up and use Globus to transfer data to and from MSI systems. For advanced documentation, see the Globus How To webpage, the Globus FAQ webpage, and the Globus documentation site.

Transferring Data With Globus

In order to use Globus to transfer data between MSI and another system, follow these simple steps:

  1. Navigate to https://www.globus.org/ and click the Login button in the upper right. When prompted for an organization, select the University of Minnesota. Complete the login using your UMN username and password.
     
  2. From the Globus menu, select Transfer Files. This will lead to a webpage showing two panels which can be used to select Globus endpoints.

     
  3. To select the MSI home directory endpoint, click in one of the Endpoint fields, and type umnmsi#home
    After selecting this endpoint you will be asked to log in using your MSI username and password (which may be the same as your UMN username and password, if your MSI and UMN accounts are unified).

    By default your MSI endpoint credentials will stay active for 24 hours after login. When credentials expire, any active transfer will pause until new credentials are obtained. Longer credentials may be requested during endpoint login by clicking the "advanced" link below the password field, and entering a requested credential lifetime. The credential lifetime field is in units of hours, and accepts values up to a maximum of 168 (168 hours is equivalent to 7 days).

    Note: If you wish to manually deactivate credentials on any endpoint, you may follow the Endpoints link in the Globus interface, select the relevant endpoint (eg. umnmsi#home), and click "Deactivate Credentials".
     
  4. Use the transfer panels to select files or directories, and transfer them between any two Globus endpoints using the left and right arrows in the center of the interface. Following a completed transfer, the Globus system will send a notification email message. Active and pending transfers can be viewed or canceled under the Activity link. If you would like to set up a Globus endpoint on your local computer, see the relevant section below.

Warnings: 

By default Globus transfers are not encrypted. If you wish to encrypt your transfer, check the "encrypt transfer" box below the transfer panel. Enabling encryption will reduce the effective transfer speed to some degree.

Globus automatically retries failed transfers over a period of 24 hours following an initial transfer request. If a transfer request contains an error such that it will never succeed, it is beneficial to delete the request using the activity panel, to prevent continued failed transfer attempts.

Users are advised against using one endpoint as both source and destination for a single transfer, as this slows the endpoint, and more effective techniques for moving data within MSI are available.


Transfer Restrictions

As with other transfer methods, users may only transfer data for which they have normal filesystem read access, and may only place new data in directories for which they have normal filesystem write access. Globus is bound by filesystem permissions in the usual way.

Additionally, a user may only perform Globus transfers involving data within subdirectories of their home directory ~, their group shared directory ~/../shared, their group public directory ~/../public, or the global scratch directory /panfs/roc/scratch.  It is recommended to use paths relative to the ~ symbol for transfers to home and group directories; Globus may deny transfers to some other equivalent paths.

On MSI systems Globus will deny most transfers involving home directory hidden files.

Setting Up a Globus Endpoint on a Local Machine

After logging in to https://www.globus.org/ select Endpoints, and click on the link for Add Globus Connect Personal Endpoint. This will lead to the page shown below, where it is possible to generate a Globus key, and to download the Globus Connect Personal client.

Follow the steps on the page to create a name for your local endpoint and generate a Globus key. Next, follow the directions to download the Globus Connect Personal client appropriate for your local operating system. Install the Globus Connect Personal client by running the installation program, and when prompted, paste in the Globus key which was generated for your personal endpoint.

While the Globus Connect Personal client is running on your local machine, your local machine contains a Globus endpoint. The Globus endpoint on your local machine may be selected by name from the Endpoint field in the Transfer Files panels. You may now use Globus to transfer files between your local machine and other Globus endpoints.

When the Globus Connect Personal client on your local machine is stopped, the local endpoint will stop.

Globus transfers involving a personal endpoint are likely to be slower than transfers between institutional endpoints.

Sharing Data With Globus

Globus may be used to share data with other Globus users, at UMN or other institutions. Sharing is performed on a directory-by-directory basis, and users may choose which directories they would like to share with collaborators. MSI currently allows users to enable read-only Globus sharing of certain directories. Sharing is subject to the same transfer restrictions outlined in the Transfer Restrictions section above. To enable sharing of a directory, select the directory from within a transfer panel, and click on the "share" option in the upper right of the transfer panel.

The user will be prompted to choose a Globus shared endpoint name for the shared directory. The shared directory will be treated as its own "shared endpoint", and the owner may manage or delete the share from the Endpoints link in the Globus interface. During creation of the shared directory, the user will be asked if they wish to share the directory with a particular Globus user, Globus group, or with a wider collection of users.
More specific information regarding Globus sharing is available in the Globus documentation: How to Share Data Using Globus


Warning:

Globus shared directories remain shared until sharing is removed using the Globus interface. Any material within a Globus shared directory (including newly added material) is shared via Globus. For security, it is recommended that users disable sharing when their need for sharing has ended.


Globus write-sharing is disabled on MSI systems by default, and attempts to write to MSI directories via sharing will fail. If you believe you have a need for Globus write-sharing within your collaboration, please contact help@msi.umn.edu.