How do I use Second Tier Storage from the command line?

 


To access the MSI second-tier storage, each user must have a set of s3 credentials. The s3 credentials act like a username and password to control access to the storage behind the S3 gateway server called s3.msi.umn.edu. These credientals are automatically generated and stored in a configuration file called ".s3cfg" when you are logged into an MSI system and run the s3 command described below for the first time.

Command-line Access

The default command-line tool is s3cmd. This command line tool provides access to the full range of S3 features, including creating, deleting, and managing the permissions of buckets and files stored in buckets. A bucket is a container for all the files (AKA objects) that you store on the Second-tier storage systems. You can think of a bucket in the same way that you think of a top level file folder. The full manual for the s3cmd is available from its homepage and below are some examples of the most commonly used s3cmd features. Note that the external s3cmd documentation includes some features that only apply to Amazon's own S3 service.

The s3cmd has been installed on the MSI Linux lab nodes as well as the Itasca cluster. After you connect to Linux lab node, you can use the s3cmd command-line tool.

To create your own bucket (AKA folder) on s3.msi.umn.edu, you can use the "mb" command (make bucket):

  s3cmd mb s3://mynewbucket

Note: There is a shared namespace for everyone. Replace mynewbucket with a bucket name of your choosing. Do not use any underscores in the name.

To copy a file to S3:

  s3cmd put ~/localfile.txt s3://mynewbucket

To copy a file from S3 to your home directory:

  s3cmd get s3://mynewbucket/localfile.txt ~/

To recursively make a bucket and all its contents publically readable by everyone:

  s3cmd setacl -P -r s3://mynewbucket
For information about backing up intact nested directories, go to: http://s3tools.org/s3cmd-sync