How can my programs interface with Second Tier Storage?

Application Programming Interface (API) Support for S3 Access

For advanced tasks or in writing your own software, you may want to interact with Second Tier Storage directly through the S3 programming API. Libraries exist to do this from many programming languages.

  • boto is a useful Python library for accessing S3 gateways.  We have some simple examples at bottom of the page.
  • Amazon::S3 is a Perl library for working with and managing S3 buckets and keys.
  • AWS SDK for Java is developed directly by Amazon.

A full list of language libraries supported by Ceph S3 with many examples can be found in the Ceph documentation.

In every case, your program will need to use your s3 credentials to access the S3 service.

Example: Programmatic Access to S3 credentials

To assist with obtaining one's s3 credentials programmatically, MSI has created an "s3info" command that retrieves the S3 credentials associated to the user that runs it.  This command is available on login nodes and on Mesabi. A --machine-output option causes it to provide easily parseable output, so you can run it from a bash script or from the language of your choice. The below shell session shows its usage with and without the --machine-output option:

juser@login02 [~]% s3info

Access key: QPBEF6DD3AUY97V89DCH

Secret key: Zcl6xbjQSloQRp2tEdkUdZjPtOdJvN4hWgUH0j6h

 

Sharing address: juser@umn.edu

 

juser@login02 [~]% s3info keys --machine-output

QPBEF6DD3AUY97V89DCH Zcl6xbjQSloQRp2tEdkUdZjPtOdJvN4hWgUH0j6h

juser@login02 [~]% 

Example: Programmatic Access with Python

s3.msi.umn.edu can also be accessed programatically. boto is a useful Python library to interact with an S3 gateway. Below is an example of how to establish a connection and print a list of your buckets.

import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key
 
access_key = 'ABCDEFGHIJKLMNOP'
secret_key = 'kjsdlhfguihkejrg4ghih8a43934h2oih'
conn = S3Connection(access_key, secret_key, host='s3.msi.umn.edu')
 
rs = conn.get_all_buckets()
for b in rs:
    print b.name

 

Create a URL that allows anyone with the URL to download a file in the next hour:

import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key
 
access_key = 'ABCDEFGHIJKLMNOP'
secret_key = 'kjsdlhfguihkejrg4ghih8a43934h2oih'
conn = S3Connection(access_key, secret_key, host='s3.msi.umn.edu')
 
bucket = conn.get_bucket('mybucket')
filekey = bucket.get_key('bigfiletoshare.txt')
url_to_share = filekey.generate_url(3600, query_auth=True, force_http=True)
print url_to_share