How can my programs interface with Second Tier Storage?
Application Programming Interface (API) Support for S3 Access
For advanced tasks or in writing your own software, you may want to interact with Second Tier Storage directly through the S3 programming API. Libraries exist to do this from many programming languages.
- boto is a useful Python library for accessing S3 gateways. We have some simple examples at bottom of the page.
- Amazon::S3 is a Perl library for working with and managing S3 buckets and keys.
- AWS SDK for Java is developed directly by Amazon.
A full list of language libraries supported by Ceph S3 with many examples can be found in the Ceph documentation.
In every case, your program will need to use your s3 credentials to access the S3 service.
Example: Programmatic Access to S3 credentials
To assist with obtaining one's s3 credentials programmatically, MSI has created an "s3info" command that retrieves the S3 credentials associated to the user that runs it. This command is available on login nodes and on Mesabi. A --machine-output option causes it to provide easily parseable output, so you can run it from a bash script or from the language of your choice. The below shell session shows its usage with and without the --machine-output option:
juser@login02 [~]% s3info
Access key: QPBEF6DD3AUY97V89DCH
Secret key: Zcl6xbjQSloQRp2tEdkUdZjPtOdJvN4hWgUH0j6h
Sharing address: juser@umn.edu
juser@login02 [~]% s3info keys --machine-output
QPBEF6DD3AUY97V89DCH Zcl6xbjQSloQRp2tEdkUdZjPtOdJvN4hWgUH0j6h
juser@login02 [~]%
Example: Programmatic Access with Python
s3.msi.umn.edu can also be accessed programatically. boto is a useful Python library to interact with an S3 gateway. Below is an example of how to establish a connection and print a list of your buckets.
import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key
access_key = 'ABCDEFGHIJKLMNOP'
secret_key = 'kjsdlhfguihkejrg4ghih8a43934h2oih'
conn = S3Connection(access_key, secret_key, host='s3.msi.umn.edu')
rs = conn.get_all_buckets()
for b in rs:
print b.name
Create a URL that allows anyone with the URL to download a file in the next hour:
import boto
from boto.s3.connection import S3Connection
from boto.s3.key import Key
access_key = 'ABCDEFGHIJKLMNOP'
secret_key = 'kjsdlhfguihkejrg4ghih8a43934h2oih'
conn = S3Connection(access_key, secret_key, host='s3.msi.umn.edu')
bucket = conn.get_bucket('mybucket')
filekey = bucket.get_key('bigfiletoshare.txt')
url_to_share = filekey.generate_url(3600, query_auth=True, force_http=True)
print url_to_share