Last Updated: October 27, 2017

The Globus AWS S3 Connector can be used for access and sharing of data on AWS S3. The connector is available as an add-on subscription to organizations with a Globus Standard subscription - please contact us for pricing.

This document describes the steps needed to install an endpoint, and the AWS S3 Connector needed to access the storage system. This installation should be done by a system administrator, and once completed users can use the endpoint to access AWS S3 via Globus to transfer, share and publish data on the system.

Prerequisites

A functional Globus Connect Server installation is required for installation and use of the AWS S3 Connector. The server can be hosted on any machine that can connect to the AWS S3 system. The Globus Connect Server Installation Guide provides detailed documentation on the steps for installing and configuring a server endpoint. Supported Linux Distributions

The AWS S3 Connector is available for the following Linux distributions:

  • RHEL 7

  • RHEL 6

  • CentOS 7

  • CentOS 6

  • Ubuntu 14.04

  • Ubuntu 16.04

  • Debian 7

  • Debian 8

  • SLES 12

Supported Globus Connect Server versions

The AWS S3 Connector should be used with the latest version of GCS.

Installation

Install the package globus-gridftp-server-s3 from the Globus repository.

For Red Hat-based systems:

% yum install globus-gridftp-server-s3

For Debian-based systems:

% apt-get install globus-gridftp-server-s3

For SLES 11-based systems:

% zypper install globus-gridftp-server-s3

Configuration

The AWS S3 Connector requires the following steps for configuration:

  • Configure the AWS S3 Connector

  • Create a gridmap to S3 credentials

  • Restart the GridFTP server

Configure the AWS S3 Connector

Create the file /etc/gridftp.d/gridftp-s3 containing these lines:

 threads 2
 load_dsi_module s3

Edit the file /etc/globus/globus-gridftp-server-s3.conf and set the host_name option to be the appropriate Amazon S3 endpoint hostname for the Amazon region that hosts the bucket(s) you want the AWS S3 Connector to access.

For example, to configure the AWS S3 Connector to use buckets in the US Standard Region:

host_name = s3.amazonaws.com

A list of Amazon S3 endpoint hostnames by region can be found here:

http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

Create a file for each user containing their AWS S3 credentials

Each user will need to have a special file created which specifies the S3 credentials associated with their local user account. The default configuration for the AWS S3 Connector looks in $HOME/.globus/s3 for a file mapping the current user’s ID to S3 access keys. Each user who will be using the AWS S3 Connector must create such a file with their credentials. This file can be created and populated by the user with the following commands:

 # mkdir -m 0700 -p ~/.globus
 # (umask 077; echo "$(id -un);$S3_ACCESS_KEY_ID;$S3_SECRET_ACCESS_KEY" \
    > ~/.globus/s3)

The S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY correspond to the Access Key ID and Secret Access Key for the user’s S3 credentials that have been granted access to the S3 buckets the user intends to access.

Debugging Tips

To enable a debugging log for the AWS S3 Connector, set the environment variable GLOBUS_S3_DEBUG "1023,/tmp/s3.log" to enable a highly verbose log of the connector. This can be easily done for a gridftp configuration by creating a file /etc/gridftp.d/s3-debug with the contents

 $GLOBUS_S3_DEBUG "1023,/tmp/s3.log"

Basic Endpoint Functionality Test

After completing the installation, you should do some basic transfer tests with your endpoint to ensure that it is working. We document a process for basic endpoint functionality testing here.

Known Limitations of the AWS S3 Connector

At the present time, rename operations are not supported for the AWS S3 Connector.

Note:For the legacy AWS S3 Connector, please see https://docs.globus.org/premium-storage-connectors/legacy/aws-s3-legacy.

© 2010- The University of Chicago Legal