Last Updated: November 6, 2018


The Globus AWS S3 storage connector can be used for access and sharing of data on AWS S3. The connector is available as an add-on subscription to organizations with a Globus Standard subscription - please contact us for pricing.

This document describes how to install and configure the AWS S3 Connector connector as well as create AWS S3 Storage Gateway. After the installation is complete, any authorized user can establish a connection to the AWS S3 buckets that they have access to by following the steps in this How To in order to create a collection

The installation must be done by a system administrator, and has the following distinct set of steps:

  • Installation of the packages needed for Globus Connect Server version 5 endpoint as well as creation of the endpoint itself.

  • Create a storage gateway on the endpoint configured to use the AWS S3 Connector.

Please contact us at support@globus.org if you have questions or need help with installation and use of the AWS S3 Connector.


Table of Contents

Endpoint Installation

The AWS S3 Connector requires a functional Globus Connect Server 5 endpoint in order to be used. Instructions for installing and configuring and endpoint using Globus Connect Server 5 can be found here. The rest of this document assumes that a functional Globus Connect Server 5 endpoint is being used when attempting to configure the AWS S3 Connector.

Creating a Storage Gateway using the AWS S3 Connector

A system administrator can create two different types of Storage Gateways with the AWS S3 Connector: authenticated and unauthenticated. The differences between these are

authenticated

In an authenticated AWS S3 Storage Gateway, each user provides an S3 Credential prior to creating a collection, and that credential is used to when accessing the data on the collection. The collection can only access AWS Buckets that have permissions based on the provided credential.

unauthenticated

In an unauthenticated AWS S3 Storage Gateway, each user does not provide a credential when creating a collection. The Collection can only access public AWS Buckets.

Creating an Authenticated AWS S3 Storage Gateway

To create an Authenticated AWS S3 Storage Gateway on an endpoint, use the `globus-connect-server-config storage-gateway create’ command . For example:

$ sudo globus-connect-server-config storage-gateway create \
    --connector S3 \
    --root "/" \
    --display-name "S3 Storage Gateway" \
    --domain example.edu \
    --s3-user-credential \
    --s3-endpoint https://s3.amazonaws.com \
    --s3-bucket data-bucket1 \
    --s3-bucket data-bucket2

Storage Gateway Created: 99d351bc-cdb4-4cee-be86-3bb01e4b1022

Note that the ID of the new storage gateway is given in the output.

This would create a storage gateway on the endpoint that:

  1. Causes new collections to be rooted in the bucket space of the s3.amazonws.com S3 endpoint.

  2. Allows Globus users with a Globus Account that includes an identity from the Identity Provider that controls the example.edu domain to create collections using an AWS credential.

  3. Uses the "S3" storage connector.

  4. Has a display name of "S3 Storage Gateway".

  5. Allows access to the data-bucket1 and data-bucket2 S3 buckets as /data-bucket1 and /data-bucket2 on the storage gateway.

Creating an Unauthenticated AWS S3 Storage Gateway

To create an Authenticated AWS S3 Storage Gateway on an endpoint, use the `globus-connect-server-config storage-gateway create’ command . For example:

$ sudo globus-connect-server-config storage-gateway create \
    --connector S3 \
    --root "/" \
    --display-name "S3 Public Storage Gateway" \
    --domain example.edu \
    --s3-unauthenticated \
    --s3-endpoint https://s3.amazonaws.com \
    --s3-bucket public-data-bucket1 \
    --s3-bucket public-data-bucket2

Storage Gateway Created: dddf4070-585e-41ed-ad3d-2d920e406e7f

Note that the ID of the new storage gateway is given in the output.

This would create a storage gateway on the endpoint that:

  1. Causes new collections to be rooted in the bucket space of the s3.amazonws.com S3 endpoint.

  2. Allows Globus users with a Globus Account that includes an identity from the Identity Provider that controls the example.edu domain to create collections that can access the public buckets.

  3. Uses the "S3" storage connector.

  4. Has a display name of "S3 Public Storage Gateway".

  5. Allows access to the public-data-bucket1 and public-data-bucket2 S3 buckets as /public-data-bucket1 and /public-data-bucket2 on the storage gateway.

The the globus-connect-server-config storage-gateway create command supports the following options for storage gateways configured to use the AWS S3 Connector, in addition to the common options supported for all storage connectors:

--s3-endpoint ENDPOINT

Full URL of S3 compatible storage location. May include a port. e.g. https://s3.amazonaws.com:443. A list of Amazon S3 endpoint hostnames by region can be found here: http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region This is a required parameter for this connector.

--s3-bucket BUCKET, -b BUCKET

S3 compatible bucket to make available in the root of the storage gateway. This option may be used 0 or more times. For an unauthenticated storage gateway, only buckets added by this option will be visible in the root of the storage gateway. If none are added, then the storage gateway root directory will appear empty. For an authenticated storage gateway, only these buckets will be made available in the root (if the S3 credential allows them). If none are added, then the storage gateway root directory will include all buckets for which the AWS S3 Credential has been granted access permissions.

--s3-user-credential | --s3-unauthenticated

Choose whether the AWS S3 Storage Gateway requires a user credential or allows access to public buckets. If the --s3-user-credential option is used when creating an AWS S3 Storage Gateway, then a user will be required to provide an S3 credential prior to creating a guest collection. If the --s3-unauthenticated option is used when creating an AWS S3 Storage Gateway, then the collection will only have access to public buckets.
If neither of these options are present when creating a storage gateway, the default is --s3-user-credential.

Creating a collection on a AWS S3 Storage Gateway

Once a AWS S3 Storage Gateway has been configured on the endpoint, permitted users can then create collections using the storage gateway. These collections allow permitted Globus users access to the AWS S3 service using their credential (for authenticated S3 Storage Gateways) or using unauthenticated public access. The process of creating a new collection using a storage gateway configured to use the AWS S3 Connector is found here. Please refer to the Globus Connect Server install document for the various options available in the tool to manage storage gateways.


© 2010- The University of Chicago Legal