S3
The Globus Connect Server S3 Connector provides access to S3-compatible storage via Globus Connect Server.
This document describes the data types and API routes provided by this connector.
S3 Connector Virtual Filesystem
The S3 connector provides a distributed object store, where each data object is accessed based on a bucket name and an object name.
The S3 connector attempts to make this look like a regular filesystem,
by treating the bucket name as the name of a directory in the root of
the storage gateway’s file system. For example, if a user has access
to buckets bucket1 and bucket2, then those buckets would show up as
directories when listing /.
The S3 connector also treats the / character as a delimiter in the S3
API so that it can present something that looks like like
subdirectories. For example, the object object1 in bucket1 would
appear as /bucket1/object1 to the S3 connector, and the object
object2/object3 in bucket2 would appear as a file called object3
in the directory /bucket2/object2.
Authenticated and Anonymous Access
Each S3 Storage Gateway can be configured to perform either authenticated or unauthenticated access to S3 data. When creating an S3 Storage Gateway, you must choose which type of access to require.
- authenticated
-
Globus users must register an S3 Credential with Globus Connect Server in order to access data on its collections. The credential must be associated with a policy that allows the IAM permissions used by the {s3_connector_type}.
- unauthenticated
-
Globus users can only access public AWS Buckets.
S3 Storage Gateway Policies
The s3_user_credential_required, s3_buckets, and s3_endpoint properties control access to an Amazon S3 or compatible resource.
Endpoint
The s3_endpoint property is used by Globus Connect Server to contact the S3 API to access data on this storage gateway. This may be an Amazon S3 URL, a regional Amazon S3 URL, or the URL endpoint of another compatible storage system.
Access Mode
If the s3_user_credential_required property is enabled, then each user accessing collections on this storage gateway must register an S3 key_id and secret_key with the storage gateway.
Bucket Restrictions
The s3_buckets property value is the list of bucket names which are allowed access by this storage gateway.
If no buckets are configured, then any buckets accessible using the user’s registered S3 key_id and secret_key may be accessed by collections on this storage gateway. If any are configured, then they act as restrictions to which buckets are visible and accessible on collections on this storage gateway.
S3 User Credentials
S3 user credentials can either be a placeholder for unauthenticated bucket access, or can contain an S3 key_id and secret access_key value. The type of credential needed for any particular collection is based on the configuration of the storage gateway that it uses.
Storage Gateway Allowing Anonymous Bucket Access
In order to access a collection using the S3 connector configured for anonymous bucket access, users don’t have to do anything to create a credential. It will be automatically provisioned on first use. Users may disable and reenable a credential by changing the value of the invalid property.
Storage Gateway Requiring a Key
In order to access a collection using the S3 connector configured to
use a S3 key, users must supply key information to the GCS Manager.
This can be most easily done via the Globus web application. If a user
does not supply key information, the credential will have its
invalid property set to true and must be updated by the user
to include key data.
For AWS, the key provided by the user must have permissions to access buckets and objects as described in the Globus AWS documentation. Consult your storage system documentation for key requirements for other S3-compatible systems supported by Globus.
Creating a Credential
When creating a credential, the caller MUST include values for the DATA_TYPE and storage_gateway_id properties. If any of these are missing, the credential can not be created. If the identity_id value is missing, the identity with the lowest-valued identity id value in the caller’s identity set which meets the storage gateway policies is used.
In order for a credential to be valid, it must have values for the
DATA_TYPE, s3_key_id and s3_secret_key in the policies
document. If any of these are missing, the credential is created,
but its invalid property is set to true and it cannot be used
until the user has updated the credential.