Globus Connect Server Storage Gateway Create S3
Synopsis
globus-connect-server storage-gateway create s3 [OPTIONS]…
Description
The globus-connect-server storage-gateway create command creates a new storage gateway. When creating a storage gateway, provide the policies to access a storage system via Globus Connect Server collections.
The S3 connector provides access to Amazon S3 and other compatible file stores.
S3 Connector Virtual Filesystem
The S3 connector provides a distributed object store, where each data object is accessed based on a bucket name and an object name.
The S3 connector attempts to make this look like a regular filesystem,
by treating the bucket name as the name of a directory in the root of
the storage gateway’s file system. For example, if a user has access
to buckets bucket1
and bucket2
, then those buckets would show up as
directories when listing /
.
By default, the buckets listed in the root directory are generated by
listing the buckets owned by the registered default key, as well as any
buckets included as a path prefix of additional keys.
However, when the storage gateway --bucket
policy option is used, only
those buckets are shown. If bucket listing is not possible with the default
key, additional keys have not been registered, and the storage gateway
--bucket
policy is not set, root listing will not be possible. In that case users may attempt to access any bucket path directly.
The S3 connector treats the /
character as a delimiter in the S3
API so that it can present something that looks like
subdirectories. For example, the object object1
in bucket1
would
appear as /bucket1/object1
to the S3 connector, and the object
object2/object3
in bucket2
would appear as a file called object3
in the directory /bucket2/object2
.
Authentication Policies
There are three command-line options that control which user identities are allowed access to the data on a storage gateway: --domain, --authentication-timeout-mins, and --high-assurance.
The value of the --domain command-line option restricts access to users who have an identity in the given domain. This may be configured to be multiple values to allow authentication by multiple identity providers. If more than one domain is allowed, the storage gateway needs to have an identity mapping method configured to decide how to process names from the different identity namespaces. See Identity Mapping Policies for more information.
The value of the --authentication-timeout-mins command-line option defines the timeout (in minutes) after which a user will need to re-authenticate in order to access mapped collections on non high assurance storage gateways or for any data access on high assurance storage gateways. If this is not supplied, the default value of this timeout is 11 days.
The value of the --high-assurance command-line option defines whether the storage gateway manages high assurance data. If it is set, then the authentication timeout is enforced on per application sessions. This option can only be set when the storage gateway is created and is immutable.
Identity Mapping Policies
Globus Connect Server v5.4 supports a flexible system for mapping user identity information in Globus to the local account needed to access data on a variety of storage systems. This includes a default mapping for cases where there is only one allowed domain, as well as pattern-based mappings and callouts to external programs for custom mapping algorithms.
Default Identity to Username Mapping
When Globus Connect Server maps an identity to an account, it retains the entire username by
default, so the username user@example.org
is mapped to the account
user@example.org
.
Custom Identity to Username Mapping
The --identity-mapping command-line option configures a storage gateway to use either an expression based identity mapping or an external identity mapping program. See the Identity Mapping Guide for more information.
The --identity-mapping command-line option can be passed on the command-line with a few different types of data as its arguments:
-
--identity-mapping external:
CMD
-
When mapping a identity to a username, Globus Connect Server invokes the command-line program
CMD
to map the identity. The value of theCMD
string will be parsed as a shell command-line, so arguments may be included if quoted. A full description of the input, output, and arguments to the program are included in Identity Mapping Guide. -
--identity-mapping file:
JSON_FILE
-
--identity-mapping
JSON
-
The
JSON_FILE
argument is a path to a file which contains a JSON document containing the mapping configuration, as described in the Identity Mapping Guide. TheJSON
argument is the json document itself.
User Policies
The --user-allow and --user-deny command-line options control which users may access data on a storage gateway. These operate on the result of the identity mapping, a user name that is in the namespace of storage gateway. This may be a user name, id, or email address based on the storage gateway requirements.
A username is allowed or denied access depending on whether the --user-allow and --user-deny command-line option are set on a storage gateway, and whether the username is present in one or both of those policies. In general, if a username is in the value of --user-deny it is always denied, and if a --user-allow policy is provided the username must be in the policy value in order to be allowed access.
The full set of effects of these policies are contained in the following table:
--user-allow | --user-deny | result |
---|---|---|
member |
- |
Allowed |
member |
not a member |
Allowed |
- |
- |
Allowed |
- |
not a member |
Allowed |
- |
member |
DENIED |
not a member |
- |
DENIED |
not a member |
not a member |
DENIED |
not a member |
member |
DENIED |
member |
member |
DENIED |
S3 Storage Gateway Policies
The --s3-user-credential, --s3-requester-pays, --s3-unauthenticated, --bucket, and --s3-endpoint command-line options control access to an Amazon S3 or compatible resource.
Endpoint
The --s3-endpoint command-line option is used by Globus Connect Server to contact the S3 API to access data on this storage gateway. This may be an Amazon S3 URL, a regional Amazon S3 URL, or the URL endpoint of another compatible storage system.
IPv6 Configuration
This connector supports transferring files over IPv6 networks. This requires
the s3_endpoint
value to be one of the Amazon dualstack endpoints. The list
of regions and their dualstack endpoints is available from the
S3 documentation.
Access Mode
The --s3-user-credential and --s3-unauthenticated command-line options are mutually exclusive.
If the --s3-user-credential command-line option is enabled, then each user accessing collections on this storage gateway must register an S3 access key id and secret key with the storage gateway.
The --admin-managed-credential command-line option can also be set to allow admins the ability to register an S3 access key id and secret key for users.
If the --s3-unauthenticated command-line option is enabled, then all accesses to collections on this storage gateway will be done using unauthenticated access. In this case, the root of the S3 Connector Virtual Filesystem will only be able to list buckets that are explicitly made visible by using the --bucket command-line option.
Requester Pays (new in 5.4.59)
If the --s3-requester-pays command-line option is enabled, then requests made to S3 will include the request-payer parameter which allows the costs associated with those requests to be charged to the AWS account making the request. See the AWS Requester Pays documentation for more information.
The --s3-requester-pays command-line option requires the --s3-user-credential command-line option.
If the --s3-requester-pays command-line option is enabled, S3 operations from mapped and guest collection accesses will be charged to the AWS account associated with the registered user credential. Globus users must acknowledge this behavior when creating the user credential.
Bucket Restrictions
The --bucket command-line option argument is the name of a bucket which is allowed access by this storage gateway.
If no buckets are configured, then any buckets accessible using the user’s registered S3 key_id and secret_key may be accessed by collections on this storage gateway. If any are configured, then they act as restrictions to which buckets are visible and accessible on collections on this storage gateway.
Data Access Policies
The --restrict-paths command-line option controls access to subtrees of the data provided by the storage gateway. This is configured using the PathRestrictions document type.
Path restrictions provide a framework for administrators to constrain data access on the storage gateway. Restrictions can be set at the folder level. They may allow read, write, or deny access to data. These are absolute paths from the root of the storage gateway virtual file system.
Network Use Policies
The command line option --network-use alters the network performance and scalability parameters used by collections created using this storage gateway, overriding the default behavior set on the endpoint. This feature can only be used with a Globus subscription.
If the network use is set to custom
, then all of the
--preferred-parallelism, --max-parallelism, --preferred-concurrency, and
--max-concurrency options must also be set. See the
network use section
of the installation guide for a description of what these values mean.
If the network use is set to null
, then the default behavior is restored, and
collections use the settings on the endpoint.
OPTIONS
- -h, --help
-
Show help message and exit.
- --version
-
Show the version and exit.
- -F, --format "text"|"json"
-
Output format for this command. If the format is json, then the resulting role document is displayed.
- --use-explicit-host IP_ADDRESS (new in 5.4.23)
-
IP address of the GCS node to use for this request. If not specified, any available GCS node in the endpoint will be used.
- --user-deny username
- --user-deny file:PATH (new in 5.4.79)
-
Connector-specific username for a user denied access to this Storage Gateway. Give this option multiple times to deny multiple users. Set a value of "" to clear this value. If the parameter value begins with
file:
, read the input file path and parse as one or more lines of a whitespace delimited list of usernames to deny access to this storage gateway. - --user-allow username
- --user-allow file:PATH (new in 5.4.79)
-
Connector-specific username for a user allowed access to this Storage Gateway. Give this option multiple times to allow multiple users. Set a value of "" to clear this value. If the parameter value begins with
file:
, read the input file path and parse as one or more lines of a whitespace delimited list of usernames to allow access to this storage gateway. -
--identity-mapping external:
CMD
-
--identity-mapping file:
JSON_FILE
|JSON
-
Identity Mapping configuration for use in this Storage Gateway. You can use JSON input to specify a complete mapping document, or, if you want to use an external command for mapping, use external:command --arguments. Give this option multiple times to set multiple mappings in order of precedence. Set a value of null to clear this value.
-
--restrict-paths
JSON
| file:JSON_FILE
-
Path restrictions for accessing data on collections created using this storage gateway.
- --domain DOMAIN
-
Allowed domain. Give this option multiple times to allow multiple domains. Users creating credentials or collections on this storage gateway must have an identity in one of these domains.
- --authentication-timeout-mins INT
-
Timeout (in minutes) during which a user is required to have authenticated in a session to access this storage gateway.
- --high-assurance
-
Flag indicating that High Assurance features are required on this storage gateway. This can only be set on create and is immutable.
- --mfa / --no-mfa (new in 5.4.21)
-
Flag indicating that access to collections on this storage gateway require that the user has authenticated with multi factor authentication using an identity in the allowed_domains. Only usable on high assurance storage gateways.
- --network-use "normal"|"minimal"|"aggressive"|"custom"|"null" (new in 5.4.76)
-
Set storage gateway network use. If custom, the storage gateway’s max and preferred concurrency and parallelism must be set. If this is set to a non-null value, then all collections which use this storage gateway will use this value instead of the endpoint’s network use setting.
- --preferred-parallelism INTEGER
-
Set the storage gateway’s preferred parallelism; requires
--network-use=custom
- --max-parallelism INTEGER
-
Set the storage gateway’s max parallelism; requires
--network-use=custom
- --preferred-concurrency INTEGER
-
Set the storage gateway’s preferred concurrency; requires
--network-use=custom
- --max-concurrency INTEGER
-
Set the storage gateway’s max concurrency; requires
--network-use=custom
- --s3-user-credential, --s3-unauthenticated
-
Either require an S3 user credential to access this Storage Gateway or not. If unauthenticated, the Storage Gateway will only allow access to public S3 buckets.
- --s3-requester-pays, --no-s3-requester-pays (new in 5.4.59)
-
Set Requester Pays parameters on S3 requests. The Globus user must acknowledge this when registering a credential.
- --bucket
BUCKET
-
Bucket to include in the root of the Storage Gateway. Give this option multiple times to include multiple buckets. This is the only way to make public S3 buckets visible to the root of an S3 Storage Gateway. Set a value of null to clear this value. If not specified, any bucket on the S3 endpoint which a user’s registered S3 key_id can access may be accessed via this storage gateway.
- --s3-endpoint URL
-
Region-specific URI of the S3 API.
- --admin-managed-credentials, --no-admin-managed-credentials
-
Flag indicating that endpoint administrators can create or update user credentials for other accounts. See globus-connect-server user-credentials s3-create.