Automated Endpoint Deployment Guide

1. Introduction

The following guide explains the process of automating a GCS v5.4 endpoint deployment and configuration. This guide does not replace the Installation Guide, you should have a completed GCS installation.

2. Register for Service Credentials

More traditional GCS deployment and configuration procedures require human interaction in order to provide the necessary authorization for the deployment and to identify the individual or organization owning the endpoint. In order to perform these operations without human interaction, you must use Globus Auth service credentials. Similar to user credentials, service credentials use an ID and secret in order to authenticate with Globus services. Scope consent is automatic for service credentials and so no human interaction is required.

One set of service credentials can be used to deploy one or more GCS endpoints. The service credentials must exist for the lifetime of the endpoint or until the endpoint’s owner is changed to a user. For more details, see Changing the Endpoint Owner.

Note
The service credentials are not the same as the endpoint’s client registration which are created when running globus-connect-server endpoint setup. The service credentials effectively own the endpoint and take the place of a human identity.

  1. Go to https://app.globus.org/settings/developers.

  2. Select "Register a service account or application credential for automation"

  3. Select an existing project or create a new project with a descriptive name such as "Service Identities for Automation".

  4. In the "App Name" field, give the service credentials a useful name, for example "GCS Endpoint Deployment Automation". Select "Register App".

  5. Note the value for "Client UUID"; that is the identity id of the new service credentials.

  6. Select "Add Client Secret" and save the generated secret; this is the password portion of the new service credentials.

Note
You may return to this service credentials page at any time to invalidate older secrets and generate new secrets for use with endpoint deployments.

The examples below will assume the service credentials are:

  • Client UUID: e0558739-6e6f-4600-a46d-983d309f88ff

  • Secret: pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg=

3. Create a Project Administrator Role for the Service Identity

The globus-connect-server endpoint setup command will register an endpoint client under a project administered by the identity running the command. In the case of an automated deployment, the endpoint’s client registration will be under a Globus Auth project administered by the service identity you just created.

  1. Go to https://auth.globus.org/v2/web/developers.

  2. Select an existing project, or create a new project, to use for the client registration of the endpoint you will deploy using the service identity.

  3. Select "Add" → "Add/remove admins"

  4. Add the service identity in the "Add admin to project" text field. For our example, the value would be e0558739-6e6f-4600-a46d-983d309f88ff@clients.auth.globus.org.

The examples below will assume the project ID is:

  • Project ID: 06bada42-decf-42bb-9439-5c5132a1cd09

4. Create the Endpoint

Use the service credentials to perform the endpoint setup through use of the environment variables $GCS_CLI_CLIENT_ID and $GCS_CLI_CLIENT_SECRET. Set these environment variables to the client’s UUID and secret, respectively. For more information on using the GCS CLI with client credentials, see the GCS CLI Reference.

The following options are important for automating endpoint setup:

--agree-to-letsencrypt-tos

When this option is not specified, the GCS CLI will prompt the caller in order to confirm agreement with the Let’s Encrypt TOS. You must use this option in order to automate the deployment.

--project-id

If the service identity is has an administrator role on more than one Globus Auth project, you must specify the project ID to use when creating the endpoint’s client registration.

--owner

If you wish to automate configuration of the endpoint, you must specify the service identity as the endpoint owner. That way, the service identity will have the authorization to make configuration changes on the endpoint using the GCS CLI. However, if you wish to deploy an unconfigured endpoint which will be configured by a another user, you can specify the user’s identity as the value to the --owner option.

--dont-set-advertised-owner

This option is not strictly necessary for automation, however, using it with automated deployments may avoid confusion. When this option is given, the endpoint’s "Advertised Owner" will be set to the endpoint ID. Without this option, the "Advertised Owner" will be set to the service identity.

Example
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  globus-connect-server endpoint setup "Automated Endpoint" \
  --owner e0558739-6e6f-4600-a46d-983d309f88ff@clients.auth.globus.org \
  --project-id 06bada42-decf-42bb-9439-5c5132a1cd09 \
  --organization Globus \
  --contact-email support@globus.org \
  --agree-to-letsencrypt-tos \
  --dont-set-advertised-owner
  [#########################]  100%
Created endpoint 9d74df99-b7b6-41b6-8c62-44ccef04c4e1

5. Set Up a Data Transfer Node

Node setup does not require any special considerations for automation. Perform the setup as you would for any other endpoint. This step requires the deployment key created in the previous step.

Example
$ sudo globus-connect-server node setup
IP address not specified, using 52.33.140.104
Configuring endpoint
  [#########################]  100%
Starting services
  [#########################]  100%

6. (Optional) Set the Endpoint as Managed

In order to set the endpoint as managed using service credentials, the service identity must be assigned subscription manager role. Contact support@globus.org with the service credentials ID to perform this step. This only needs to be done once for the service identity.

7. Configure the Endpoint using Service Credentials

In order to configure the endpoint, use the GCS CLI as you would with traditional endpoint configuration except set the environment variables $GCS_CLI_CLIENT_ID, $GCS_CLI_CLIENT_SECRET and $GCS_CLI_ENDPOINT_ID to the service credentials ID, secret and ID of the created endpoint, respectively. For more information on using the GCS CLI with client credentials, see the GCS CLI Reference.

Example: Create a storage gateway
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  GCS_CLI_ENDPOINT_ID=9d74df99-b7b6-41b6-8c62-44ccef04c4e1 \
  globus-connect-server storage-gateway create posix "My Storage Gateway" \
  --domain globus.org
Storage Gateway ID: 0cde732b-a807-4904-8fa4-2e774453647b
Example: Create a mapped collection
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  GCS_CLI_ENDPOINT_ID=9d74df99-b7b6-41b6-8c62-44ccef04c4e1 \
  globus-connect-server collection create 0cde732b-a807-4904-8fa4-2e774453647b / "My Collection"
Collection ID: 4eceba6c-6e4d-4960-bd4c-46335443ec36
Example: Add an endpoint admin role (managed endpoints only)
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  GCS_CLI_ENDPOINT_ID=9d74df99-b7b6-41b6-8c62-44ccef04c4e1 \
  globus-connect-server endpoint role create administrator johndoe@globus.org
Role ID: a37b4072-41d4-11ee-9912-89de40d8f604

8. (Optional) Changing the Endpoint Owner

You may change the owner of the endpoint to another identity. If the endpoint is managed, the service identity will no longer be the endpoint owner but it will continue to have an endpoint administrator role. If the endpoint is not managed, the service identity will no longer be authorized to configure the endpoint.

Note
Only remove the service identity’s ability to configure the endpoint once automate configuration and deployment has been completed.
Example: Change the endpoint’s owner
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  GCS_CLI_ENDPOINT_ID=9d74df99-b7b6-41b6-8c62-44ccef04c4e1 \
  globus-connect-server endpoint set-owner johndoe@globus.org
Example: Delete the service identity’s endpoint admin role (managed endpoints only)
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  GCS_CLI_ENDPOINT_ID=9d74df99-b7b6-41b6-8c62-44ccef04c4e1 \
  globus-connect-server endpoint role list
Role ID                              | Role          | Principal
------------------------------------ | ------------- | ---------------------
27651aac-ea87-11ed-9f1d-9b0823457c31 | administrator | johndoe@globus.org
5c9d256a-349f-4a95-940d-4ee3db0467a8 | owner         | johndoe@globus.org
9e1b583c-d7b3-11ed-b1ac-ed14b07a09ef | administrator | e0558739-6e6f-4600-a46d-983d309f88ff@clients.auth.globus.org

$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  GCS_CLI_ENDPOINT_ID=9d74df99-b7b6-41b6-8c62-44ccef04c4e1 \
  globus-connect-server endpoint role delete 9e1b583c-d7b3-11ed-b1ac-ed14b07a09ef

9. Cleaning Up a Data Transfer Node

Node cleanup does not require any special considerations for automation. Perform the cleanup as you would for any other endpoint.

Example
$ sudo globus-connect-server node cleanup
  [#########################]  100%

10. Deleting the Endpoint

Endpoint cleanup does not require any special considerations for automation. Perform the cleanup as you would for any other endpoint. This step requires the deployment key created in the endpoint setup step.

The following options are important for automating endpoint cleanup:

--lost-deployment-node-key

When this option is not specified, the GCS CLI will prompt the caller if the deployment key does not contain a valid node key. Using this option allows the cleanup to proceed though some GCS resources will not be deleted.

--agree-to-delete-endpoint

When this option is not specified, the GCS CLI will prompt the caller in order to confirm that the endpoint should be deleted. This is a safeguard for interactive cleanups.

Example
$ GCS_CLI_CLIENT_ID=e0558739-6e6f-4600-a46d-983d309f88ff \
  GCS_CLI_CLIENT_SECRET=pIee1Pl9ygusHx+GZhJuNNgF03lJFCpaAQoLh+16STg= \
  globus-connect-server endpoint cleanup --agree-to-delete-endpoint
Loaded deployment key ga_HBQiJaza06lfgqBJP9GkL4ZFx1RTC_sLCL8tfGeU
This command will completely remove all configuration information associated
with this endpoint. This is irreversible, and the client_id used to create the
endpoint may not be reused to recreate this endpoint.
  [#########################]  100%