Accessing Globus Connect Server with Application Credentials

This document describes how to configure a Globus Connect Server v5.4 endpoint to allow a portal or other service to interact with the GCS Manager API. This allows programmatic creation and management of guest collections.

Note
The functionality described here requires globus-connect-server54 version 5.4.26 or newer.

1. Installation

Follow the instructions in the GCS v5 Installation Guide to configure an endpoint. Be sure to set the endpoint as managed in order to create guest collections.

2. Register your application with Globus Auth

Visit the Globus Developers web interface to create a client ID and secret for use with your application.

  1. Follow the prompts to select or create a project that will contain the registration.

  2. Enter a descriptive name for your client in the "App Name" field.

  3. Select 'Register App'.

Note: The app client configuration presented here is the minimum viable for the examples in the document. Other app client configurations are possible. See https://docs.globus.org/api/auth/developer-guide/#register-app for details on what else can be entered in the various fields.

Once the application is created, create a secret by clicking "Add Client Secret" button, give the secret a name, and then record the client ID value and secret for future use.

We’ll refer to the ID and secret as CLIENT_ID and CLIENT_SECRET.

3. Configure GCS to Allow Access to your data

This section describes how to configure GCS to allow access via your new Globus application.

3.1. Create a service user for your application on your storage system

Details vary based on the connector being used:

Posix, Posix Staging

Create an account on the data transfer node for the application to be mapped to. Accesses to the file system are performed as this user.

We’ll refer to this account as SERVICE_USER.

3.2. Create the identity mapping file

First, you’ll need to create an identity mapping file which will map the CLIENT_ID of your application to the SERVICE_USER.

The identity mapping file should have contents that look like the following example. Replace CLIENT_ID and SERVICE_USER with the values you obtained earlier. Call the file idmap.json.

{
  "DATA_TYPE": "expression_identity_mapping#1.0.0",
  "mappings": [
    {
      "source": "{username}",
      "match": "CLIENT_ID@clients.auth.globus.org",
      "literal": true,
      "output": "SERVICE_USER"
    }
  ]
}

For simplicity, this example assumes that the application is the only identity that will access the mapped collection (ie. users will not be listing directories, transferring files or creating guest collections without the use of your app). If you intend to allow users normal access to the endpoint, you can use the techniques described in the GCS Identity Mapping Guide to add additional allowed domains and map them to local users.

3.3. Create the storage gateway

The storage gateway contains the policy that governs what storage is available to collections and what users are allowed to access data on those collections. For this case, create the storage gateway using the following command-line. Replace the string DISPLAY_NAME with something appropriate for your site.

globus-connect-server storage-gateway create posix \
    --domain clients.auth.globus.org \
    --identity-mapping file:idmap.json \
    DISPLAY_NAME

This will output the new storage gateway’s ID, which we’ll refer to as STORAGE_GATEWAY_ID.

3.4. Create the mapped collection

A mapped collection allows access to data for users who have accounts allowed by the storage gateway. Use the following command to create a mapped collection that allows the creation of guest collections. See the collection create reference for more options.

globus-connect-server collection create STORAGE_GATEWAY_ID \
    --allow-guest-collections \
    BASE_PATH DISPLAY_NAME

This command will output the ID of the mapped collection, which we’ll refer to as MAPPED_COLLECTION_ID.

4. Creating guest collections in your application

In order to create a guest collection within your application, you’ll need the following information available to your application:

CLIENT_ID, CLIENT_SECRET

You obtained these values in Register your application with Globus Auth.

ENDPOINT_ID, GCS_MANAGER_DOMAIN_NAME

You can obtain this from the endpoint_id and domain_name keys in the file /var/lib/globus-connect-server/info.json on any active GCS node, or the ID and the domain name portion of the GCS Manager URL output from globus-connect-server endpoint show. Exclude the https:// prefix from GCS_MANAGER_DOMAIN_NAME.

STORAGE_GATEWAY_ID

This was created earlier. You can use globus-connect-server storage-gateway list to find the ID if you’ve forgotten it.

MAPPED_COLLECTION_ID

This was created earlier. You can use globus-connect-server collection list to find the ID if you’ve forgotten it.

CONNECTOR_ID

Each connector has a specific UUID that identifies it. Each storage gateway is associated with a single connector. In order to get the CONNECTOR_ID for STORAGE_GATEWAY_ID, run: globus-connect-server storage-gateway show -F json STORAGE_GATEWAY_ID.

Your application will need to perform the following steps:

  • Obtain Access Tokens to access the GCS Manager API for your endpoint and the Globus Transfer API

  • (one time) Create a user credential object on your storage gateway

  • Create the guest collection

  • (Optional) Add roles to the guest collection

  • (Optional) Add permissions to the guest collection

Note
In each of the examples given below, exception handling and error handling have been omitted for clarity.

4.1. Setup the Proper Python Environment for the Following Examples

The following examples require Python 3.6 or newer and the globus_sdk and requests packages. You can setup and activate a proper python environment with the following:

python3 -mvenv globus_examples
. ./globus_examples/bin/activate
pip3 install --upgrade pip
pip3 install globus-sdk requests

4.2. Finding More Information on the Globus SDK and GCS API

More information on the Globus SDK can be found in the Globus SDK Docs. More information on the GCS API can be found in the GCS API Docs.

4.3. Obtain Access Tokens

To access the GCS Manager API to be able to create collections, you’ll need to request a scope specific to the endpoint and collection.

For a high-assurance storage gateway, you’ll need to request the scope

urn:globus:auth:scope:ENDPOINT_ID:manage_collections

For a non high-assurance storage gateway, you’ll need to request a dependent data access scope as well:

urn:globus:auth:scope:ENDPOINT_ID:manage_collections[*https://auth.globus.org/scopes/MAPPED_COLLECTION_ID/data_access]

If you are adding permissions to the collection as part of your application, you’ll also need to request the scope

urn:globus:auth:scope:transfer.api.globus.org:all

To do so, you’ll need to post a request to the Globus Auth service requesting a client credentials grant for the desired scopes. In this example, we use the Globus SDK to perform the Auth request and manage our access token.

#!/usr/bin/env python3

import globus_sdk

# Substitute your values here:
ENDPOINT_ID = "ENDPOINT_ID"
CLIENT_ID = "CLIENT_ID"
CLIENT_SECRET = "CLIENT_SECRET"

# The authorizer manages our access token for the scopes we request
authorizer = globus_sdk.ClientCredentialsAuthorizer(
    # The ConfidentialAppAuthClient authenticates us to Globus Auth
    globus_sdk.ConfidentialAppAuthClient(
        CLIENT_ID,
        CLIENT_SECRET
    ),
    f"urn:globus:auth:scope:{ENDPOINT_ID}:manage_collections"
)

# The access token is stored in authorizer.access_token

4.4. Create a user credential object (one time)

Globus Connect Server v5.4 contains a user credential object that contains the mapping between a Globus Auth identity and a storage-specific user name, along with any other credential information needed to access the user’s data. For the POSIX connector, the credential only contains the username. Normally this is created transparently on the user’s first data access in the Globus Web App, but since that is not possible for application credentials, you’ll need to create this manually.

This only needs to be done one time and can be done as part of your application deployment.

To do so, you can post a JSON body to the GCS Manager API user credentials endpoint at https://GCS_MANAGER_DOMAIN_NAME/api/user_credentials using an access token with the manage collections scope.

#!/usr/bin/env python3

import globus_sdk
import requests

# Substitute your values here:
ENDPOINT_ID = "ENDPOINT_ID"
CLIENT_ID = "CLIENT_ID"
CLIENT_SECRET = "CLIENT_SECRET"
GCS_MANAGER_DOMAIN_NAME = "GCS_MANAGER_DOMAIN_NAME"
STORAGE_GATEWAY_ID = "STORAGE_GATEWAY_ID"
MAPPED_COLLECTION_ID = "MAPPED_COLLECTION_ID"
SERVICE_USER = "SERVICE_USER"
CONNECTOR_ID = "CONNECTOR_ID"

# We need an access token with this scope to manage credentials
SCOPES=f"urn:globus:auth:scope:{ENDPOINT_ID}:manage_collections"

# The authorizer manages our access token for the scopes we request
authorizer = globus_sdk.ClientCredentialsAuthorizer(
    # The ConfidentialAppAuthClient authenticates us to Globus Auth
    globus_sdk.ConfidentialAppAuthClient(
        CLIENT_ID,
        CLIENT_SECRET
    ),
    SCOPES
)

# Get our existing user credentials, if any
resp = requests.get(
    'https://' + GCS_MANAGER_DOMAIN_NAME + '/api/user_credentials',
    params={'storage_gateway': STORAGE_GATEWAY_ID},
    headers={'Authorization': f'Bearer {authorizer.access_token}'},
)

USER_CREDENTIAL_ID = None
for cred in resp.json()['data']:
    if cred['storage_gateway_id'] == STORAGE_GATEWAY_ID:
        USER_CREDENTIAL_ID = cred['id']
        print(f'Existing user credential {USER_CREDENTIAL_ID} found')

if USER_CREDENTIAL_ID is None:
    resp = requests.post(
        'https://' + GCS_MANAGER_DOMAIN_NAME + '/api/user_credentials',
        headers={'Authorization': f'Bearer {authorizer.access_token}'},
        json={
            'DATA_TYPE': "user_credential#1.0.0",
            'connector_id': CONNECTOR_ID,
            'username': SERVICE_USER,
            'storage_gateway_id': STORAGE_GATEWAY_ID,
        }
    )
    print(f'Created user credential {resp.json()["data"][0]["id"]}')

The form of the JSON body is different for non-POSIX connector types.

On success, this API call will return a JSON response containing a "data" key. An element of that value’s array is the resulting user credential object. The "id" property of this dictionary is needed to create guest collections.

We’ll refer to the user credential ID as USER_CREDENTIAL_ID.

4.5. Create the Guest Collection

Post the new collection to the GCS Manager API https://GCS_MANAGER_DOMAIN_NAME/api/collections collections endpoint. You’ll need to authorize this using the bearer token with the manage collections scope.

The document can contain the fields described on the GCS v5 API Collection Schema page.

Replace USER_CREDENTIAL_ID with the id property returned from the user credential creation above.

#!/usr/bin/env python3

import globus_sdk
import requests

# Substitute your values here:
ENDPOINT_ID = "ENDPOINT_ID"
CLIENT_ID = "CLIENT_ID"
CLIENT_SECRET = "CLIENT_SECRET"
GCS_MANAGER_DOMAIN_NAME = "GCS_MANAGER_DOMAIN_NAME"
STORAGE_GATEWAY_ID = "STORAGE_GATEWAY_ID"
MAPPED_COLLECTION_ID = "MAPPED_COLLECTION_ID"
USER_CREDENTIAL_ID = "USER_CREDENTIAL_ID"
GUEST_COLLECTION_DISPLAY_NAME = "ENTER DISPLAY NAME"
GUEST_COLLECTION_BASE_PATH = "/"

#
# Choose one of the two values for SCOPES:
#

# 1) Use this value of SCOPES for collections that are not high assurance. The data_access token is
# required for creating guest collections that are not high assurance.
SCOPES=(
    f"urn:globus:auth:scope:{ENDPOINT_ID}:manage_collections"
    f"[*https://auth.globus.org/scopes/{MAPPED_COLLECTION_ID}/data_access]"
)

# 2) Use this value of SCOPES for HA collections
#SCOPES=f"urn:globus:auth:scope:{ENDPOINT_ID}:manage_collections"

# The authorizer manages our access token for the scopes we request
authorizer = globus_sdk.ClientCredentialsAuthorizer(
    # The ConfidentialAppAuthClient authenticates us to Globus Auth
    globus_sdk.ConfidentialAppAuthClient(
        CLIENT_ID,
        CLIENT_SECRET
    ),
    SCOPES
)

resp = requests.post(
    'https://' + GCS_MANAGER_DOMAIN_NAME + '/api/collections',
    headers={'Authorization': f'Bearer {authorizer.access_token}'},
    json={
        'DATA_TYPE': 'collection#1.4.0',
        'collection_type': 'guest',
        'display_name': GUEST_COLLECTION_DISPLAY_NAME,
        'identity_id': CLIENT_ID,
        'storage_gateway_id': STORAGE_GATEWAY_ID,
        'collection_base_path': GUEST_COLLECTION_BASE_PATH,
        'public': True,
        'user_credential_id': USER_CREDENTIAL_ID,
        'mapped_collection_id': MAPPED_COLLECTION_ID,
    }
)
print(resp.json())

This returns a document that contains the new ID of the collection. This is referenced as GUEST_COLLECTION_ID in the rest of this document.

4.6. Add Roles

If you want to allow users to perform administrative operations on the collection, you can create roles on the collection using the GCS Manager API. To do so post a Role document to the roles API endpoint on your GCS Endpoint at https://GCS_MANAGER_DOMAIN_NAME/api/roles.

In the example below, IDENTITY_ID is the UUID that of the linked identity that will be assigned the administrator role. You can get identity IDs manually using Globus CLI get-identities or programmatically using the Globus SDK.

#!/usr/bin/env python3

import globus_sdk
import requests

# Substitute your values here:
ENDPOINT_ID = "ENDPOINT_ID"
CLIENT_ID = "CLIENT_ID"
CLIENT_SECRET = "CLIENT_SECRET"
GCS_MANAGER_DOMAIN_NAME = "GCS_MANAGER_DOMAIN_NAME"
GUEST_COLLECTION_ID = "GUEST_COLLECTION_ID"
# This is the uuid of the user you wish to assign the role to
IDENTITY_ID = "IDENTITY_ID"

SCOPES=f"urn:globus:auth:scope:{ENDPOINT_ID}:manage_collections"

# The authorizer manages our access token for the scopes we request
authorizer = globus_sdk.ClientCredentialsAuthorizer(
    # The ConfidentialAppAuthClient authenticates us to Globus Auth
    globus_sdk.ConfidentialAppAuthClient(
        CLIENT_ID,
        CLIENT_SECRET
    ),
    SCOPES
)

resp = requests.post(
    'https://' + GCS_MANAGER_DOMAIN_NAME + '/api/roles',
    headers={'Authorization': f'Bearer {authorizer.access_token}'},
    json={
        "DATA_TYPE": "role#1.0.0",
        "collection": GUEST_COLLECTION_ID,
        "principal": f"urn:globus:auth:identity:{IDENTITY_ID}",
        "role": "administrator"
    }
)
print(resp.json())

See https://docs.globus.org/globus-connect-server/v5.4/reference/collection/role for a description of the roles available for guest collections.

4.7. Create permissions

By default, the owner of the collection is granted read-write access to the collection’s data. In this case, the owner is the application which created the collection. To allow additional users access to the data, you’ll need to use the Globus Transfer API to add permissions.

To do so, post a permission document to the Transfer service. You’ll need to use the access token with the urn:globus:auth:scope:transfer.api.globus.org:all scope as a bearer token authorization header for this request.

#!/usr/bin/env python3

import globus_sdk
import requests

# Substitute your values here:
CLIENT_ID = "CLIENT_ID"
CLIENT_SECRET = "CLIENT_SECRET"
GUEST_COLLECTION_ID = "GUEST_COLLECTION_ID"
IDENTITY_ID = "IDENTITY_ID"

SCOPES="urn:globus:auth:scope:transfer.api.globus.org:all"

authorizer = globus_sdk.ClientCredentialsAuthorizer(
    globus_sdk.ConfidentialAppAuthClient(
        CLIENT_ID,
        CLIENT_SECRET,
    ),
    SCOPES
)

# This time we are using the TransferClient supplied by the SDK
transfer_client = globus_sdk.TransferClient(authorizer=authorizer)

rule_data = {
    "DATA_TYPE": "access",
    "principal_type": "identity",
    "principal": IDENTITY_ID,
    "path": "/",
    "permissions": "rw",
}
transfer_client.add_endpoint_acl_rule(GUEST_COLLECTION_ID, rule_data)