Guide for applications accessing data on Globus Connect Server v5

Table of Contents

1. Introduction
2. Terminology
3. Using multiple APIs
4. Determining collection type
5. Using Globus Connect Server version 5 mapped collections
- 5.1. data access consents
- 5.2. Authentication Timeouts
6. Using Globus Connect Server version 5 guest collections

Last Updated: October 14, 2022

1. Introduction

This guide is to aid developers in writing applications that access data on Globus Connect Server version 5. Doing so requires a combination of calls against both the Transfer API and GCS Manager API(s), so this guide will primarily cover those areas of API functionality at a high level with links to the respective APIs' documentation for further details.

This guide assumes the developer is familiar with how to get and use tokens from Globus Auth. If this is not the case, the Globus Auth Developer Guide and Globus SDK tutorial are good places to start.

2. Terminology

Globus Connect Server version 5 has four types of entities:

An endpoint, which manages server and network configuration across all connectors, and does not have any data access interfaces.
A storage gateway, which provides connector specific configuration to collections created on it and does not have any data access interfaces.
A mapped collection, which provides a data access interface for users with local accounts. This uses OAuth2 for authentication, and user identity is mapped to a local account.
A guest collection, which provides a data access interface that uses permissions created in the Transfer service to allow users data access without requiring local accounts.

Note

The Transfer API uses older terminology in most of its paths and fields that does not differentiate between a collection and an endpoint. Because of this Globus Connect Server version 5 collections will frequently be referred to as endpoints by the Transfer API.

3. Using multiple APIs

In many cases to access data on a Globus Connect Server version 5 collection, an application will need to make calls against both the Transfer API and the GCS Manager API for that collection. To do so, the application will need to get multiple tokens scoped for each service separately.

Note

Since your application will be talking to at least two services, and may need to handle an unknown number of GCS Manager services, token management can be a point of complexity. SDK users may want to use TokenStorage to help manage tokens for multiple services.

The Transfer API is a central service for task submission and management, permission management, and collection discovery. The GCS Manager API is an endpoint specific service for collection creation and administration. Many entities, such as Globus Connect Server version 5 collections and roles, are managed by the GCS Manager API but are reflected as read-only entities in the Transfer API.

The table below lists many of the actions an application may want to take to access or manage data on a Globus Connect Server version 5 collection, which API the application will need to use, the API path for that API action, and the corresponding Globus SDK helper method(s).

Note

Entities that can be viewed by both the Transfer and GCS Manager APIs will have similar but not identical response documents. Some fields may not be present in both documents, or use different naming or formatting.

Action Description	Which API to use	Request Path	SDK method
Perform a directory listing on a collection	Transfer	GET /operation/endpoint/{collection_id}/ls	TransferClient.operation_ls
Submit a transfer task between two collections	Transfer	POST /transfer	TransferClient.submit_transfer
Submit a delete task on a collection	Transfer	POST /delete	TransferClient.submit_delete
Make a directory on a collection	Transfer	POST /operation/endpoint/{collection_id}/mkdir	TransferClient.operation_mkdir
Rename a file or directory on a collection	Transfer	POST /operation/endpoint/{collection_id}/rename	TransferClient.operation_rename
Search for collections	Transfer	GET /endpoint_search	TransferClient.endpoint_search
View a collection	Either	GET /endpoint/{collection_id} (Transfer) GET /collection/{collection_id} (GCS Manager)	TransferClient.get_endpoint GCSClient.get_collection
Create a collection	GCS Manager	POST /collections	GCSClient.create_collection
Update a collection	GCS Manager	PATCH /collections/{collection_id}	GCSClient.update_collection
Delete a collection	GCS Manager	DELETE /collections/{collection_id}	GCSClient.delete_collection
List roles on a collection	Either	GET /endpoint/{collection_id}/role_list (Transfer) GET /roles (GCS Manager)	TransferClient.endpoint_role_list GCSClient.get_role_list
Create a role on a collection	GCS Manager	POST /roles	GCSClient.create_role
Delete a role on a collection	GCS Manager	DELETE /roles/{role_id}	GCSClient.delete_role
View permissions on a guest collection	Transfer	GET /endpoint/{collection_id}/access_list	TransferClient.endpoint_acl_list
Create a permission on a guest collection	Transfer	POST /endpoint/{collection_id}/access	TransferClient.add_endpoint_acl_rule
Delete a permission on a guest collection	Transfer	DELETE /endpoint/{collection_id}/access/{rule_id}	TransferClient.delete_endpoint_acl_rule

4. Determining collection type

Mapped and guest collections behave differently in many ways due to their differing method of local authentication. If your application is discovering collections through Transfer endpoint search, or being passed UUIDs without context, then you will need to be able to determine what kind of collection (or other entity) you are dealing with.

The most reliable way to determine what type an endpoint or collection your application is dealing with is a Transfer endpoint document.

If your application is using Transfer’s endpoint search (TransferClient.endpoint_search in the SDK) to discover endpoints and collections, then the documents returned will be sufficient.

If your application only has the UUID of an endpoint or collection, then you can make a GET /endpoint/{collection_id} API call (TransferClient.get_endpoint in the SDK) to get a Transfer endpoint document for that endpoint or collection.

Once you have a Transfer endpoint document, the endpoint or collection type can be determined from specific fields within as described in the Transfer API’s endpoint types documentation. Globus Connect Server version 5 mapped collections will have a non null non_functional_endpoint_id but a null mapped_collection_id. Globus Connect Server version 5 guest collections will have both a non null non_functional_endpoint_id and a non null mapped_collection_id.

5. Using Globus Connect Server version 5 mapped collections

Globus Connect Server version 5 mapped collections differ from other collections types in three key ways. They (1) use local filesystem permissions to determine access, (2) require a collection specific user consent for data access, and (3) and have authentication timeouts regardless of high assurance status.

5.1. data access consents

Note

High assurance Globus Connect Server version 5 mapped collections do not require a data access consent since they already have stricter session requirements. For more information see the High Assurance Collections guide.

To access data on a mapped collection or create a guest collection on that mapped collection, users will need to consent to the Transfer service accessing the specific mapped collection on behalf of the user. This is implemented as a dependent scope on the transfer scope for the specific mapped collection. Transfer API access will need to present tokens that have the dependent data access scope for the specific mapped collection. These consents are not time limited and are granted once per application unless explicitly revoked by the user.

If you know ahead of time which mapped collections your application will be using, you should add their data access scopes to the Authorization Grant’s scope field when users authenticate to your application. The scopes will be in the format urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/COLLECTION_ID/data_access] where COLLECTION_ID is the uuid of the mapped collection to which the user is consenting.

If you don’t know which mapped collections your application will be using, you will need to respond reactively to ConsentRequired errors raised when calls are made to mapped collections users have not yet consented to. These error responses include the required scopes in their JSON bodies, which should then be used in a new user authentication flow to get the user’s consent and new tokens. SDK users may want to use the ConsentRequiredInfo class for ease of accessing the scope information from the error.

Note

If your application uses a Client Credentials Grant to get tokens to act as itself, add the required data access scopes to the scope filed requested in that grant. Client Identities consent to scopes just by requesting them.

5.2. Authentication Timeouts

Mapped collections are always configured with an authentication timeout, visible as the authentication_timeout_mins in the Globus Connect Server version 5 Mapped Collection document document or Transfer Endpoint document.

Users must have authenticated with an identity from one of the mapped collection’s required domains within that timeout in order to use that mapped collection. Otherwise some form of Permission Denied error will be returned. These errors will be in the session error format with an authorization_parameters field containing the values needed for the user to re-authenticate. Only high assurance collections will enforce that the authentication is in the same session as your application however.

Globus SDK users may wish to use the AuthorizationParameterInfo class for ease of accessing the authorization parameters needer for the user.

Note

Authentication timeouts do not apply to client identities. If your application uses a Client Credentials Grant to get tokens to act as itself rather than as a user, then you do not need to handle timeouts when using those tokens.

6. Using Globus Connect Server version 5 guest collections

Unlike mapped collections guest collections are designed for external sharing and automation use cases. They use permissions in the Transfer service to determine access to the collection owner’s local account, do not require additional consents, and only have authentication timeouts when high assurance.

Because of this, once guest collections have been set up, applications generally only need to use the Transfer service to interact with them similar to Globus Connect Personal collections and the tutorial collections.

Because of this, any examples using the tutorial collections, such as the SDK’s minimal transfer example can be adapted to use guest collections by replacing the endpoint_id variables with guest collection ids and ensuring a permission or role allowing the user data access to the guest collection.