How to use Application (app) credentials or Service Accounts to automate data transfer

This document covers the use case where the data is moved as the application itself, and does NOT use a user’s credential. This allows for complete automation of the transfer, without requiring any user intervention.

  • Application credential: An application can have its own identity and credentials in Globus. This is setup by registering the application as a client with Globus. These identities are of the form CLIENT_ID@clients.auth.globus.org. The identity of the application is treated as any other identity in the Globus ecosystem, and can be used to grant permissions and roles in other services. For example, the app can be granted monitor role on an endpoint, read or write permission on a collection, manager role on a group.

  • Sharing: a guest collection (v5) or shared endpoint (v4) or GCP shares allows the user to set permission for an identity to access the data (read or read/write). The application identity can be granted that permission as well.

The use case here is for the data to be moved as the application itself, and NOT using a user’s credentials.

1. Usage Patterns

Some example scenarios where this is applicable:

  • Automate data transfer: A script or application can be set up to transfer data using service credentials for full automation.

  • Pull data from a user’s collection: A user can create a guest collection or share, and set permission for an application to read data from the specific folder. This will be a one time operation. Reading data from the guest collection can then be automated using application credentials.

  • Push data to a user’s collection: A user can create a guest collection or share, and set permission for an application to write data from the specific folder. This will be a one time operation. The application can now automatically push data to the folder it has write access on.

2. Steps

  1. Obtain an application/service service credential

    1. On https://developers.globus.org/, choose the option to "Register your app with Globus"

    2. Create a Project, or choose an existing Project

      1. The application registration needs to be part of a Project. This allows you to grant permission for other users to manage the registration, and rotate the application secret.

      2. (Optionally) Set other users as administrator(s) of the Project

      3. Further details are provided in the documentation on managing projects

    3. Under the "Add" drop down, choose "Add new app" option

      1. Provide a display name for the app. This will be shown when permissions are set for this application.

      2. Callback URL can be any valid URL for the purpose of this use case, where the application credentials will be used to operate as itself, rather than on behalf of the user.

      3. Rest of the fields can be left to default, since this is going to be used for the application authenticating as itself.

      4. Further details are provided in the documentation on registering applications

    4. Save application registration information to use in the code that will initiate the transfer

      1. Client ID

      2. The Client Identity Username of the app

      3. Generate a secret and save the secret

        1. The label allows you to rotate the secret and identify the secret. It is not used elsewhere

  2. Create a guest collection on source

    1. Find the mapped collection or host endpoint to create a guest collection https://app.globus.org/file-manager

    2. The user who authenticates, and the local account the user is mapped to, is the account that will be used to read the data

  3. Set permission for the app to read from the guest collection

    1. On the guest collection, choose the folder the app needs to read data from, and set permissions. The identity to set permission on the Client Identity Username of the app.

      1. <CLIENTID>@clients.auth.globus.org

    2. Using CLI: https://docs.globus.org/cli/reference/endpoint_permission_create/

      1. globus endpoint permission create <Guest Collection ID>/<PATH> --permissions r --identity <CLIENTID>@clients.auth.globus.org

  4. Create a guest collection on destination

    1. Find the mapped collection or host endpoint to create a guest collection https://app.globus.org/file-manager

    2. The user who authenticates, and the local account the user is mapped to, is the account that will be used to read the data

  5. Set permission for the app to write to the guest collection

    1. On the guest collection, choose the folder the app needs to write data to and set permissions. The identity to set permission on the Client Identity Username of the app.

      1. <CLIENTID>@clients.auth.globus.org

    2. Using CLI: https://docs.globus.org/cli/reference/endpoint_permission_create/

      1. globus endpoint permission create <Guest Collection ID>/<PATH> --permissions rw --identity <CLIENTID>@clients.auth.globus.org

  6. Application to start the transfers using the credentials

    1. Using the client credential grant, an application can submit transfers using the client id and secret for authentication.

    2. Using the Globus SDK: https://globus-sdk-python.readthedocs.io/en/stable/examples/client_credentials.html

    3. Example Python app: https://github.com/globus/automation-examples/blob/master/globus_folder_sync.py

  7. Script using the Globus CLI to start the transfers using the credentials

    1. The Globus CLI can use the client id and secret to authenticate, and commands in the CLI can be used to submit transfers.

    2. Configuring client id and secret: https://docs.globus.org/cli/environment_variables/#client_credentials_with_globus_cli_client_id

    3. Example script that uses the CLI: https://github.com/globus/automation-examples/blob/master/cli-sync.sh