How To Use Application Credentials or Service Accounts to Automate Data Transfer

This document covers the use case where data are moved as the application itself and does not use a user’s credential. This allows for complete automation of the transfer, without requiring any user intervention.

  • Application credential: An application can have its own identity and credentials in Globus. This is setup by registering the application as a client with Globus. These identities are of the form CLIENT_ID@clients.auth.globus.org. The identity of the application is treated as any other identity in the Globus ecosystem, and can be used to grant permissions and roles in other services. For example, the app can be granted monitor role on an endpoint, read or write permission on a collection, or manager role on a group.

  • Sharing: A guest collection allows the user to set permission for an identity, including a client identity, to access the data (read or read/write).

1. Usage Patterns

Example scenarios where data movement as the application itself is applicable follow.

  • Automate data transfer: A script or application can be set up to transfer data using service credentials for full automation.

  • Pull data from a user’s guest collection: A user can create a guest collection and set permission for an application to read data from a specific folder. This will be a one time operation. Reading data from the guest collection can then be automated using application credentials.

  • Push data to a user’s collection: A user can create a guest collection and set permission for an application to write data to a specific folder. This will be a one time operation. The application can now automatically push data to the folder on which it has write access.

2. Steps

  1. Obtain an application/service credential.

    1. At https://app.globus.org/settings/developers, choose the option to "Register a service account or application credential for automation".

    2. Create a new Project or choose an existing Project.

      1. The application registration needs to be part of a Project. This allows you to grant permission for other users to manage the registration and to rotate the application secret.

      2. Further details are provided in the documentation on managing projects.

    3. Provide a display name for the application.

      1. This will be shown when permissions are set for this application.

      2. (Optionally) Provide Privacy Policy and/or Terms & Conditions URLs.

      3. Further details are provided in the documentation on registering applications.

    4. Note the Client ID.

      1. This is the username of the application identity.

    5. Generate and save the secret.

      1. Select "Add Client Secret".

      2. Name the secret.

        1. The secret name allows you to identify the secret, which can be useful when rotating the secret. It is not used elsewhere.

      3. Save the secret. You will not be able to access the secret again.

  2. Create a guest collection on the source.

    1. Find the mapped collection or host endpoint to create a guest collection https://app.globus.org/file-manager.

    2. The user who authenticates, and the local account the user is mapped to, is the account that will be used to read the data.

  3. Set permission for the application to read from the guest collection.

    1. Using the Globus Web App

      1. On the guest collection, choose the folder from which the app needs to read data, and set permissions.

      2. Paste in the client identity (<CLIENTID>@clients.auth.globus.org) as the Username or Email when adding the permission.

    2. Using the CLI: https://docs.globus.org/cli/reference/endpoint_permission_create/

      1. globus endpoint permission create <Guest Collection ID>/<PATH> --permissions r --identity <CLIENTID>@clients.auth.globus.org

  4. Create a guest collection on the destination

    1. Find the mapped collection to create a guest collection at https://app.globus.org/file-manager.

    2. Create a guest collection.

      1. The user who authenticates to create the guest collection, and the local account the user is mapped to, is the account that will be used to read the data by the application.

  5. Set permission for the application to write to the guest collection.

    1. Using the Globus Web App

      1. On the guest collection, choose the folder from which the application needs to read data, and set permissions.

      2. Paste in the client identity (<CLIENTID>@clients.auth.globus.org) as the Username or Email when adding the permission.

    2. Using CLI: https://docs.globus.org/cli/reference/endpoint_permission_create/

      1. globus endpoint permission create <Guest Collection ID>/<PATH> --permissions rw --identity <CLIENTID>@clients.auth.globus.org

  6. Application to start the transfers using the credentials

    1. Using the client credential grant, an application can submit transfers using the client id and secret for authentication.

    2. Using the Globus SDK: https://globus-sdk-python.readthedocs.io/en/stable/examples/client_credentials.html

    3. Example Python app: https://github.com/globus/automation-examples/blob/master/globus_folder_sync.py

  7. Script using the Globus CLI to start the transfers using the credentials

    1. The Globus CLI can use the client id and secret to authenticate, and commands in the CLI can be used to submit transfers.

    2. Configuring client id and secret: https://docs.globus.org/cli/environment_variables/#client_credentials_with_globus_cli_client_id

    3. Example script that uses the CLI: https://github.com/globus/automation-examples/blob/master/cli-sync.sh