Last Updated: June 15, 2020


The Globus Google Drive storage connector can be used for access and sharing of data on Google Drive. The connector is available as an add-on subscription to organizations with a Globus Standard subscription - please contact us for pricing.

This document describes how to use the Google Drive Connector to configure Google Drive Storage Gateways and Collections. After these steps are complete, any Globus user you have authorized can register a credential to access Google Drive files that they have access to and, if enabled, can create guest collections for sharing access using those credentials by following the instructions in How To Share Data Using Globus.

This document assumes that you or another administrator has already installed Globus Connect Server v5 on one or more data transfer nodes, and that you have an administrator role on that endpoint.

The installation must be done by an system administrator, and has the following distinct set of steps:

  • Create a storage gateway on the endpoint configured to use the Google Drive Connector.

  • Create a mapped collection using the Google Drive Storage Gateway to provide access to Google Drive Storage Gateway data.

Please contact us at support@globus.org if you have questions or need help with installation and use of the Google Drive Connector.


Google Drive Connector Virtual Filesystem

Google Drive presents its data as a hierarchical list of files owned by folders, but has different semantics than a POSIX filesystem, such as associating the file name with the data rather than the directory entry, allowing multiple drives, allowing multiple objects with the same name in the same folder, and attributes to allow files owned by others to become visible in another user’s file space.

The Google Drive Connector provides these subdirectories of the root directory of a Google Drive Connector storage gateway:

/My Drive

Files owned by the user’s Google account that are located in the user’s root directory. This is treated the user’s home directory on collections created using the Google Drive Connector.

/Shared With Me

Files and directories owned by others which have been shared with the user’s Google account.

/Starred

Files and directories to which the user’s Google account has added the starred attribute.

/Team Drives

Directories which are Google Shared Drives (formerly called Team Drives) which the user’s Google account has been granted access to.

/Trash

Files and directories which the user’s Google account has deleted.

Note

Google Drive allows multiple objects to have the same name in the same directory. If this situation is present in your data, Globus will not be able to process these ambiguous files. If you need to transfer a directory that contains such files, you must first delete or rename them.

Registration of endpoint with Google

The Globus Connect Server v5 endpoint needs to be registered as an application with Google so that users can authorize the endpoint to access Google Cloud Storage or Google Drive on their behalf. The following steps describe how the endpoint can be registered as a Google OAuth client to obtain a client id and secret from Google.

Note

The same client id and secret may be used for both Google Cloud Storage and Google Drive connectors — it is not necessary to register twice.

Prerequisites

It is necessary that these steps be performed on a fully functional Globus Connect Server 5 endpoint, as discussed above.

You will need a Google account to complete these steps, and the registration will be stored under that Google account. This account is only for registration of the application and has no bearing on Google accounts that will be allowed to use this endpoint to access data. An administrator may use an existing Google account.

Steps

  1. To register the endpoint with Google, go to the Google Developer Console

  2. If you have never created a project with Google, you will be prompted to create one. If you create a project, you do not have to change the default permissions for the project when given the option to do so.

  3. After you have created or selected a project, you will use the Google API Dashboard to enable APIs, configure the OAuth consent screen, and create credentials for use with your endpoint.

  4. Enable this project to use the APIs required to interact with Google Cloud Storage and Google Drive. Select the "Libary" menu, and repeat the following steps for these API names: Cloud Storage, Google Cloud Storage JSON API, Cloud Resource Manager API, and Google Drive API

    1. Search for the API name and select the matching result.

    2. Once on the API page, select "Enable".

  5. Select the "OAuth consent screen" menu to configure the OAuth consent screen that will be shown to users.

    1. For the "Application name", enter "Globus Connect Server".

    2. For the "Scopes for Google APIs" section, select "Add Scopes", then select "manually paste", and paste the following scopes before selecting "ADD":

      1. Google Drive

         https://www.googleapis.com/auth/drive.appdata
         https://www.googleapis.com/auth/drive
      2. Google Cloud Storage

         https://www.googleapis.com/auth/cloudplatformprojects.readonly
         https://www.googleapis.com/auth/devstorage.read_write
    3. For "Authorized domains", add globus.org and your own domain

    4. For "Application Homepage link" and "Application Privacy Policy link", enter a URL from your own domain, or "https://globus.org".

    5. Other fields are optional.

    6. Select "Save".

  6. Select the "Create credentials" button, and then the "OAuth client ID" option.

    1. You will be prompted to select an application type. Choose "Web application" and configure it as follows:

      1. Name: set a descriptive name to be able to identify the registration of this endpoint in your projects on the Google API Manager. For example, the endpoint Display Name can be used for this.

      2. Authorization redirect URIs: set to the value that was displayed when the endpoint was created. If you don’t have that value handy, you can run the command

        globus-connect-server endpoint show

        You’ll see output that looks somthing like this:

        Display Name:    Test Endpoint
        ID:              669ec822-ca79-455c-89a7-cccb7aefbf8e
        Subscription ID: 6e62e6d7-e368-45f4-a23d-fb41243e8005
        Public:          True
        GCS Manager URL: https://21542.data.globus.org
        Network Use:     normal

        You can construct the auth callback URL by appending /api/v1/authcallback_google to the value of the GCS Manager URL. In this example case, the result is https://21542.data.globus.org/api/v1/authcallback_google.

      3. Select "Create".

  7. Make note of the client ID and secret you get from Google for this application, as you will need them to configure the storage gateway.

Storage Gateway

A Google Drive Storage Gateway is created with the command globus-connect-server storage-gateway create google-drive, and can be updated with the command globus-connect-server storage-gateway update google-drive.

Before looking into the policy options specific to the Google Drive Connector, please familiarize yourself with the Globus Connect Server v5 Data Access Guide which describes the steps to create and update a storage gateway, using the POSIX connector as an example. The commands to create and update a storage gateway for the Google Drive Connector are similar.

Google Drive Connector Storage Gateway Policies

The Google Drive Connector has policies to manage application credentials, and set the user api rate quota.

Application Credentials

The --google-client-id and --google-client-secret command-line options provide information for Globus Connect Server to authenticate with Google Drive Connector. These values must be configured in order to be able to access data on collections created with the Google Drive Connector type.

Example 1. Setting Google Drive Connector Application Credentials.

For our example, we’ll assume we’ve obtained credentials as described above. We’ll use the command-line options --google-client-id and --google-client-secret to configure these on our storage gateway.

    --google-client-id GOOGLE_CLIENT_ID \
    --google-client-secret GOOGLE_CLIENT_SECRET

User API Quota

The --google-drive-api-rate-quota command-line option allows you to configure a value for the User API Quota in order to try to avoid issues when interacting with the Google Drive API. This can be helpful to increase the quota if you have negotiated a larger API quota on your Google Drive API project or to decrease the value if you are experiencing frequent API quota errors when performing transfers. The value of the setting is a number of API operations per 100 seconds per user.

Example 2. Setting a User API Quota

For our example, we’ll assume we’ve negotiated a larger API rate quota, and of 5000 operations per 100 seconds per user. We’ll set this using the --google-drive-api-rate-quota option.

    --google-drive-api-rate-quota 5000

Creating the Storage Gateway

Now that we have decided on all our policies, we’ll use the command to create the storage gateway.

% globus-connect-server storage-gateway create google-drive \
    "Google Drive Storage Gateway" \
    --domain example.org \
    --google-client-id GOOGLE_CLIENT_ID \
    --google-client-secret GOOGLE_CLIENT_SECRET \
    --google-drive-api-rate-quota 5000

Storage Gateway Created: 7187a9a0-68e4-48ea-b3b9-7fd06630f8ab

This was successful and the output the ID of the new storage gateway ( 7187a9a0-68e4-48ea-b3b9-7fd06630f8ab in this case) for our reference. Note that this will always be a unique value if you run the command. If you forget the id of a storage gateway, you can always use the command globus-connect-server storage-gateway list to get a list of the storage gateways on the endpoint.

You can also add other policies to configure additional identity mapping and path restriction policies as described in the Globus Connect Server v5 Data Access Guide.

Note that this creates the storage gateway, but does not yet make it accessible via Globus and HTTPS. You’ll need to follow the steps in the next section.

Collection

A Google Drive Collection is created with the command globus-connect-server collection create, and can be updated with the command globus-connect-server collection update.

As the Google Drive Connector does not introduce any policies beyond those used by the base collection type, you can follow the sequence in the Collections Section of the Globus Connect Server v5 Data Access Guide. Recall however, that the paths are interpreted as described above in Google Drive Connector Virtual Filesystem.

User Credential

As mentioned in above, access to mapped collections on a Google Drive require users to register credentials. These credentials are created by performing an authentication flow with Google. This is initiated by visiting the Credentials tab of the collection. The user is directed to that page when they first attempt to access that collection.

Appendix A: Document Types for the Google Drive Connector

GoogleDriveStoragePolicies Document

This type describes the public and private policies for a Google Drive Storage Gateway.

Name

Type

Description

DATA_TYPE

string google_drive_storage_policies#1.0.0

Type of this document

client_id

string

Client ID registered with the Google Application console to access Google Drive.[Private]

secret

string

Secret created to access access Google Drive with the client_id in this policy.[Private]

user_api_rate_quota

integer

User API Rate quota associated with this client ID.[Private]

auth_callback

string <uri>

URL of the auth callback that must be registered on the Google API console for the application client_id in order to process Google credentials.

{
  "DATA_TYPE": "google_drive_storage_policies#1.0.0",
  "client_id": "string",
  "secret": "string",
  "user_api_rate_quota": 1000,
  "auth_callback": "https://example.globus.org/api/v1/authcallback_google"
}

GoogleDriveUserCredentialPolicies Document

The GoogleDriveUserCredentialPolicies document describes the google-specific policy information included in a UserCredential. This document contains read-only data about the user’s Google OAuth tokens.

Name

Type

Description

DATA_TYPE

string google_drive_user_credential_policies#1.0.0

Type of this document

sub

string

OAuth subject identifier claim.

email

string <email>

OAuth email claim.

access_token

string

OAuth access token.

refresh_token

string

OAuth refresh token.

scopes

array (string)

OAuth scopes associated with this access token.

token_expiry

string <date-time>

OAuth access token expiration time.

{
  "DATA_TYPE": "google_drive_user_credential_policies#1.0.0",
  "sub": "string",
  "email": "user@example.com",
  "access_token": "string",
  "refresh_token": "string",
  "scopes": [
    "email",
    "profile",
    "https://www.googleapis.com/auth/drive",
    "https://www.googleapis.com/auth/drive.appfolder"
  ],
  "token_expiry": "2020-02-04T21:44:12Z"
}

© 2010- The University of Chicago Legal