Google Drive Connector

Last Updated: June 15, 2020

The Globus Google Drive storage connector can be used for access and sharing of data on Google Drive. The connector is available as an add-on subscription to organizations with a Globus Standard subscription - please contact us for pricing.

This document describes how to use the Google Drive Connector to configure Google Drive Storage Gateways and Collections. After these steps are complete, any Globus user you have authorized can register a credential to access Google Drive files that they have access to and, if enabled, can create guest collections for sharing access using those credentials by following the instructions in How To Share Data Using Globus.

This document assumes that you or another administrator has already installed Globus Connect Server v5 on one or more data transfer nodes, and that you have an administrator role on that endpoint.

The installation must be done by a system administrator, and has the following distinct set of steps:

  • Create a storage gateway on the endpoint configured to use the Google Drive Connector.

  • Create a mapped collection using the Google Drive Storage Gateway to provide access to Google Drive Storage Gateway data.

Please contact us at support@globus.org if you have questions or need help with installation and use of the Google Drive Connector.

Google Drive Connector Virtual Filesystem

Google Drive presents its data as a hierarchical list of files owned by folders, but has different semantics than a POSIX filesystem, such as associating the file name with the data rather than the directory entry, allowing multiple drives, allowing multiple objects with the same name in the same folder, and attributes to allow files owned by others to become visible in another user’s file space.

The Google Drive Connector provides these subdirectories of the root directory of a Google Drive Connector storage gateway:

/My Drive

Files owned by the user’s Google account that are located in the user’s root directory. This is treated the user’s home directory on collections created using the Google Drive Connector.

/Shared With Me

Files and directories owned by others which have been shared with the user’s Google account.

/Starred

Files and directories to which the user’s Google account has added the starred attribute.

/Team Drives

Directories which are Google Shared Drives (formerly called Team Drives) which the user’s Google account has been granted access to.

/Trash

Files and directories which the user’s Google account has deleted.

Note
Google Drive allows multiple objects to have the same name in the same directory. If this situation is present in your data, Globus will not be able to process these ambiguous files. If you need to transfer a directory that contains such files, you must first delete or rename them.

Registration of endpoint with Google

The Globus Connect Server v5 endpoint needs to be registered as an application with Google so that users can authorize the endpoint to access Google Cloud Storage or Google Drive on their behalf. The following steps describe how the endpoint can be registered as a Google OAuth client to obtain a client id and secret from Google.

Note
The same client id and secret may be used for both Google Cloud Storage and Google Drive connectors — it is not necessary to register twice.
Note
When configuring a service account key for Google Cloud Storage, it is not necessary to complete these steps. See Google’s service account documentation for more info on creating a service account key.

Prerequisites

It is necessary that these steps be performed on a fully functional Globus Connect Server 5 endpoint, as discussed above.

You will need a Google account to complete these steps, and the registration will be stored under that Google account. This account is only for registration of the application and has no bearing on Google accounts that will be allowed to use this endpoint to access data. An administrator may use an existing Google account.

Steps

  1. To register the endpoint with Google, go to the Google Developer Console

  2. If you have never created a project with Google, you will be prompted to create one. If you create a project, you do not have to change the default permissions for the project when given the option to do so. The project that you create should be associated with your Google/GSuite organization.

  3. After you have created or selected a project, you will use the Google API Dashboard to enable APIs, configure the OAuth consent screen, and create credentials for use with your endpoint.

  4. Enable this project to use the APIs required to interact with Google Cloud Storage and Google Drive. Select the "Libary" menu, and repeat the following steps for these API names: Cloud Storage, Google Cloud Storage JSON API, Cloud Resource Manager API, and Google Drive API

    1. Search for the API name and select the matching result.

    2. Once on the API page, select "Enable".

  5. Select the "OAuth consent screen" menu to configure the OAuth consent screen that will be shown to users.

    1. When prompted for the "User Type", we recommend that you select "Internal" when possible. You should only use "External" if you need to allow access to accounts outside of your Google/GSuite organization, or if you are not part of a Google/GSuite organization. Select "Create".

      Note
      External apps will be subject to Google’s unverified app restrictions.

    2. For the "Application name", enter "Globus Connect Server".

    3. For the "Scopes for Google APIs" section, select "Add Scopes", then select "manually paste", and paste the following scopes before selecting "ADD":

      1. Google Drive

         https://www.googleapis.com/auth/drive.appdata
 https://www.googleapis.com/auth/drive

      2. Google Cloud Storage

         https://www.googleapis.com/auth/cloudplatformprojects.readonly
 https://www.googleapis.com/auth/devstorage.read_write

    4. For "Authorized domains", add globus.org and your own domain

    5. For "Application Homepage link" and "Application Privacy Policy link", enter a URL from your own domain, or "https://globus.org".

    6. Other fields are optional.

    7. Select "Save".

  6. Select the "Create credentials" button, and then the "OAuth client ID" option.

    1. You will be prompted to select an application type. Choose "Web application" and configure it as follows:

      1. Name: set a descriptive name to be able to identify the registration of this endpoint in your projects on the Google API Manager. For example, the endpoint Display Name can be used for this.

      2. Authorization redirect URIs: set to the value that was displayed when the endpoint was created. If you don’t have that value handy, you can run the command

        globus-connect-server endpoint show

        You’ll see output that looks something like this:

        Display Name:    Test Endpoint
ID:              669ec822-ca79-455c-89a7-cccb7aefbf8e
Subscription ID: 6e62e6d7-e368-45f4-a23d-fb41243e8005
Public:          True
GCS Manager URL: https://21542.data.globus.org
Network Use:     normal

        You can construct the auth callback URL by appending /api/v1/authcallback_google to the value of the GCS Manager URL. In this example case, the result is https://21542.data.globus.org/api/v1/authcallback_google.

      3. Select "Create".

  7. Make note of the client ID and secret you get from Google for this application, as you will need them to configure the storage gateway.

Google Drive Configuration Encryption

All configuration information, including Google Drive secrets and user credential information, is encrypted with a secret key on the node servicing the request before storing it locally and uploading it to GCS cloud services for distribution to other nodes in the endpoint. The encryption key is only available locally to the node and is secured such that only the node admin has access.

Storage Gateway

A Google Drive Storage Gateway is created with the command globus-connect-server storage-gateway create google-drive, and can be updated with the command globus-connect-server storage-gateway update google-drive.

Before looking into the policy options specific to the Google Drive Connector, please familiarize yourself with the Globus Connect Server v5 Data Access Guide which describes the steps to create and update a storage gateway, using the POSIX connector as an example. The commands to create and update a storage gateway for the Google Drive Connector are similar.

Google Drive Connector Storage Gateway Policies

The Google Drive Connector has policies to manage application credentials, and set the user api rate quota.

Application Credentials

The --google-client-id and --google-client-secret command-line options provide information for Globus Connect Server to authenticate with Google Drive Connector. These values must be configured in order to be able to access data on collections created with the Google Drive Connector type.

Example 1. Setting Google Drive Connector Application Credentials.

For our example, we’ll assume we’ve obtained credentials as described above. We’ll use the command-line options --google-client-id and --google-client-secret to configure these on our storage gateway.

    --google-client-id GOOGLE_CLIENT_ID \
    --google-client-secret GOOGLE_CLIENT_SECRET

User API Quota

The --google-drive-user-api-rate-quota command-line option allows you to configure a value for the User API Quota in order to try to avoid issues when interacting with the Google Drive API. This can be helpful to increase the quota if you have negotiated a larger API quota on your Google Drive API project or to decrease the value if you are experiencing frequent API quota errors when performing transfers. The value of the setting is a number of API operations per 100 seconds per user.

Example 2. Setting a User API Quota

For our example, we’ll assume we’ve negotiated a larger API rate quota, and of 5000 operations per 100 seconds per user. We’ll set this using the --google-drive-user-api-rate-quota option.

    --google-drive-user-api-rate-quota 5000

Creating the Storage Gateway

Now that we have decided on all our policies, we’ll use the command to create the storage gateway.

% globus-connect-server storage-gateway create google-drive \
    "Google Drive Storage Gateway" \
    --domain example.org \
    --google-client-id GOOGLE_CLIENT_ID \
    --google-client-secret GOOGLE_CLIENT_SECRET \
    --google-drive-user-api-rate-quota 5000

Storage Gateway Created: 7187a9a0-68e4-48ea-b3b9-7fd06630f8ab

This was successful and outputs the ID of the new storage gateway ( 7187a9a0-68e4-48ea-b3b9-7fd06630f8ab in this case) for our reference. Note that this will always be a unique value if you run the command. If you forget the id of a storage gateway, you can always use the command globus-connect-server storage-gateway list to get a list of the storage gateways on the endpoint.

You can also add other policies to configure additional identity mapping and path restriction policies as described in the Globus Connect Server v5 Data Access Guide.

Note that this creates the storage gateway, but does not yet make it accessible via Globus and HTTPS. You’ll need to follow the steps in the next section.

Collection

A Google Drive Collection is created with the command globus-connect-server collection create, and can be updated with the command globus-connect-server collection update.

As the Google Drive Connector does not introduce any policies beyond those used by the base collection type, you can follow the sequence in the Collections Section of the Globus Connect Server v5 Data Access Guide. Recall however, that the paths are interpreted as described above in Google Drive Connector Virtual Filesystem.

User Credential

As mentioned in above, access to mapped collections on a Google Drive require users to register credentials. These credentials are created by performing an authentication flow with Google. This is initiated by visiting the Credentials tab of the collection. The user is directed to that page when they first attempt to access that collection.

Appendix A: Limitations

Google Document Types

The Google Drive connector is not intended to transfer Google document types (doc, sheet, slides, etc) between different Google accounts or outside of Google Drive. The content of Google documents are never downloaded or uploaded. When downloading a Google document, the data that gets created on the destination endpoint is simply a metadata file that contains the Google document id. When uploading, a Google document is created which is a copy of that id, but this requires that your Google account already has permission to access the original document id.

Transfers of Google document metadata back into Google Drive do not support checksum verification, and will fail if verification is enabled for that task.

Google Shortcuts

Google Shortcuts are not currently supported, and attempting to transfer a Google Shortcut will fail.

Appendix B: Document Types for the Google Drive Connector

GoogleDriveStoragePolicies Document

Connector-specific storage gateway policies for the Google Drive connector

Name

Type

Description

DATA_TYPE

string google_drive_storage_policies#1.0.0

Type of this document

auth_callback

string

URL of the auth callback that must be registered on the Google API console for the application client_id in order to process Google credentials.

client_id

string

Client ID registered with the Google Application console to access Google Drive. [Private]

secret

string

Secret created to access access Google Drive with the client_id in this policy. [Private]

user_api_rate_quota

integer

User API Rate quota associated with this client ID[Private]

 
{
  "DATA_TYPE": "google_drive_storage_policies#1.0.0",
  "auth_callback": "string",
  "client_id": "string",
  "secret": "string",
  "user_api_rate_quota": 0
}

GoogleDriveUserCredentialPolicies Document

Connector-specific user credential policies for the Google Drive connector

Name

Type

Description

DATA_TYPE

string google_drive_user_credential_policies#1.0.0

Type of this document

access_token

string

OAuth access token[Private]

email

string

OAuth email claim

refresh_token

string

OAuth refresh token[Private]

scopes

array of string

OAuth scopes associated with this access token

sub

string

OAuth subject identifier claim

token_expiry

string <date-time>

OAuth access token expiration time

 
{
  "DATA_TYPE": "google_drive_user_credential_policies#1.0.0",
  "access_token": "string",
  "email": "string",
  "refresh_token": "string",
  "scopes": [
    "email",
    "profile",
    "https://www.googleapis.com/auth/drive",
    "https://www.googleapis.com/auth/drive.appfolder"
  ],
  "sub": "string",
  "token_expiry": "2019-08-24T14:15:22Z"
}