Google Cloud Storage Connector
- Endpoint Installation
- Registration of endpoint with Google
- Creating a Storage Gateway using the Google Cloud Storage Connector
- Creating a collection via a Google Cloud Storage storage gateway
Last Updated: November 4, 2019
The Google Cloud Storage connector allows Globus Connect Server to access Google Cloud Storage buckets associated with Google accounts. Access to Google Cloud Storage through the Google Cloud Storage connector is facilitated by the creation of Google Cloud Storage storage gateways on an endpoint. The Google Cloud Storage connector is available as an add-on subscription to organizations with a Globus Standard subscription - please contact us for pricing.
This document describes how to install and configure the Google Cloud Storage connector as well as create Google Cloud Storage storage gateways. After the installation is complete, any authorized user can establish a connection to their Google Cloud Storage buckets by following the steps in this How To in order to create a collection using a configured Google Cloud Storage storage gateway on the endpoint.
The installation must be done by a system administrator, and has the following distinct set of steps:
Installation of the packages needed for Globus Connect Server version 5 endpoint and the Google Cloud Storage connector, as well as creation of the endpoint itself.
Registration of the endpoint with Google to obtain credentials for the endpoint to securely use the Google Cloud Storage APIs for accessing data.
Create a storage gateway on the endpoint configured to use the Google Cloud Storage connector and the credentials from Google.
Please contact us at email@example.com if you have questions or need help with installation and use of the Google Cloud Storage connector.
The Google Cloud Storage Connector requires a functional Globus Connect Server 5 endpoint in order to be used. Instructions for installing and configuring and endpoint using Globus Connect Server 5 can be found here. The rest of this document assumes that a functional Globus Connect Server 5 endpoint is being used when attempting to configure the Google Cloud Storage Connector.
Install the package globus-gridftp-server-google-cloud from the Globus repository.
For RedHat-based systems:
yum install globus-gridftp-server-google-cloud
For Debian-based systems:
apt-get install globus-gridftp-server-google-cloud
Registration of endpoint with Google
The Globus Connect Server v5 endpoint needs to be registered as an application with Google so that users can authorize the endpoint to access Google Cloud Storage or Google Drive on their behalf. The following steps describe how the endpoint can be registered as a Google OAuth client to obtain a client id and secret from Google.
It is necessary that these steps be performed on a fully functional Globus Connect Server 5 endpoint, as discussed above.
You will need a Google account to complete these steps, and the registration will be stored under that Google account. This account is only for registration of the application and has no bearing on Google accounts that will be allowed to use this endpoint to access data. An administrator may use an existing Google account.
To register the endpoint with Google, go to the Google Developer Console
If you have never created a project with Google, you will be prompted to create one. If you create a project, you do not have to change the default permissions for the project when given the option to do so.
After you have created or selected a project, you will use the Google API Dashboard to enable APIs, configure the OAuth consent screen, and create credentials for use with your endpoint.
Enable this project to use the APIs required to interact with Google Cloud Storage and Google Drive. Select the "Libary" menu, and repeat the following steps for these API names: Cloud Storage, Google Cloud Storage JSON API, Cloud Resource Manager API, and Google Drive API
Search for the API name and select the matching result.
Once on the API page, select "Enable".
Select the "OAuth consent screen" menu to configure the OAuth consent screen that will be shown to users.
For the "Application name", enter "Globus Connect Server".
For the "Scopes for Google APIs" section, select "Add Scopes", then select "manually paste", and paste the following scopes before selecting "ADD":
https://www.googleapis.com/auth/drive.appdata https://www.googleapis.com/auth/drive https://www.googleapis.com/auth/cloudplatformprojects.readonly https://www.googleapis.com/auth/devstorage.read_write
For "Authorized domains", add "glob.us" and either your own domain or globus.org.
Other fields are optional.
Select the "Create credentials" button, and then the "OAuth client ID" option.
You will be prompted to select an application type. Choose "Web application" and configure it as follows:
Name: set a descriptive name to be able to identify the registration of this endpoint in your projects on the Google API Manager. For example, the endpoint Display Name can be used for this.
Authorization redirect URIs: set to the value that was given for the "Google OAuth Redirect URL" when the Globus Connect Server 5 endpoint was created, as discussed in the Create Globus Endpoint section of the Globus Connect Server 5 Install Guide. If neccesary, running
globus-connect-server-setupagain will output the URL.
Make note of the client ID and secret you get from Google for this application, as you will need them to configure the storage gateway.
Creating a Storage Gateway using the Google Cloud Storage Connector
To create storage gateways on an endpoint, the
globus-connect-server-config storage-gateway create command is used. For example:
$ sudo globus-connect-server-config storage-gateway create --root "/" --display-name "Google Cloud Storage Storage Gateway" --domain "example.edu" --connector "Google Cloud Storage" --client-secret 13Dsbcsecretl-xnsecret5K3s --client-id 1866039G8774-r255xclientid0i4ho1gik791bcgscxj8.apps.googleusercontent.com Storage Gateway Created: 3b2dc912-af31-4244-82e5-f3818f486a4f
Note that the ID of the new storage gateway is given in the output.
This would create a storage gateway on the endpoint that:
Causes new collections to be rooted at "/" in the Google Cloud Storage Project that users configure their collections to use. The "/" directory contains all buckets in a project.
Allows Globus users with a Globus Account that includes an identity from the Identity Provider that controls the example.edu domain to create collections for any Google Cloud Storage project associated with their example.edu identity.
Uses the "Google Cloud Storage" storage connector.
Has a display name of "Google Cloud Storage Storage Gateway".
Uses the Google app with client ID = “1866039G8774-r255xclientid0i4ho1gik791bcgscxj8.apps.googleusercontent.com”.
Uses the “13Dsbcsecretl-xnsecret5K3s” client secret to communicate with the Google app specified above.
The globus-connect-server-config storage-gateway create command supports the following options for storage gateways configured to use the Google Cloud Storage connector, in addition to the common options supported for all storage connectors:
Identities from this domain are allowed to use the storage gateway to create collections for the Google Cloud Storage projects associated with this identity. For example, if this value was set to
example.edu, then a Globus user would need to have logged into Globus with a Globus Account that included an
example.edu identity to be able to create collections using this storage gateway. A Globus user that did have an
example.edu identity in their Globus Account would be able to use this storage gateway to create collections for any Google Cloud Storage projects associated with their
The Client ID of the Google app that the storage gateway has been configured to use.
The Client Secret of the Google app that the storage gateway has been configured to use.
Google Cloud Storage project to restrict access to. To allow multiple projects, pass this option multiple times. Defaults to all accessible projects.
Google Cloud Storage bucket to restrict access to. To use multiple buckets, pass this option multiple times. Defaults to all accessible buckets in the project if none specified.
/etc/globus/globus-gridftp-server-google-cloud.conf. For more advanced bucket controls, users should create or manage buckets via the Google Cloud Storage console.
Creating a collection via a Google Cloud Storage storage gateway
Once a Google Cloud Storage storage gateway has been configured on the endpoint, permitted users can then create collections using the storage gateway. These collections allow permitted Globus users access to any Google Cloud Storage projects that are accessible by the user that created the collection. The process of creating a new collection using a storage gateway configured to use the Google Cloud Storage connector is found here. Please refer to the Globus Connect Server install document for the various options available in the tool to manage storage gateways.