Migrating from Globus Connect Server v5.3

Last Updated: July 24, 2020

1. Introduction

This migration guide explains the process for upgrading a Globus Connect Server v.5.3 endpoint to Globus Connect Server v5.4. After completing the tasks in this guide, your endpoint will have all of its configuration data upgraded to be compatible with Globus Connect Server v5.4 and you will be able to use the new features, such as new globus-connect-server command-line tool with role-based authorization and be able to deploy the endpoint on additional Data Transfer Nodes.

2. Changes in Globus Connect Server v5.4

Globus Connect Server v5.4 contains changes to the representation of the endpoint and its configuration. This migration tool converts the endpoint to the new format. Depending on which features are used in the endpoint, this may result in the creation of additional storage gateways, mapped collections, and roles to provide equivalent functionality to the v5.3 endpoint.

The Appendix describes the main changes in the configuration objects that will occur when an endpoint is migrated to v5.4.

Important

The DNS names used to access collections are not preserved on update. Applications which rely on the DNS name (such as those which use the HTTPS interface to access collections) will need to be updated to use the new DNS name.

Also, the endpoint UUID changes to match the client_id used to create the endpoint. All collection UUIDs remain unchanged.

3. Procedure

3.1. Pause Transfers on the Endpoint (optional)

Visit the Admin console to create a pause rule to suspend transfers on the endpoint during migration. This is optional, as the services on the Data Transfer Node are disabled during migration, but will allow your users to see a reason why their transfers are not working.

3.2. Disable Globus Services

Use systemctl to stop the Globus services.

Disable globus-gridftp-server
% sudo systemctl stop globus-gridftp-server

3.3. Install New Software

Remove the globus-connect-server53 package and install the globus-connect-server54 package. This will not make any changes to your Globus Connect Server configuration. The migration steps in the output are the preparation of a new (unpopulated) configuration database for the v5.4 configuration.

% sudo yum remove globus-connect-server53
% sudo yum install globus-connect-server54
% sudo apt remove globus-connect-server53
% sudo apt install globus-connect-server54

3.4. Migrate the Endpoint

The globus-connect-server endpoint migrate53 command converts the configuration of the local v5.3 endpoint into the format used by Globus Connect Server v5.4 software.

This command loads the existing v5.3 configuration, converts it to the format used with v5.4, and then deploys the new configuration on the current node. In some sense, it’s a combination of both the globus-connect-server endpoint setup and the globus-connect-server node setup commands.

Important

Since this command will be interacting with the local databases, deploying service configurations, and starting services, this command must be run as the root user.

This command has a few command-line options:

--deployment-key PATH

The path to output the deployment key. This key is needed to deploy this endpoint on additional Data Transfer Nodes and to clean up the endpoint cleanly. If not specified, the default value of deployment-key.json is used. If the file already exists, it will be parsed and used.

--owner USERNAME

The name of the Globus user who will act as the owner of the endpoint. This user must have an active Globus account, and will be granted all privileges to edit and delete endpoint configuration. If the endpoint is managed by a subscription, this user will also be granted an endpoint administrator role in the transfer service for the endpoint.

--ip-address IP_ADDRESS

The public IP address of the current node. The globus-connect-server command will attempt to determine this automatically, but if it is not able to do so, it must be specified using this option.

Example 1. Migrate the Endpoint

For an example, will set the administrator to be admin@example.edu and use the defaults for the other command-line options.

% sudo globus-connect-server endpoint migrate53 --owner admin@example.edu
Preparing
  [#########################]  100%
Setting up endpoint
  [#########################]  100%
Migrating configuration
  [#########################]  100%
As part of this migration, this script created 1 mapped collection(s)
to support changes in the collection model for v5.4. To help your
users find the collections on your endpoint, please visit the
following URLs to update the metadata and names of those collections
https://app.globus.org/file-manager/collections/aa0ac4f8-f7d0-43c8-988e-394520d4493e/overview/edit
Configuring services
  [####################################]  100%
Starting services
  [####################################]  100%
Migrated endpoint 9a4c234d-c84e-4820-9aea-1d9991aa308b. You'll need
the following information to deploy additional nodes, as well as the
contents of deployment-key.json:
	client_id: "3113dd2a-6199-4c3e-b08f-a4ac4b5ae5c3"
	secret: iefeuz0Twaequ6UbSeid6tieaefe8IexLaiL8uxoXZa=

This command outputs status information as it creates objects in the database and deploys the configuration to the local machine. It also writes a deployment key file, which can be used to deploy this endpoint configuration on additional Data Transfer Nodes, as described in Add Data Transfer Nodes to the endpoint.

3.5. Update collections

Globus Connect Server v5.4 has some model changes from v5.3 that affect how collections are created. All connectors now support standard mapped collections. Also, all guest collections are created with a relationship to a mapped collection.

If a v5.3 endpoint is migrated with existing guest collections, the migration script creates mapped collections with temporary names to satisfy this new relationship between guest and mapped collections.

These new mapped collections have display names like Migration 9a4c234d-c84e-4820-9aea-1d9991aa308b 0 and no other metadata such as a description or keywords. These mapped collections are what your users need to find to access data on your storage and to create guest collections to share data. So the mapped collection name and definitions should be updated to give them meaningful names and metadata.

These collections can be updated by either following the links provided in the migration output, or by using the globus-connect-server collection update command.

To update these collections, you must have access to Globus Account indicated as the owner during endpoint migration, or (if this is a managed endpoint) have the owner grant you a role to be an administrator on the endpoint or the new collections.

3.6. Validate configuration

You can now use the globus-connect-server command to view the configuration of the endpoint to verify that things are configured as expected.

3.7. Disable Pause Rules (optional)

If you created a pause rule in Pause Transfers on the Endpoint (optional), you can now disable that pause rule to re-enable access to your endpoint.

3.8. Deploy the Endpoint on Additional Data Transfer Nodes

If you want to use multiple Data Transfer Nodes, you can use the globus-connect-server node setup command on those nodes, using the deployment-key file and the client_id and secret values printed at the end of the migration. An example of how to add Data Transfer Nodes is in the installation guide.

Appendix A: Storage Gateway Policy Changes

allow_guest_collections

The allow_guest_collections property of the storage gateway is moved to mapped collection configuration.

allow_mapped_collections

The allow_mapped_colections property of the storage gateway is removed. All connectors now support mapped collections.

root

The root property of the storage gateway is removed. For a mapped collection, this can be set as the collection_base_path property of the mapped collection.

identity_mappings

The identity_mappings property of the storage gateway is used in place of the gridmap property of a mapped collection.

A.1. Mapped Collection Policy Changes

gridmap

The gridmap property of a mapped collection is removed, with its contents moved to the identity_mappings property of the corresponding storage gateway. Since this was a collection-specific property in v5.3, a new storage gateway is created in the v5.4 migration for each collection that has a non-empty gridmap file, with the contents of the file translated into the identity_mappings property of that storage gateway.

path

The path property of a mapped collection is replaced with the collection_base_path property, which is a concatenation of the v5.3 storage gateway root and collection path properties.

allow_guest_collections

The allow_guest_collections property of the storage gateway is moved to the mapped_collection object, since Globus Connect Server v5.4 guest collections are always created relative to a mapped collection.

A.2. Guest Collection Policy Changes

mapped_collection_id

Every guest collection is now related directly to a mapped collection. This allows us to add features (such as the ability to select the base path of a guest collection interactively) to the Globus web application. During migration, a mapped collection for each storage gateway that has any guest collections configured on it is created.

path

The path property of a guest collection is replaced with the root property, which is a concatenation of the collection_base_path of the mapped collection and the guest collection.

A.3. Other configuration data

Any updates to the collection or endpoint metadata done via the Globus web app prior to migrating to Globus Connect Server v5.4 are preserved during migration.

The Let’s Encrypt account used in v5.3 is used with the updated v5.4 endpoint. Globus Connect Server v5.4 uses its own client to interact with Let’s Encrypt in order to work better with a deployment across multiple Data Transfer Nodes.

The DNS name of the Globus Connect Server Manager API is preserved on upgrades. The DNS names of collections are not.

4. Globus Help Resources

4.1. Documentation Website

This website (docs.globus.org) contains a wealth of information about configuring and using the Globus service. Many common issues can be resolved quickly by browsing our frequently asked questions and reading the relevant guides and how-to’s. We recommend consulting these resources first when looking for fast resolution to any issue you are having with the Globus service.

4.2. Mailing Lists

If you use Globus, then participating in one or more of the public email lists is an excellent way to keep in touch with your peers in the Globus Community. For questions about managing your Globus deployment, e.g. installing software for a Globus endpoint, configuring your firewall, and integrating your institution’s identity system, subscribe to the admin list. For other inquiries and discussions, try the user or developer lists. For more information on mailing lists and how to subscribe, click here.

4.3. Globus Support

Questions or issues that pertain to {gcsv5} installation or to any client or service that is used in the Globus software-as-a-service (SaaS) or platform-as-a-service (PaaS) offering can be directed to the Globus support team by submitting a ticket. Subscriptions include a guaranteed support service level.

When submitting a ticket for an issue with Globus Connect Server, please include the endpoint name, a description of your issue, and screenshot/text dumps of any errors you are seeing. Please also include the output of Globus Connect Server’s self-diagnostic command, run as root, from the server hosting the endpoint:

globus-connect-server self-diagnostic