Globus Connect Server v5.4 Changes

Add support for S3 Requester Pays.

Improvements to API documentation generation.

S3: Update AWS bucket region discovery to better support cross-account access.

The GCS CLI can sort the output of node list by IP address. See the --sort option in the CLI reference.

Fix collection deletion errors on multi-node endpoints.

Box: retry rate-limit errors on file downloads.

S3: storage gateway --bucket entries will always show in the root listing.

Allow mapped collection admins to delete related guest collections

Improved responses when using --include-private-policies for gateways

Collection HTTPS API replies triggered with bearer tokens

Fixed v4 migration of endpoint roles.

The Globus OIDC service starts only when configured.

Added API error code replies for collection HTTPS interfaces.

Improve error handling in migration tools when file permissions are incorrect.

Error with Https access to collections when DocRoot is unreadable

Node cleanup on Debian and Ubuntu leaves apache configuration enabled

Allow DNS pattern validation for TLDs longer than 6 characters.

Improve visibility rules of root_path on guest collections.

globus-connect-server collection set-owner-string fails for unmanaged endpoint

Improve handling of OAuth credentials for connectors.

Increase user_message limit to 256 characters

Add support for guest collection authentication policies

Add delete protection option for mapped collections.

OneDrive: Fix issue listing /Shared libraries after recent MS API changes.

OneDrive: Increase maximum upload size from 100GB to 250GB.

Fix v4 migration issue when an endpoint is configured to use only gridmap authorization.

Fix problem creating an OIDC server if the domain was previously associated with the email identity provider.

Fix v4 migration error "TypeError: …​ are not of the same version".

Fix v4 migration handling when 'gcs-migration update' is run on a node which does not have GCS installed.

Fix v4 migration of shares setup using proxy certificates.

Fix 'node setup' error "ValueError: not enough values to unpack (expected 2, got 1)".

Fix 'endpoint upgrade' error "AttributeError: 'NoneType' object has no attribute 'domain'".

Add support for migrating a v4 endpoint that uses the Ceph or HPSS premium storage connectors.

The GridFTP process can be configured to accept control channel connections on a port other than TCP port 443. See Changing defaults in the Globus Connect Server v5 Installation Guide for details.

The Box connector now supports OAuth 2 user authentication as an alternative to the existing Enterprise authentication. See the documentation for configuration details.

Fix v4 migration error "Unsupported update to gsi-authz.conf".

Fix v4 migration update with mismatched plan versions.

Fix v4 migration 's3-credential create' when identity@domain does not exist.

Fix gridftp connection error after package update.

Snapshots can omit secrets so that they can be shared with support.

Fix migration of v4 endpoints with multiple shares owned by the same user.

Add support for migrating v4 endpoints that use the BlackPearl premimum storage connector.

Improve GCSv5 HTTPS collection access for large-latency deployments.

On RPM-based systems, fix an issue where the Globus Connect Server services become disabled on package upgrade.

Improve diagnostics when globus-connect-server endpoint migrate4 encounters a problem with the migration plan.

Add support for provisioning identities when configuring a myproxy idenitty mapping. This can be used when migrating from myproxy or myproxy OAuth to the Globus OIDC service.

Add new commands to allow endpoint administrators to create, list, and delete user credentials.

Add support for migrating a v4 endpoint that uses the S3 premium storage connector.

Migration: Added support for migrating shares with anonymous permissions.

Fix HTTP downloading of files with special characters in the name.

Migration: It is now possible to migrate v4 POSIX endpoints with more than 100 shares.

Migration: Migration keywords are now removed from collections when using the --finalize option.

Migration: Moved logging location of globus-connect-server endpoint migrate4 to /var/log/globus-connect-server/gcs-manager/migrate4.log.

Improved response time when changing inheritted attributes on mapped collections with a large number of shares.

Allow ~ or $HOME in guest collection base paths if mapped collection base path is /.

Migration: Updates to the migration plan can no longer be applied to the v5 endpoint once globus-connect-server endpoint migrate4 --finalize has been run.

Migration: Fix for recovering from previous failed runs of globus-connect-server endpoint migrate4.

Migration: Fix for force_encryption inheritence.

Migration: Fix for grid map identity mapping.

Improved error messages for GCS CLI login for multiple failure types.

Migration: Fix conflicts when migrating ACLs and roles on guest collections

Migration: Fix loading sharing state files when local account does not exist

Migration: Fix validation of set-guest-collection-owner -g <USERNAME>

Fix error messages for CLI options that take prefixes (ex. file:)

Fix for globus-connect-server node update --ip-address <ip>

Files viewed in a browser using a collection’s HTTP interface are no longer forced to download as an attachment

Errors encountered when attempting to transfer Google Drive shortcuts now clearly indicate that shortcuts are not supported, and may be skipped via Transfer options.

Fix possible failures when transferring small files to Google Drive on Ubuntu 20.04.

Prevent adding too many keys to an endpoint’s keychain.

Fix globus-connect-server oidc-register when not given any additional domains.

Fix mixed-case mapping issues with the Box connector

Improve share state file processing in the v4 migration tools.

Allow syslog_t to read GCS logs when SELinux is enforcing.

Improve behavior synchronizing state when badly behaved clients access the service.

Fix error running globus-connect-server oidc create

Handle existing roles which aren’t stored in the GCS configuration in globus-connect-server endpoint migrate4

Fix handling OIDC credential refresh failures so that the web application can direct users to the credentials page.

Improve error handling in https uploads so that write failures are returned in the HTTPS status code.

Improve startup time of the globus-connect-server command-line tool.

Add support for HPSS connector

Add support to set the advertised owner string in the Transfer endpoints used by GCS endpoints and collections. This makes it easier for users to search for relevant endpoints.

Add support for migrating GCSv4 endpoints to GCSv5.4 with (up to 100) guest collections.

Checksum verification performance improvements for the S3, Google Cloud, Ceph, and Azure connectors.

Node setup starts services in the correct order, so apache doesn’t need to be restarted after completion for things to work.

Fix error syncing changes to the data_interface of a node

Improve error handling when running node setup or node cleanup after deleting an endpoint.

Fix issues with timeout and retry behavior in the globus-connect-server CLI

Fix authorization failure when accessing collection via HTTPS when using Google as identity provider.

Fix error when applications use client credential to interact with the GCS Manager API.

Fix conflict error when an object is changed rapidly or when multiple collections are created simultaneously.

Fix memory leak in HTTPS server when the client requests a partial download or resumes a download.

Fix HTTPS server incorrectly sends multipart response when a single byte range is returned.

Fix issues in the GCS v4 migration tools

Fix command-line parsing issues in the oidc-register command.

Google Drive: Fix GD-Parse-Error faults possible during path resolution.

Box: Fix for large uploads failing due to token expiration.

Box: Clarify error when attempting to replace a file without Editor permission.

Added manual GridFTP configuration to enable basic listings for specific paths.

Azure: Fix for large uploads failing due to token expiration.

Box: Fix for failed uploads on platforms with curl-2.66+.

Fix for bad client credentials passed to 'node setup'.

New 'oidc register' command enables registration of your own OIDC server for endpoint authorization.

Fix for 'collection show' on private collections

OneDrive: Allow writing to subfolders under /Shared.

OneDrive: Improved account mapping.

Fix for HTTP public ACL downloads.

Minor improvemnts to the GCS CLI.

Fix handling of custom domains when deleting collections.

Box: Fix intermittent login failures.

Google Drive: Fix accessing /My Drive when Google profile language was set to non-english.

Several small fixes for migrations from v4 and v5.3.

Azure: fix for duplicate directory entries in listings.

OneDrive, Azure, Google Drive, Google Cloud Storage: Fix credential registration when identity mapping produces a username with uppercase characters.

Fix for v4 migrations

Endpoint setup now requires contact email. Mapped collections inherit this setting.

Roles creation supports identities that have not yet logged into Globus.

Fix for collection role creation.

OneDrive: Fix broken "/Shared libraries" listings

Fix for collection custom domains.

Google Cloud Storage: Support for admin-configured service account credentials

Improvements in certificate renewal handling

Custom domain names must be valid DNS domains

Audit log parser fix for duplicate stat operation error records

Box: Improve failure handling when calculating MD5

S3: Fix for listing collections rooted at a directory created by other tools

Ceph: Fix for listings timing out due to empty NextMarker

Posix Staging Connector: Fix a regression in the Posix Staging connector which caused credential registration and access to collections to fail.

mod-auth-openidc: Update module to fix an open redirect issue.

Added support for the Azure Blob connector.

Set high_assurance flag only when appropriate in AuthenticationTimeout error details.

Standard mapped collections can now be configured to disallow anonymous write permissions on guest collections.

OneDrive Connector: Fix automatic credential refresh for single tenant configurations.

Box Connector: Improvements to directory listing performance.

Allow @clients.auth.globus.org identities to meet storage gateway policies

Better error diagnostic when providing a bad value to globus-connect-server login

Better error message when creating more collections than are allowed by the Globus transfer service.

Improve error handling when network errors occur.

Fix response code on globus-connect-server collection list for private endpoints without any collections.

Disable the gcs_manager.socket unit on globus-connect-server node cleanup.

Add the status of the gcs_manager.socket unit to the globus-connect-server self-diagnostics output.

The CLI token storage required additional protection in some situations.

When creating a mapped collection, if --contact-email is not specified, it will default to the email address of the endpoint admin creating the collection.

The CLI lists more than 16 sharing policies.

Sharing policies are deleted when the associated collection is deleted.

High assurance storage gateways can require multi factor authentication for access to collections.

Collections can force checksum verification or disable it altogether.

Added ActiveScale and OneDrive connectors.

Now enforcing storage-gateway --restrict-paths policy on guest collections with --sharing-restrict-paths policy set.

Improved the performance when checking supplemental groups for POSIX authorization checks.

Fix for migration of mapped collections from v5.3 endpoints.

Fix for selinux installation on CentOS.

Fix for log file permissions after 'node cleanup'.

Fix for syncing --sharing-restrict-paths between nodes.

Fix for timeouts during OIDC authentication.

Cleanup GridFTP processes that hang during authentication.

When accessing a guest collection via its HTTPS interface, clients such as wget and curl can be used without special HTTP headers if the access is allowed with an anonymous ACL.

Transfers would fail after upgrading to previous versions if the GridFTP data interface was never explicitly set.

Collection HTTPS interface downloads support the HTTP header 'Range: bytes=0-' syntax.

Some endpoints were previous unable to delete a collection’s user_message and user_message_link attributes.

Fix an error deleting a Google Cloud Storage Gateway that was introduced in the 5.4.17 release.

The sharing_restrict_paths is now the default sharing path restrictions for a mapped collections. Additional permissions to share paths can be added to individual users. See the Data Access Guide and command line reference documentation for information on how to configure sharing policies.

If a Google Drive or Google Cloud Storage user credential is either revoked due to an explicit removal of consent, or invalidated because it hasn’t been used for a long period, Globus Connect Server v5 now correctly marks the credential as invalid. When this occurs, users who access a collection that uses the credential will be redirected to the credential page to repair the credential.

When creating a guest collection, if the endpoint is not managed by a subscription, always return a missing subscription error. In earlier versions, other errors (such as OAuth missing scopes) could be returned, even though fixing that situation would in no way allow the collection creation to succeed.

Add a warning and require confirmation when cleaning up an endpoint that is acting as an OIDC server. When the endpoint is cleaned up, the OIDC server is no longer functional and the identity domain can not be reused.

When renewing the endpoint’s certificate, the new private key is not rewritten to the local filesystem, which can cause https errors. Contact support@globus.org for information about how to fix this situation.

The implementation of the force_encryption policy has changed for guest collections. When creating a guest collection, it is associated with another mapped collection. In previous versions, the force_encryption policy was completely independent for the collections. In this version, when you set force_encryption to true on a mapped collection, then all guest collections associated with that mapped collection will also have that policy set to true and it cannot be set to false. If a mapped collection has force_encryption set to false, the guest collection’s value of that property can be set. In either case, if the storage gateway is high assurance, then the force_encryption policy is always true for mapped and guest collections.

The relationship between guest and mapped collections is now represented in the Globus transfer service. As a result, pause rules that are set on a mapped collection will also pause transfers on the related guest collections.

If Globus Connect Server v5 is installed on a system with a FIPS-compliant cryptography module it would be unable to create new collection, as Globus Connect Server v5 used md5 to generate pseudo-random domain names.

If globus-connect-server endpoint setup is run a second time with either a different or missing deployment key file, it would fail with an error Object WILDCARD_DOMAIN already exists.. The incorrect deployment key is now detected and the user is instructed how to proceed.

Fix a crash in globus-connect-server endpoint migrate53 after migrating configuration.

Restrict roles to non-email identities. In previous versions, it was possible to configure Globus Connect Server to assign an email-only identity as either an endpoint owner or another role. However, these roles could not be used to manage the endpoint, as they do not provide authentication information. If a user attempts to create a role for an email-only identity it is now rejected.

Normalize Google Cloud Platform user names. In some cases, Google Cloud platform usernames or email addresses may contain contain mixed-case values. Globus Connect Server would report mismatched username errors when the Google ID token contained uppercase characters.

Handle Ceph usernames that contain reserved characters. If a Ceph username contained a reserved character such as "=", it would be unusable with Globus Connect Server.

Update the Node configuration to allow multiple IP addresses at setup time, as well as allowing the incoming and outgoing port ranges to be changed on a per-node basis.

Fix a bug in the letsencrypt certificate renewal, where the certificate was not being serialized to disk correctly after renewals.

Remove duplicate entries from --help output.

Improve validation of node port ranges.

Improve error messages when attempting to log in to a collection using an invalid collection id.

Collections migrated using the globus-connect-server endpoint migrate53 command did not have their scopes updated as part of the migration process, so the https interface did not work correctly. (As a workaround, change the display_name of a collection to cause the scope values to be fixed.)

Support for admin-supplied domains and certificates for Globus Connect Server services. See the new Globus Connect Server domain guide.

Add an optional OIDC service that integrates with the local PAM modules to act as an identity provider. This allows sites with no OIDC compatible identity service to use their local login policies to generate tokens for use with Globus Connect Server. See the OIDC guide.

New commands to support these features:

Allow identities with the status "private" to be granted roles. Previous versions would disallow those identities.

Fix directory permissions after migrating a v5.3 endpoint using the globus-connect-server endpoint migrate53 command.

Don’t allow attempt to reload the apache configuration until after configuration synchronization is complete.

Ensure that a Unicode-compatible locale is used in wsgi configuration on Debian-based systems.

When a Google Cloud Platform credential fails to refresh or consent is revoked, mark the credential as invalid so that the web interface can prompt the user to repair the credentials.

Add user and group sharing restrictions to mapped collections. See the Sharing configuration section of the data access guide for information on the new configuration. If the new policies are not used, then if the sharing_allowed property is set to true, then any user which maps to a valid local account may create shares.

New command-line to set the new sharing restrictions. These are document in