Globus Connect Server v5.4 Changes

New GCS CLI command collection check enables admins to locate invalid collections

New GCS CLI command node new-secret for rotating per-node authentication credentials

New identity mapper validation tool /usr/sbin/globus-idm-validator

New GCS CLI option self-diagnostic --collection to list info about a collection and storage gateways

Google Drive: (Beta) Support for exporting Google Documents. See Google Exports.

POSIX: Support for CRC32C checksums

Improve logging of account mapping

Fix advertised owner missing from new collections

Fix GCS CLI session consent example command output

Fix HTTPS Collection authentication for multiple authenticated matches

Collection API routes return 404 when collection does not exist

S3 credential PATCH disallows > 128 credentials

Google Drive: Improved error handling; storageQuotaExceeded retries can be disabled, and cannotDownloadAbusiveFile errors can be skipped.

Box: Dropped legacy enterprise mode dependency. The package cjose (rpm) or libcjose0 (deb) can be removed.

Posix: Fixed failed listings when unable to stat all entries of a directory.

Box: Fixed upload failures when the completion response is delayed, and improved large upload faults.

Add HTTP_PROXY support for GCS

Support for high-assurance-only transfers between collections

Allow file input for --user-allow, --user-deny, --posix-group-allow, --posix-group-deny, --sharing-user-allow, --sharing-user-deny, --posix-sharing-group-allow, --posix-sharing-group-deny, --posix-staging-sharing-group-allow and --posix-staging-sharing-group-deny

S3 multi key user credential default key is now obvious in user-credentials show

Fix for S3 multi key user credential path matching

Fix for S3 multi key user credential key deletion on update

Fix for IPv6 address ambiguity in self-diagnostics

Fix for HTTPS upload or download on guest collections with an Auth policy applied

Fix for GCS version rollback on Ubuntu and Debian

Added an OpenIDC authentication handler to the mod-globus Apache module, which replaces the mod_auth_openidc Apache module. The mod_auth_openidc package can now be removed. This change is expected to resolve some Apache crashing issues.

Added the ability to associate multiple S3 key pairs with specific bucket and object paths in the S3 connector. See multiple S3 keys.

Removed libraries from the GridFTP server that are no longer used.

Fixed issues accessing shared drives via alternate paths in the Google Drive connector.

Prevent a truncated transfer from being reported as successful in the S3 connector.

Fixed the preservation of modification time in the Google Cloud Storage connector.

Improved subscription error messages

Updated "endpoint setup" subscription message

GCS CLI fix for "session update" suggestions after an authentication timeout

Fix for updating scopes when storage gateway allowed domains change

Modify SELinux policies for NoNewPrivileges

Update GARE responses to use IDs in session_required_idenities

Fixed validation of gateway network_use on unmanaged endpoints

Add support for network usage settings on storage gateways.

Add support for permission expiration set on guest collections.

Improve self-diagnostics network interface reporting.

Fix usages statements for the GCS CLI endpoint key convert subcommand.

Fix vhost configuration conflict with perfSONAR.

Fix the GCS CLI endpoint setup subcommand’s progress meter messages.

Fix for guest collection admins retrieving private policies.

Add support for Ubuntu Noble and Fedora 40.

Handle updated api response during subscription verification.

Fix some instances of httpd crashing.

Fix possible GridFTP log corruption when using the undocumented stdio_ng module.

Add support for guest collection permission expiration.

Fix for GCS cli to prevent preferred > max custom network settings

Fix missing shlib dependencies in globus-python

Improved error message when setting subscriptions when not within the authentication timeout

Fix for credentials listing with multiple local account mappings

Fix for database detached and error processing

Fix for Debian and Ubuntu LICENSE.txt install

Add support for self-managed subscriptions

Globus OIDC fix for cookie-related authentication issues

Improve paginated collection/storage gateway list queries

Improve error message for incomplete network settings

Improve endpoint setup project creation

Update GCS CLI to use the latest SDK’s Auth policy functions

Improve errors listing roles for non-existent or deleted collections

Add details to more forbidden exceptions

Globus OIDC logs authentications to INFO level

Fix HTTPS download and upload for HA mapped collections.

Add support for Ubuntu Mantic and Fedora 39.

Add support for changing a mapped collection owner.

Improve collection creation when multiple identity mappings are possible.

Updated mod_auth_openidc to the latest upstream release.

Update OneDrive connector to use preferred metadata checksums.

Various migration improvements.

Fix for guest collections owned by unlinked OIDC identity.

Fix for Apache crashes during logrotation on el9.

Remove confusing log output after successful GridFTP connections.

Google Drive connector:

Support accessing GCS services on IPv6-only endpoints

Add support for the Dropbox connector

Mapped collections are delete protected by default

v4 endpoint deprecation notices are no longer migrated to v5 collections

Support credential creation with first mapping on initial access to POSIX collections

Improve GridFTP server shutdown

Improve messaging when the GCS Manager fails to start if the node is misconfigured

Improve error when unable to list buckets in the S3, Ceph, and Google Cloud Storage connectors

Added GridFTP logging when collection certificates are not accessible

Fix for endpoint setup AssertionError

Fix for collection HTTPS downloads

Fix for migration of GCSv4 HPSS storage policies

Fix migration of HPSS default values

Fix for gcs-migration grammar

Fix globus oidc create with bad contact email

The GCS CLI’s endpoint setup and cleanup commands have been updated to support non-interactive use.

Added dual stack support to the S3 connector.

Improved retry mechanics in GCS v4 migrations.

$HOME and ~ allowed in sharing_restrict_paths when base_path is /.

self-diagnostics now includes a network connectivity check for Globus cloud services.

Client identities can now be assigned the endpoint owner role.

Endpoint delete will now remove the endpoint’s client registration on the developers page.

Improved response time for collection listings.

Added --page-size to GCS CLI listing commands.

endpoint key convert can be used to update old-style deployment keys.

Collection listings can be filtered by storage gateway ID.

Mapped collection admins can view guest collections roles.

Fix for globus-connect-server collection list -F list.

Fix when accessing a guest collection whose owner identity is no longer the gateway’s allowed domains.

Improve reporting for last collection access. The endpoint now contains info about when it started adding *last_access` records to collections. The CLI uses that to help determine if a collection has not been accessed at all or if it hasn’t been accessed since the endpoint began recording that info.

Migration improvement in identity caching.

GCS local database can now be created with 'node setup'.

HPSS connector LoginName is now configurable.

HPSS connector caching improvements.

HPSS connector error reporting improvements.

GCS assistant process startup delayed until networking is available.

Fix for collection deletion.

Fix GCS CLI 'auth-policy update' for resetting included domains.

Migration fix for missing local user while listing sharing state.

Migration fix for missing gridmap entry for guest collection owner.

Make project id optional on google cloud collections

Add oauth connector storage gateway policy "allow_any_account"

Add web template for completion of oauth credential generation via cli

Improve error handling for deployment key mismatch in node setup

Improve CLI error when a service is unreachable to distinguish Globus Auth from GCS Manager API.

Add more checks to self-diagnostic to detect SELinux and service account issues.

Update SELinux policy to fix issues starting Globus OIDC service on Red Hat 9 and derivatives.

Fix locale-related errors installing on el9

Fix GCS-migration compatibility issue with el7

Fix "Unbound local error" in gcs-migration identity-mapping cilogon|myproxy

Fix identity mapping errors with external identity map apps.

The OneDrive connector can now create empty (0 byte) files.

Fix for node setup with multiple nodes.

Fix for an RPM install error.

GCS mapped collection administrators can now modify guest collection metadata and delete roles and permissions.

Improve CLI tab completion.

Reduce Auth calls for gridftp connections.

endpoint setup displays project display name alongside project id.

Update GCS collection/endpoint contact email and info fields to make their purpose clear.

Update help message for endpoint setup --owner and --project-admin.

Clarify output when OIDC update run without any options

Improve the endpoint cleanup message when the new-style deployment key is missing so the client-id is missing.

Update endpoint setup error message when deployment key is reused.

Fix for running audit commands.

Fix for multiple local mappings for identities.

The CLI supports tab completion.

globus-connect-server endpoint setup has a new option, --project-admin, that is used to specify the admin username of the Globus Auth project where the new endpoint will be registered.

Collection list no longer silently ignores unauthorized requests for private policies or private collections.

globus-connect-server migrate53 has been removed.

globus-connect-server migrate4 token reuse has been improved.

Migration from GCSv4 supports mixed-case usernames.

Attempting to delete the administrator role associated with the advertised owner is handled gracefully.

globus-connect-server endpoint setup correctly configures administrator roles when the owner is changed on successive runs.

Role creation handles the case when the role exists in Transfer but not in GCS.

Fix Box connector error response handling.

Fix excessive memory usage after certain transfer failures to cloud connectors.

globus-connect-server endpoint setup will now register the endpoint with Globus Auth which eliminates the requirement to visit the Developer’s Console prior to creating an endpoint.

Fix grammar in GCS CLI messages.

Updates to globus-connect-server logout to handle token revocation.

GCS CLI syntax is checked prior to checking if the user is logged in. This is legacy behavior prior to 5.4.60.

Add last_access and created_at to collection

Add API to query based on last-access or created-at

Add guest collection batch delete operation

Improve error message when trying to set network_use without a subscription

Update transfer with information about s3 requester_pays

Update SElinux policies to work with systemd socket activation changes

Add support for S3 Requester Pays.

Improvements to API documentation generation.

S3: Update AWS bucket region discovery to better support cross-account access.

The GCS CLI can sort the output of node list by IP address. See the --sort option in the CLI reference.

Fix collection deletion errors on multi-node endpoints.

Box: retry rate-limit errors on file downloads.

S3: storage gateway --bucket entries will always show in the root listing.

Allow mapped collection admins to delete related guest collections

Improved responses when using --include-private-policies for gateways

Collection HTTPS API replies triggered with bearer tokens

Fixed v4 migration of endpoint roles.

The Globus OIDC service starts only when configured.

Added API error code replies for collection HTTPS interfaces.

Improve error handling in migration tools when file permissions are incorrect.

Error with Https access to collections when DocRoot is unreadable

Node cleanup on Debian and Ubuntu leaves apache configuration enabled

Allow DNS pattern validation for TLDs longer than 6 characters.

Improve visibility rules of root_path on guest collections.

globus-connect-server collection set-owner-string fails for unmanaged endpoint

Improve handling of OAuth credentials for connectors.

Increase user_message limit to 256 characters

Add support for guest collection authentication policies

Add delete protection option for mapped collections.

OneDrive: Fix issue listing /Shared libraries after recent MS API changes.

OneDrive: Increase maximum upload size from 100GB to 250GB.

Fix v4 migration issue when an endpoint is configured to use only gridmap authorization.

Fix problem creating an OIDC server if the domain was previously associated with the email identity provider.

Fix v4 migration error "TypeError: …​ are not of the same version".

Fix v4 migration handling when 'gcs-migration update' is run on a node which does not have GCS installed.

Fix v4 migration of shares setup using proxy certificates.

Fix 'node setup' error "ValueError: not enough values to unpack (expected 2, got 1)".

Fix 'endpoint upgrade' error "AttributeError: 'NoneType' object has no attribute 'domain'".

Add support for migrating a v4 endpoint that uses the Ceph or HPSS premium storage connectors.

The GridFTP process can be configured to accept control channel connections on a port other than TCP port 443. See Changing defaults in the Globus Connect Server v5 Installation Guide for details.

The Box connector now supports OAuth 2 user authentication as an alternative to the existing Enterprise authentication. See the documentation for configuration details.

Fix v4 migration error "Unsupported update to gsi-authz.conf".

Fix v4 migration update with mismatched plan versions.

Fix v4 migration 's3-credential create' when identity@domain does not exist.

Fix gridftp connection error after package update.

Snapshots can omit secrets so that they can be shared with support.

Fix migration of v4 endpoints with multiple shares owned by the same user.

Add support for migrating v4 endpoints that use the BlackPearl premimum storage connector.

Improve GCSv5 HTTPS collection access for large-latency deployments.

On RPM-based systems, fix an issue where the Globus Connect Server services become disabled on package upgrade.

Improve diagnostics when globus-connect-server endpoint migrate4 encounters a problem with the migration plan.

Add support for provisioning identities when configuring a myproxy identity mapping. This can be used when migrating from myproxy or myproxy OAuth to the Globus OIDC service.

Add new commands to allow endpoint administrators to create, list, and delete user credentials.

Add support for migrating a v4 endpoint that uses the S3 premium storage connector.

Migration: Added support for migrating shares with anonymous permissions.

Fix HTTP downloading of files with special characters in the name.

Migration: It is now possible to migrate v4 POSIX endpoints with more than 100 shares.

Migration: Migration keywords are now removed from collections when using the --finalize option.

Migration: Moved logging location of globus-connect-server endpoint migrate4 to /var/log/globus-connect-server/gcs-manager/migrate4.log.

Improved response time when changing inherited attributes on mapped collections with a large number of shares.

Allow ~ or $HOME in guest collection base paths if mapped collection base path is /.

Migration: Updates to the migration plan can no longer be applied to the v5 endpoint once globus-connect-server endpoint migrate4 --finalize has been run.

Migration: Fix for recovering from previous failed runs of globus-connect-server endpoint migrate4.

Migration: Fix for force_encryption inheritance.

Migration: Fix for grid map identity mapping.

Improved error messages for GCS CLI login for multiple failure types.

Migration: Fix conflicts when migrating permissions and roles on guest collections

Migration: Fix loading sharing state files when local account does not exist

Migration: Fix validation of set-guest-collection-owner -g <USERNAME>

Fix error messages for CLI options that take prefixes (ex. file:)

Fix for globus-connect-server node update --ip-address <ip>

Files viewed in a browser using a collection’s HTTP interface are no longer forced to download as an attachment

Errors encountered when attempting to transfer Google Drive shortcuts now clearly indicate that shortcuts are not supported, and may be skipped via Transfer options.

Fix possible failures when transferring small files to Google Drive on Ubuntu 20.04.

Prevent adding too many keys to an endpoint’s keychain.

Fix globus-connect-server oidc-register when not given any additional domains.

Fix mixed-case mapping issues with the Box connector

Improve share state file processing in the v4 migration tools.

Allow syslog_t to read GCS logs when SELinux is enforcing.

Improve behavior synchronizing state when badly behaved clients access the service.

Fix error running globus-connect-server oidc create

Handle existing roles which aren’t stored in the GCS configuration in globus-connect-server endpoint migrate4

Fix handling OIDC credential refresh failures so that the web application can direct users to the credentials page.

Improve error handling in https uploads so that write failures are returned in the HTTPS status code.

Improve startup time of the globus-connect-server command-line tool.

Add support for HPSS connector

Add support to set the advertised owner string in the Transfer endpoints used by GCS endpoints and collections. This makes it easier for users to search for relevant endpoints.

Add support for migrating-to-v5.4/[migrating GCSv4 endpoints to GCSv5.4 with (up to 100) guest collections.

Checksum verification performance improvements for the S3, Google Cloud, Ceph, and Azure connectors.

Node setup starts services in the correct order, so apache doesn’t need to be restarted after completion for things to work.

Fix error syncing changes to the data_interface of a node

Improve error handling when running node setup or node cleanup after deleting an endpoint.

Fix issues with timeout and retry behavior in the globus-connect-server CLI

Fix authorization failure when accessing collection via HTTPS when using Google as identity provider.

Fix error when applications use client credential to interact with the GCS Manager API.

Fix conflict error when an object is changed rapidly or when multiple collections are created simultaneously.

Fix memory leak in HTTPS server when the client requests a partial download or resumes a download.

Fix HTTPS server incorrectly sends multipart response when a single byte range is returned.

Fix issues in the GCS v4 migration tools

Fix command-line parsing issues in the oidc-register command.

Google Drive: Fix GD-Parse-Error faults possible during path resolution.

Box: Fix for large uploads failing due to token expiration.

Box: Clarify error when attempting to replace a file without Editor permission.

Added manual GridFTP configuration to enable basic listings for specific paths.

Azure: Fix for large uploads failing due to token expiration.

Box: Fix for failed uploads on platforms with curl-2.66+.

Fix for bad client credentials passed to 'node setup'.

New 'oidc register' command enables registration of your own OIDC server for endpoint authorization.

Fix for 'collection show' on private collections

OneDrive: Allow writing to subfolders under /Shared.

OneDrive: Improved account mapping.

Fix for HTTP public permission downloads.

Minor improvements to the GCS CLI.

Fix handling of custom domains when deleting collections.

Box: Fix intermittent login failures.

Google Drive: Fix accessing /My Drive when Google profile language was set to non-english.

Several small fixes for migrations from v4 and v5.3.

Azure: fix for duplicate directory entries in listings.

OneDrive, Azure, Google Drive, Google Cloud Storage: Fix credential registration when identity mapping produces a username with uppercase characters.

Fix for v4 migrations

Endpoint setup now requires contact email. Mapped collections inherit this setting.

Roles creation supports identities that have not yet logged into Globus.

Fix for collection role creation.

OneDrive: Fix broken "/Shared libraries" listings

Fix for collection custom domains.

Google Cloud Storage: Support for admin-configured service account credentials

Improvements in certificate renewal handling

Custom domain names must be valid DNS domains

Audit log parser fix for duplicate stat operation error records

Box: Improve failure handling when calculating MD5

S3: Fix for listing collections rooted at a directory created by other tools

Ceph: Fix for listings timing out due to empty NextMarker

Posix Staging Connector: Fix a regression in the Posix Staging connector which caused credential registration and access to collections to fail.

mod-auth-openidc: Update module to fix an open redirect issue.

Added support for the Azure Blob connector.

Set high_assurance flag only when appropriate in AuthenticationTimeout error details.

Standard mapped collections can now be configured to disallow anonymous write permissions on guest collections.

OneDrive Connector: Fix automatic credential refresh for single tenant configurations.

Box Connector: Improvements to directory listing performance.

Allow @clients.auth.globus.org identities to meet storage gateway policies

Better error diagnostic when providing a bad value to globus-connect-server login.

Better error message when creating more collections than are allowed by the Globus transfer service.

Improve error handling when network errors occur.

Fix response code on globus-connect-server collection list for private endpoints without any collections.

Disable the gcs_manager.socket unit on globus-connect-server node cleanup.

Add the status of the gcs_manager.socket unit to the globus-connect-server self-diagnostics output.

The CLI token storage required additional protection in some situations.

When creating a mapped collection, if --contact-email is not specified, it will default to the email address of the endpoint admin creating the collection.

The CLI lists more than 16 sharing policies.

Sharing policies are deleted when the associated collection is deleted.

High assurance storage gateways can require multi factor authentication for access to collections.

Collections can force checksum verification or disable it altogether.

Added ActiveScale and OneDrive connectors.

Now enforcing storage-gateway --restrict-paths policy on guest collections with --sharing-restrict-paths policy set.

Improved the performance when checking supplemental groups for POSIX authorization checks.

Fix for migration of mapped collections from v5.3 endpoints.

Fix for selinux installation on CentOS.

Fix for log file permissions after 'node cleanup'.

Fix for syncing --sharing-restrict-paths between nodes.

Fix for timeouts during OIDC authentication.

Cleanup GridFTP processes that hang during authentication.

When accessing a guest collection via its HTTPS interface, clients such as wget and curl can be used without special HTTP headers if the access is allowed with an anonymous permission.

Transfers would fail after upgrading to previous versions if the GridFTP data interface was never explicitly set.

Collection HTTPS interface downloads support the HTTP header 'Range: bytes=0-' syntax.

Some endpoints were previous unable to delete a collection’s user_message and user_message_link attributes.

Fix an error deleting a Google Cloud Storage Gateway that was introduced in the 5.4.17 release.

The sharing_restrict_paths is now the default sharing path restrictions for a mapped collections. Additional permissions to share paths can be added to individual users. See the Data Access Guide and command line reference documentation for information on how to configure sharing policies.

If a Google Drive or Google Cloud Storage user credential is either revoked due to an explicit removal of consent, or invalidated because it hasn’t been used for a long period, Globus Connect Server v5 now correctly marks the credential as invalid. When this occurs, users who access a collection that uses the credential will be redirected to the credential page to repair the credential.

When creating a guest collection, if the endpoint is not managed by a subscription, always return a missing subscription error. In earlier versions, other errors (such as OAuth missing scopes) could be returned, even though fixing that situation would in no way allow the collection creation to succeed.

Add a warning and require confirmation when cleaning up an endpoint that is acting as an OIDC server. When the endpoint is cleaned up, the OIDC server is no longer functional and the identity domain can not be reused.

When renewing the endpoint’s certificate, the new private key is not rewritten to the local filesystem, which can cause https errors. Contact support@globus.org for information about how to fix this situation.

The implementation of the force_encryption policy has changed for guest collections. When creating a guest collection, it is associated with another mapped collection. In previous versions, the force_encryption policy was completely independent for the collections. In this version, when you set force_encryption to true on a mapped collection, then all guest collections associated with that mapped collection will also have that policy set to true and it cannot be set to false. If a mapped collection has force_encryption set to false, the guest collection’s value of that property can be set. In either case, if the storage gateway is high assurance, then the force_encryption policy is always true for mapped and guest collections.

The relationship between guest and mapped collections is now represented in the Globus transfer service. As a result, pause rules that are set on a mapped collection will also pause transfers on the related guest collections.

If Globus Connect Server v5 is installed on a system with a FIPS-compliant cryptography module it would be unable to create new collection, as Globus Connect Server v5 used md5 to generate pseudo-random domain names.

If globus-connect-server endpoint setup is run a second time with either a different or missing deployment key file, it would fail with an error Object WILDCARD_DOMAIN already exists.. The incorrect deployment key is now detected and the user is instructed how to proceed.

Fix a crash in globus-connect-server endpoint migrate53 after migrating configuration.

Restrict roles to non-email identities. In previous versions, it was possible to configure Globus Connect Server to assign an email-only identity as either an endpoint owner or another role. However, these roles could not be used to manage the endpoint, as they do not provide authentication information. If a user attempts to create a role for an email-only identity it is now rejected.

Normalize Google Cloud Platform user names. In some cases, Google Cloud platform usernames or email addresses may contain contain mixed-case values. Globus Connect Server would report mismatched username errors when the Google ID token contained uppercase characters.

Handle Ceph usernames that contain reserved characters. If a Ceph username contained a reserved character such as "=", it would be unusable with Globus Connect Server.

Update the Node configuration to allow multiple IP addresses at setup time, as well as allowing the incoming and outgoing port ranges to be changed on a per-node basis.

Fix a bug in the letsencrypt certificate renewal, where the certificate was not being serialized to disk correctly after renewals.

Remove duplicate entries from --help output.

Improve validation of node port ranges.

Improve error messages when attempting to log in to a collection using an invalid collection id.

Collections migrated using the globus-connect-server endpoint migrate53 command did not have their scopes updated as part of the migration process, so the https interface did not work correctly. (As a workaround, change the display_name of a collection to cause the scope values to be fixed.)

Support for admin-supplied domains and certificates for Globus Connect Server services. See the new Globus Connect Server domain guide.

Add an optional OIDC service that integrates with the local PAM modules to act as an identity provider. This allows sites with no OIDC compatible identity service to use their local login policies to generate tokens for use with Globus Connect Server. See the OIDC guide.

New commands to support these features:

Allow identities with the status "private" to be granted roles. Previous versions would disallow those identities.

Fix directory permissions after migrating a v5.3 endpoint using the globus-connect-server endpoint migrate53 command.

Don’t allow attempt to reload the apache configuration until after configuration synchronization is complete.

Ensure that a Unicode-compatible locale is used in wsgi configuration on Debian-based systems.

When a Google Cloud Platform credential fails to refresh or consent is revoked, mark the credential as invalid so that the web interface can prompt the user to repair the credentials.

Add user and group sharing restrictions to mapped collections. See the Sharing configuration section of the data access guide for information on the new configuration. If the new policies are not used, then if the sharing_allowed property is set to true, then any user which maps to a valid local account may create shares.

New command-line to set the new sharing restrictions. These are document in