- What are Globus collections?
- How do I manage my Globus endpoint in the Globus webapp?
- Which browsers does the Globus web user interface support?
- How do I "refresh my credentials" (or "activate an endpoint")?
- What is "activation"?
- How do I see error logs for my transfers?
- How do I check the status of my transfers?
- How can I transfer files to and from my laptop or desktop?
- How will I know when my transfer is completed?
- Which protocols does Globus support?
- What does "Beta" mean in the Globus context?
- What are GridFTP concurrency and parallelism?
- Are there any limits on using the file transfer service?
- How can I activate a GCSv4 endpoint for the maximum amount of time?
- Can I use Globus to manage identified data?
- What is the Effective Transfer Rate reported by Globus?
- How do I control file permissions during transfers?
- How Does Globus Handle Symlinks?
- How do I link directly to Globus application pages?
- Why is my transfer stalled?
- Why did my transfer expire?
- How do I resubmit a failed transfer?
- What characters should I avoid in filenames/paths?
- How does Globus handle performance tuning on transfers?
- How does load balancing work in Globus Connect Server endpoints with multiple servers?
- How long does Globus store transfer task history?
- Why does my transfer produce only empty (0 byte) files on the destination Globus endpoint?
- How do I transfer between two GCSv4 endpoints that require different institutional credentials via CILogon?
- What happens if my transfer exceeds my disk quota?
- What happens to my transfer if I don’t have permission to read a file or directory or if a file or directory cannot be found?
- How can I automate or schedule transfers with the Globus service?
Globus collections provide the interfaces required to access data. This is new terminology introduced with Globus Connect version 5, which has several enhancements and a new architecture. In previous versions, Globus endpoints provided interfaces for (a) secure data access, and (b) administration of the Globus endpoint for server management and configuration. In the new architecture these two functions are cleanly separated with collections used for data access, and GCSv5 endpoints used for management.
The Globus web user interface is tested and verified to work on the most recent desktop version of Chrome, Edge, Firefox, and Safari.
Earlier versions of these browsers, as well as other desktop browsers (e.g. Opera) and mobile/touch-based browsers, may experience loss of functionality and/or distorted visual effects. Limited or no support may be available for such browsers.
There are three ways to activate a GCSv4 endpoint:
Go to the Collections page. Find the GCSv4 endpoint for which you need to renew credentials and click the "activate" link to re-authenticate.
Go to the File Manager page, and enter the GCSv4 endpoints you need. This should prompt you to log in so you can access the GCSv4 endpoint. If you had already begun a transfer when you discovered the need to renew credentials, you do not need to start a new transfer; just logging into the GCSv4 endpoint will be sufficient to refresh your credentials.
You can refresh any credential from any network-connected computer at any time using the above mechanisms.
Globus Connect Personal endpoints do not need to be explicitly activated. If a Globus Connect Personal endpoint is part of an active transfer request, its credentials will be automatically refreshed as needed.
You do not need to resubmit a previously inactive transfer after you’ve reactivated the relevant GCSv4 endpoints; Globus will automatically resume transferring files when both the source and destination endpoints are activated.
You do not need to activate on a share or refresh credentials for a share.
Activation is the mechanism Globus uses to authenticate a user on a GCSv4 endpoint. It enables users to specify which identity should be used to authenticate with a GCSv4 endpoint. For example if I wanted to transfer a file from NERSC to ALCF, I could sign in to Globus using my Globus account name and password, and activate the
nersc#example GCSv4 endpoint using my NERSC username and password, and finally activate the
alcf#example GCSv4 endpoint using my ALCF username and pin+OTP.
You can access a detailed list of errors (and other status messages) for any Globus Online task using both the Web interface and the command line interface.
Using the Web interface:
Click on the Activity page.
Click on the label of the transfer you would like to examine. You will see a popup with the transfer details.
Click on the "Event Log" tab to see a complete list of status messages. Note: this may be a very long list (potentially thousands of pages for a large transfer).
Using the command line interface (CLI):
All of the above commands have options that control the amount and format of the output returned.
Use Globus Connect Personal, which provides a point-and-click interface for configuring and operating a Globus endpoint on your local machine. After installing Globus Connect Personal, your computer looks just like any other Globus endpoint, so all the standard Globus web and command line interface features will work. Globus Connect Personal runs behind NATs and firewalls, as long as it can make an outbound connection. Click the link for your operating system to get started with Globus Connect Personal: Mac OS X, Windows, Linux.
Globus sends a notification message to the e-mail address stored in your account profile.
Globus uses the GridFTP and UDT protocols for managed file transfer and supports direct uploads and downloads to and from endpoints via HTTPS using a web browser (suitable for smaller data sets).
We use "Beta" as an indicator of what an end user may expect from a newly-released Globus feature. We will usually tag something as "Beta" when we have a new feature that we want to expose to users for feedback, but that we know is not fully tested and quality assured. When a Globus feature/page is tagged as "Beta" it means that we expect it to work as intended, but that you may experience some issues in your context. This means that we cannot provide performance guarantees and suggest that you not rely on the feature for production use.
Globus optimizes GridFTP transfers by choosing performance optimizations based on the number and sizes of files in the workload. Thus, users do not have to be GridFTP experts to get high performance transfers. If you are an advanced user or resource owner, read on…
opens multiple login sessions (also known as control channel sessions). Each login session starts a GridFTP process on the server, usually via xinetd. Thus, a concurrency (cc) of 4 would drive 4 GridFTP processes, so you have 4 processes driving IO (each one working on a different file). Files are divided among the sessions, so this only works if you have multiple files in a job (most users do). Also note that each session may be to a different server if you had DNS round robining, a load balancer, or multiple physical DTNs defined in Globus. So concurrency is great for driving more filesystem processes, CPU cores, and even machine nodes, in addition to opening more network data streams.
is a network level optimization. Regular FTP sends a file over one TCP stream, which isn’t ideal for high latency, high throughput links. Parallelism can divide and send a file’s data blocks over multiple TCP streams, however, all the TCP streams have the same source and destination GridFTP server process. Large files over high latency links can benefit from higher parallelism.
Globus automatically selects appropriate settings for concurrency and parallelism on every transfer task - these may be overridden on per transfer or Globus endpoint level.
We enforce the following limits on usage in order to ensure acceptable quality of service for all users and protect against abuse. Globus currently imposes the following limits:
3 active transfer tasks per Globus user
100 pending transfer tasks per Globus user
100,000,000 files in a single transfer task
5,000,000 files in a single directory
100,000 files for a directory listing / ls operation
10,000 Globus endpoints and collections owned by a single user - this total includes host endpoints,  mapped and guest collections,  and shared endpoints  owned by the user. This limit applies to the GCS Manager client identity, limiting the total number of collections on a Globus Connect Server v5 endpoint to 9,999.
100 effective ACLs per user on a shared endpoint or guest collection 
1,000 total ACLs per shared endpoint or guest collection
10,000 roles per Globus user 
100 roles per endpoint or collection 
20 transfer API requests per second per authenticated Globus user. Metering is done over 10 second periods to allow for bursts. 
In addition, the Globus service will retain task details about events and completed files for up to 31 days, and the retention period for task summary data is 90 days.
The above limits are set based on our experience to-date and should accommodate the needs of most transfer users. If you have requirements that are likely to exceed these limits, please contact us to discuss.
By default, the MyProxy service that ships with Globus Connect Server is configured to grant credentials with a maximum lifetime of 7 days. However, there are many GCSv4 endpoints that are configured to use values other than this default, and so such endpoints may have a maximum credential lifetime that is more or less than 7 days. For example, CILogon identity providers will typically grant credentials that have a maximum life time of 10 days.
It is not possible for an end user to be issued credentials that will last longer than the maximum credential lifetime for which an identity provider is configured. When authenticating on a GCSv4 endpoint that uses a MyProxy identity provider, you can specify the lifetime of the credential you are requesting. You can do this on the Globus command line interface by using the globus endpoint activate command with the --proxy-lifetime option, or in the web interface by clicking on the "advanced" link on the web page when prompted to authenticate to the GCSv4 endpoint. However, no matter what credential lifetime you request, you will never be issued a credential that is longer than the maximum credential lifetime than the MyProxy server is configured to issue. When activating on GCSv4 endpoints that use a CILogon identity provider there is no way to request a specific credential lifetime. Rather, you will simply be issued a credential with the default lifetime that the identity provider is configured to issue - typically 10 days.
The security and privacy of identified data is regulated by a variety of mechanisms, such as the Health Insurance Portability and Accountability Act (HIPAA), state privacy laws, Institutional Review Board (IRB) requirements, National Institutes of Health (NIH) security best practices, and institutional policies, depending on the nature of the identified data and how the data are used, for example, research versus health care use. The suitability of Globus for managing sensitive data varies on a case by case basis, as determined by the organization charged with protecting the data, after consideration of a number of factors, including an analysis of risk and applicable regulatory laws and policies. We do not currently have the capacity to act as a Business Associate of a Covered Entity, however, we seek partners interested in working with us towards a common goal of a Business Associate Agreement.
Files managed by Globus never reside on Globus infrastructure. Files are located on storage resources that are provided and controlled by system owners and administrators, who also implement the file access control policies and permissions used by Globus. Files do not transiently reside on, or pass through, Globus infrastructure during transfer. Files are transferred through a channel that is established directly between Globus endpoints on the source and destination storage systems, and the Globus service does not have access to this channel. Globus does access file metadata, such as filename and size, for the purposes of monitoring transfer progress, security, and integrity. When transferring sensitive data, Globus users should opt to encrypt the transfer channel.
Please contact us if you would like to use Globus to manage sensitive data of any type. We are happy to help you determine if Globus can meet your needs for sensitive data management.
The "Effective Transfer Rate" included in e-mail notifications and reported by the details command is the ratio of number of bits transferred to the *total time taken to complete the transfer request*. The total time is calculated from the time the transfer request is submitted to Globus to the time the transfer is completed. It includes retry time, downtime on the Globus endpoints, time that the transfer is paused for credential renewal, and time for checksum calculations. Hence, the Effective Transfer Rate indicates the time taken for reliable file transfer and should not be interpreted as raw bandwidth or throughput information.
For example, if your credentials on a GCSv4 endpoint expire and it takes you a few hours to renew them, that idle time is included in the transfer rate calculation and can result in relatively low Effective Transfer Rates even though the actual end-to-end throughput on the network is relatively high.
It is also worth noting that Globus allows each user to have up to three simultaneous transfers in progress, with additional transfers queued. If you submit more than three simultaneous transfer requests, the additional requests are queued while the three active requests are completed, and this queue time is also included in the Effective Transfer Rate calculation for those requests.
Globus does not preserve file permissions when performing a transfer. When you transfer files with Globus, their permissions are determined entirely by the destination Globus endpoint’s configuration. There are still ways that you can control the permissions of the files created by Globus, on a destination Globus endpoint, but they do not operate on information about the original file permissions.
An obvious question that arises is "Why doesn’t Globus preserve permissions?" This behavior is an unfortunate result of the fact that it is not entirely clear what preserving permissions means for some transfer tasks.
Ideally, given Globus endpoints
user#B, with files in
user#A, then transferring those files back and forth between
user#B would not alter the permissions of those files.
So, if we submitted a transfer task, copy
user#B:/x/y/z, the file at
user#B:/x/y/z will have exactly
the same permissions as the original at
user#A:/p/q/r. Consider a second
transfer in the other direction, copy
Since this should share the same property as the previous transfer,
user#A:/p/q/r_prime should be completely
indistinguishable — there should be no way to tell which one is the original by
content or permissions.
But what if the user authenticates to
user#A as a user with read permissions to
user#A:/p/q/r, but not ownership? Then when the file is transferred back to
user#A:/p/q/r_prime, the ownership will have changed. On most systems, only the superuser can change the owner of
/p/q/r_prime to match
/p/q/r. This is the basic issue with attempting to preserve ownership for files.
Not all permissions settings are supported on all platforms. Consider what happens if
user#A:/p/q/r has UNIX octal permissions 0111 — anyone can execute the file, but no one can read or write it — and
user#B is a Windows GCP endpoint. When the file is stored in Windows as
user#B:/x/y/z, it can’t be given these same permissions because Windows does not support execute-only files. When
user#B:/x/y/z is transferred to
user#A:/p/q/r_prime, the only way for the transfer task to know to restore the original permissions is to keep track of all permissions of files transferred by Globus in case they are transferred again. Even with that extra information, it is difficult to know exactly what to do: what should Globus do if the file has been altered, or had permissions added or removed?
What if the file is moved with scp from
user#C:/w/t/u and with Globus from
user#A:/p/q/r_prime? Because permissions schemes are not uniform across all platforms, and files may move locally or remotely by means other than the Globus service, we cannot guarantee the transitivity of permissions across a series of transfers.
Having stated that the permissions of your files cannot be consistently preserved by Globus for technical reasons, what recourse do you, as a user or Globus endpoint administrator, have? Our team is always looking to improve Globus, and better permissions handling is on the To Do List. In the meantime, however, you can make some steps to better control your file permissions.
By default, the GridFTP server uses the system umask setting to determine the permissions of all files that it creates. There is an option, passed either through the command line as "-perms", or through the config file (by default, placed in /etc/gridftp.conf ) as a line "perms <value>", which can be used to further restrict the permissions of new files. The option is specified as a three digit octal integer, as typical UNIX permissions are, and is documented in the Globus Toolkit 5.2 release here.
"perms" does not override the umask, but is applied additively. Note that the "perms" option is written as a positive set of permissions bits, which are desired for new files, while the umask is a negative set of bits, which are forbidden. Since the GridFTP server attempts to create files with the "perms" permissions, the effective permissions of a new file are <PERMS> AND (NOT <UMASK>) for regular files, rather than the default of 0666 AND (NOT <UMASK>). "perms" will not alter directory permissions, so those should still be 0777 AND (NOT <UMASK>).
Because the "perms" value is ANDed together with the inverted umask, it cannot be used to apply wider permissions than the umask allows, but it can be used to further restrict access. For example, if the system umask is set to 0002, but you want to forbid world read access and group write access to files, you could set "perms" to 0042. The resulting permissions, in this case, would be 0042 AND (NOT 0002) = 0042 AND (0775) = 0040, as the umask forbids the world write permission granted by "perms".
Because Globus delegates operations to the Globus endpoint’s filesystem without inspecting ACLs on the source or destination, you can leverage your endpoint’s support of ACLs to control permissions tightly. By setting ACLs on the destination such that they are applied to all new files in a directory tree, you can effectively set ACLs on the files created by the GridFTP server. GridFTP and Globus will never attempt to explicitly get or set the filesystem ACLs, effectively leaving their application up to the destination Globus endpoint’s filesystem implementation. Since different filesystems and operating systems may implement ACLs differently, we do not provide explicit instructions for any particular local ACL setup.
Setting the umask explicitly is the only way to increase the permissions offered on files created by the GridFTP server. The most consistent and successful way to do this is to alter the Globus Connect Server init script to set the umask immediately before launching the GridFTP server. Most typically, the script is found in /etc/init.d/globus-connect-server
If you do not feel comfortable modifying the init script, this option is likely a bad choice for you. The init script is the only supported way of launching GridFTP for a Globus Connect Server installation, so damaging alterations to the script could prevent you from launching Globus Connect Server altogether. (In other words, choose this option at your own peril.)
The above techniques can be applied to Globus Connect Personal, but there are some caveats. Most notably, we do not officially support modified versions of the Globus Connect Personal client, so if you alter any files or configuration within the client application in order to achieve your desired permissions scheme, your Globus endpoint will not necessarily qualify for support from Globus staff. At present, none of the forms of Globus Connect Personal support specifying "perms" to the GridFTP server.
If you are running Globus Connect Personal for Linux, you may have some success altering your personal umask setting before launching the application, as your umask should propagate down the process tree to the GridFTP server process. Likewise, if you are running Globus Connect Personal on Mac OS X, you may be successful setting your umask before launching the Globus Connect Personal app through the command line. These actions are not guaranteed to be successful based on the exact behavior of your platform. Because Windows does not support a umask equivalent, there is no way to replicate this behavior in Globus Connect Personal for Windows.
When supported by your platform, filesystem ACLs are respected, but they are not an option for all users.
In many situations, restricting read or write access to a file can be handled correctly using Globus controlled Read and Write permissions on a Shared Endpoint. This does not alter the underlying permissions of the files, but restricts permissions when using a Globus account to access the shared endpoint. Globus will deny users without the Read permission the rights to copy files or list directory contents, and denies users without the Write permission the rights to copy a file to the specified path or directory.
These permissions settings do not alter the underlying shared endpoint’s permissions scheme in any way, so users with local access to the shared endpoint may be able to bypass these permissions settings by accessing files directly. If you know that your files are only exposed via Globus, then this option may be right for you.
At present, Globus skips symlinks in a wide class of transfers. The reason for this is that there are several notions of correct behavior for transfers of symlinks, especially with respect to their interaction with path restrictions in a Globus endpoint’s configuration. However, symlinks are not uniformly ignored, and in some actions, for which the behavior on symlinks is unambiguous, they will be followed.
This behavior is identical between Globus Connect Personal and Globus Connect Server.
When listing the contents of a directory, if the path includes symlinks, those symlinks will be followed. However, when the links are followed, they do not receive special treatment — to Globus, they are considered indistinguishable from the directories to which they are links. This is very similar to the treatment of symlinks when doing local directory listings (i.e. ls in most shells), in which the fact that a directory is a link is not necessarily exposed.
So, if you have a symlink
/tmp/myhome → /home/username/, then when you attempt to list the contents of
/tmp/myhome/Desktop/, Globus will return a list of contents of
/home/username/Desktop/. Globus will not give any indication that /tmp/myhome is a symlink; there is no path rewriting or other indication that
/tmp/myhome is anything but an ordinary directory whose contents happen to be identical to
When doing a recursive directory transfer, all symlinks in the directory tree are ignored. The one and only exception to this rule is the root of a directory transfer.
Consider the previous example,
/tmp/myhome → /home/username/. Doing a recursive directory transfer with a root directory of
/tmp/myhome will transfer all of the contents of
/home/username/, following the symlink
/tmp/myhome. However, a recursive directory transfer on
/tmp will skip
/tmp/myhome, not creating it as a directory, link, or file on the destination. Furthermore, this skipping behavior does not trigger any errors, faults, or warnings in the transfer history, as it is not considered an error condition.
Single file transfers follow the same basic rules that directory transfers do, in that they dereference symlinks to their destination files, and create the link on the destination as an ordinary file.
If I have a link,
/a/b/c → /p/q/r on my filesystem to an ordinary file, then transferring
/a/b/c to another Globus endpoint will behave as though the contents of
/p/q/r were stored in
/a/b/c, not giving any special treatment to
/p/q/r on account of its status as a link.
Globus does not follow symlinks when doing file or directory deletions. However, following the semantics of a typical UNIX rm command, Globus will unlink symbolic links by deleting them during a directory or file removal.
Globus endpoint configuration supports restricting the parts of the filesystem that can be accessed via Globus. In Globus Connect Server and GridFTP this corresponds to the RestrictPaths and SharingRestrictPaths options. By default, these settings apply to non-symlinked files and directories, not allowing access when a symlink points outside of the explicitly allowed components of the filesystem.
This behavior prevents abusive symlinks from breaking out of the path restrictions. Consider the case of a Globus endpoint which only allows access to
/p/q/r/, and a symlink
/p/q/r/root → /. If symlinks are followed irrespective of the path settings,
/p/q/r/root/home/ would be accessible, even though
/home/ is not included.
If you trust users with access to a Globus endpoint not to create this kind of exploitative symlink, you can override this behavior on Globus Connect Server endpoints with the rp-follow-symlinks option to the GridFTP server. This option is not readily available with Globus Connect Personal installations.
Many of the pages in the Globus application can be linked with parameters that allow the page to open pre-configured for your needs. When you link to a Globus application page, the application will ensure that the user is logged in to the Globus website (and prompt the user to authenticate if the user is not already logged in). If you have a specific use case or application feature that isn’t covered in this document please let us know at email@example.com.
What follows is a list of pages that are configurable.
File Manager (https://app.globus.org/file-manager)
The File Manager page provides the following parameters to preselect the two sides of the File Manager page:
origin_id - represents the id of the Globus endpoint for the left hand side of the page.
origin_path - represents the path of the Globus endpoint for the left hand side of the page.
destination_id - represents the id of the Globus endpoint and path for the left hand side of the page.
destination_path - represents the path of the Globus endpoint for the right hand side of the page.
Globus endpoints which are specified on the URL in this fashion will require activation if they are not already activated by the user. You can determine the parameters and link by setting the desired Globus endpoint(s) on the File Manager page and selecting "get link" from the function menu.
Sends the user to the File Manager page with the Globus endpoint
Globus Tutorial Endpoint 1selected on the left side.
Sends the user to the File Manager page with the Globus endpoint
Globus Tutorial Endpoint 1selected on the left side and
Globus Tutorial Endpoint 2on the right.
Sends the user to the File Manager page with the Globus endpoint
Globus Tutorial Endpoint 1and the path
/share/selected on the left side.
The groups page can be set up to automatically view a specific group by specifying the group id in the URL. For example the "Demo Group" group has an id of
7e183f30-e177-11e5-ba63-22000ab80e73, so the URL to automatically link to this group is "https://app.globus.org/groups/7e183f30-e177-11e5-ba63-22000ab80e73". Groups with visibility policies that prevent visibility to non Globus members will display an error page if the user utilizing the link does not meet the visibility requirements. You can determine the group id and link by selecting the desired group on the groups page and clicking the "Get Shareable Link" button. After clicking the "Get Shareable Link" button, the URL for the Shareable Link will be copied to the system clipboard and available for use.
When accessed without parameters, the groups page (https://app.globus.org/groups) displays a list of the groups that the user is a member of, has been invited to, or has requested membership in. The groups page will also show a notification icon for groups that are awaiting action by the user.
For a specific GCSv4 endpoint, bring the endpoint up in the File Manager. If you do not have a current activation on the GCSv4 endpoint selected, you will be prompted to activate on it. To see all GCSv4 endpoints that the user has credentials that are being used for active transfers and about to expire, go to https://app.globus.org/collections?reactivatable=1&scope=in-use.
There are several reasons a transfer may stall. You may want to start by seeing what Globus is doing with the request.
On the web, go to the "View Activity" page
Click on ID of specific transfer (probably the one at the top of the list)
Click "View Event Log".
Look through the progress messages and see if any of them indicate an error, such as an expired credential or quota exceeded.
In the CLI, the globus task show command will give you a high level overview of the progress of the task. If you run globus task show and find the task is not progressing, run the globus task event-list command. The following will show the most recent 5 events for the specified task:
$ globus task event-list --limit 5 <TaskID>
If the globus task show command shows the task is inactive, you may need to refresh your credentials. See this FAQ item for information on how to do this.
Globus will make every attempt to complete a Transfer request but sometimes may be unable to do so, based on factors outside of our control. For example, occasionally, a Globus endpoint may stop responding (due to server failure, network issues, etc.). In this case, you will receive notifications about the error and should follow up with your system administrator.
A more common cause is "credential expiration". This means that Globus is no longer authorized to access a GCSv4 endpoint on your behalf to execute/manage your transfer. When this happens, Globus will send you an e-mail notification and suspend the transfer task until you renew your credentials for the GCSv4 endpoint. If the credentials are not renewed (i.e. the endpoint is not reactivated) within 3 days of the notification being sent, the transfer task will automatically expire.
Instructions for renewing your credentials are available here.
If your transfer task has failed you should first look at the last few events in the event log to identify any problems needing human intervention (quota exceeded, out-of-disk space, etc.)
After fixing problems you can resubmit your task as follows: . Using the Web GUI, go to the File Manager page, pick your source and destination Globus endpoints, select the "Transfer & Timer Options" menu in the middle of the page, then select the "sync - only transfer new or changed files" checkbox - selecting the 'where the checksum is different' option for this item. . Using the CLI, rerun the transfer command with the --verify-checksum and --sync-level checksum options
On different filesystems, directory names and filenames may be restricted to certain characters. For instance, following character are reserved on ext2, ext3, ext4, hfs, FAT, NTFS:
|ext2, ext3, ext4 (Linux)|
|hfs (Mac OS X)|
|FAT, NTFS (MS Windows)|
||||vertical bar or pipe|
If a file or a directory with one of the characters <>:"\|?* is copied from Linux to MS Windows, then MS Windows will return the error message "The filename, directory name, or volume label syntax is incorrect" and refuse to create the file or the directory.
Also the filesystems FAT, NTFS, hfs are not case sensitive. It means that if two files on ext2, ext3, ext4 are different by case and they are transferred to a non-case sensitive filesystem they will be copied into one file.
Globus tunes transfers based on the number of servers of the source and destination Globus endpoints, the location of the endpoints, and the performance options configured for each managed endpoint in your provider subscription. We discuss how the Globus service arrives at the values for these performance parameters for transfers between two given endpoints in our GCS Installation Guide. For a discussion on how to control these settings on your endpoint, see the CLI globus endpoint update man page for more details on the --network-use performance option.
Globus load balances across data transfer nodes when a transfer contains multiple files. This is only done on transfer requests that contain more than 10 files. Also, multiple concurrent transfer requests are load balanced across servers using a round robin algorithm.
Summary task information is stored for 90 days, including Task ID, Task Status, Endpoint Display Names, and Task Transfer Settings. Detailed task information is stored for 30 days, including the faults logged for the task as well as the status information for individual files and directories.
From the web interface, users can access their task history at the following URL:
The task details under the "Event Log" tab for a given task are those that will expire after 30 days. All other details for a task will remain available for 90 days.
$ globus task show TASK_ID
The above will show the basic details for the task that will be preserved for 90 days. Users can see more detailed information on their task by using the
globus task event-list command like so:
$ globus task event-list TASK_ID
The details returned by the
globus task event-list command show the job details that are preserved for only 30 days, and thus this command will return nothing for tasks more than 30 days old.
This is often a symptom of a firewall policy problem that results in interference with data channel traffic between the source and destination Globus endpoints involved in the transfer. A good first step in troubleshooting this sort of issue is to ensure that firewall policy for both the source and destination Globus endpoints is consistent with what Globus shows in the firewall policy requirements for the software that is being used to run that endpoint. The firewall policy requirements for Globus Connect Server can be found here. The firewall policy requirements for Globus Connect Personal can be found here.
How do I transfer between two GCSv4 endpoints that require different institutional credentials via CILogon?
When a user activates a GCSv4 endpoint with an institutional credential that uses CILogon, the credential is automatically associated with all GCSv4 endpoints that require a credential that uses CILogon. When a user attempts to transfer to or from an GCSv4 endpoint that requires a different CILogon institutional credential, an error will occur on this endpoint.
To mitigate this issue, the user must find the GCSv4 endpoint with the error at https://app.globus.org/collections and choose the "Extend Activation" option on the Overview tab. This will prompt a login flow, allowing the user to choose the correct institution.
For example, GCSv4 Endpoints A and B require institutional credentials using CILogon from Institutions A and B, respectively. If a user has activated GCSv4 Endpoint A and then attempts a transfer to/from GCSv4 Endpoint B, an error will be reported indicating that Endpoint B has incorrect credentials or that the user is not found. The user can then use the "Extend Activation" option for Endpoint B in order to login with credentials from Institution B to associate the correct credentials with Endpoint B.
By default, when the storage quota is exceeded during your transfer, Globus will periodically retry the transfer. The Event Log on the web app Activity page will warn "storage quota exceeded". If no progress on your transfer can be made in three days, the transfer will fail and you will then be notified via email. However, if you select the option to fail on quota error, your transfer will fail immediately rather than retry for three days, and you will be notified immediately via email.
What happens to my transfer if I don’t have permission to read a file or directory or if a file or directory cannot be found?
There may be cases where your transfer includes source files or directories that you are not permitted to read or that have been deleted after the transfer has been submitted but before the transfer has completed. By default, the transfer will pause on the first occurrence of a permission denied or file not found error. Globus will then periodically retry the transfer. If no transfer progress can be made in three days, the transfer will fail, and you will then be notified via email.
Alternatively, you can select the option to skip over source files and directories with permission denied or file not found errors. Be aware that when this option is selected, a successfully completed transfer request does not necessarily mean all files and directories have been transferred. The status of the transfer task and the email notification will specify the number of paths skipped due to permission denied or file not found errors. If that number is greater than zero, always review the list of skipped paths, which is available via the API or the web app.
Globus provides multiple methods to automate and schedule transfers.
It is possible to automate transfers with the Globus service using scripts that make use of the Globus API or the Globus CLI. We offer examples of such scripts here. These scripts can also be run on a schedule using cron or some other system scheduler service.
The Globus Timer service can also be used to create scheduled or recurring transfers, using either the Globus webapp or the Globus Timer CLI.
When you start a transfer in the Globus web app, you can use the Transfer and Timer options of the File Manager to select a transfer start time, choose a repeat interval, and configure an end date or finite number of repeats. You may need to provide consent after selecting the start button.
You can view and manage your recurring or scheduled transfers on the Activity page under the Timers tab. The list of Timer requests can be found under the Timer Log tab. The corresponding transfer task of a successful Timer request is located under the Tasks tab of the Activity page. Please note that a successful request does not necessarily mean that the corresponding transfer task completed successfully. For example, a successful Timer request may have produced a transfer task that encountered a fatal error, such as a quota error. You should check the status of your transfer tasks under the Tasks tab to confirm that your recurring and scheduled transfers are successful.
You can also use globus-timer-cli, which is a command line client in beta release, to submit recurring or scheduled transfer tasks. Once you install globus-timer-cli and authenticate, you can submit a transfer command that defines your source and destination, the files you wish to transfer, and a time interval. Optionally, you can submit the list of files and directories you’d like to transfer as a CSV formatted file, set a delayed start time, sync files, or set other transfer options, such as encryption.
You may not submit a Timer that would result in running an average of more than one repeating transfer per minute or more than 100 Timers per user.
Timers are not supported for high assurance collections at this time.