FAQs: Transfer and Sharing
- What are Globus collections?
- How do I manage my Globus endpoint or collection in the Globus Web App?
- Which browsers does the Globus Web App support?
- How do I see error logs for my transfers?
- How do I check the status of my transfers?
- How can I transfer files to and from my laptop or desktop?
- How will I know when my transfer is completed?
- Which protocols does Globus support?
- Why do I encounter an error when uploading larger files, despite smaller files successfully uploading?
- What are GridFTP concurrency and parallelism?
- Are there any limits on using the file transfer service?
- Can I use Globus to manage identified or sensitive data?
- Does Globus store or access my files?
- What is the Effective Transfer Rate reported by Globus?
- How do I control file permissions during transfers?
- How Does Globus Handle Symlinks?
- How do I link directly to Globus application pages?
- Why is my transfer stalled?
- Why did my transfer expire?
- How do I resubmit a failed transfer?
- What characters should I avoid in filenames/paths?
- How does Globus handle performance tuning on transfers?
- How does load balancing work in Globus Connect Server endpoints with multiple servers?
- How long does Globus store transfer task history?
- Why does my transfer produce only empty (0 byte) files on the destination Globus collection?
- What happens if my transfer exceeds my disk quota?
- What happens to my transfer if I don’t have permission to read a file or directory or if a file or directory cannot be found?
- What factors affect transfer performance?
- How can I automate or schedule transfers with the Globus service?
- I know my collaborator’s email address. Will I be able to share high assurance data with them?
- Why did my collaborator not receive email notification that I shared data with them?
- Can I set permissions on guest collections to automatically expire? Can I limit how long everyone can share data from my collection?
What are Globus collections?
Globus collections provide the interfaces required to access data. This is new terminology introduced with Globus Connect version 5, which has several enhancements and a new architecture. In previous versions, Globus endpoints provided interfaces for (a) secure data access, and (b) administration of the Globus endpoint for server management and configuration. In the new architecture these two functions are cleanly separated with collections used for data access, and Globus Connect Server endpoints used for management.
How do I manage my Globus endpoint or collection in the Globus Web App?
You can manage your Globus Connect Server v5 and Globus Connect Personal collections using the Collections page.
You can manage your Globus Connect Server endpoints using the Console page.
Which browsers does the Globus Web App support?
The Globus Web App is tested and verified to work on the most recent desktop version of Chrome, Edge, Firefox, and Safari.
Earlier versions of these browsers, as well as other desktop browsers (e.g. Opera) and mobile/touch-based browsers, may experience loss of functionality and/or distorted visual effects. Limited or no support may be available for such browsers.
How do I see error logs for my transfers?
You can access a detailed list of errors (and other status messages) for any Globus task using both the Web interface and the command line interface.
Using the Web interface:
-
Click on the Activity page.
-
Click on the label of the transfer you would like to examine. You will see a page with the transfer details.
-
Click on the "Event Log" tab to see a complete list of status messages. Note: this may be a very long list (potentially thousands of pages for a large transfer).
Using the command line interface (CLI):
-
The globus task show command provides a summary of the transfer.
-
The globus task event-list command lists the complete event log for the transfer (this command returns a very long list but you can use various command options to find the event(s) of interest).
All of the above commands have options that control the amount and format of the output returned.
How do I check the status of my transfers?
You can use the Web interface or the globus task list and globus task show commands on the Globus command-line interface to check the status of your transfers. More information and examples are available here.
How can I transfer files to and from my laptop or desktop?
Use Globus Connect Personal, which provides a point-and-click interface for configuring and operating a Globus mapped collection on your local machine. After installing Globus Connect Personal, your computer looks just like any other Globus collection, so all the standard Globus Web App and command line interface features will work. Globus Connect Personal runs behind NATs and firewalls, as long as it can make an outbound connection. Click the link for your operating system to get started with Globus Connect Personal: Mac OS X, Windows, Linux.
How will I know when my transfer is completed?
Globus sends a notification message to the e-mail address associated with your identity.
Which protocols does Globus support?
Globus uses the GridFTP and UDT protocols for managed file transfer and supports direct uploads and downloads to and from collections via HTTPS using a web browser (suitable for smaller data sets).
Why do I encounter an error when uploading larger files, despite smaller files successfully uploading?
When uploading via HTTPS rather than transferring, the Apache/HTTPD configurations of the servers hosting the Collection you’re uploading to may be set in such a way that they prevent uploads over a specified size ("LimitRequestBody").
If you encounter this scenario, you will need to contact the Endpoint admin for further assistance.
What are GridFTP concurrency and parallelism?
Globus optimizes GridFTP transfers by choosing performance optimizations based on the number and sizes of files in the workload. Thus, users do not have to be GridFTP experts to get high performance transfers. If you are an advanced user or resource owner, read on…
- concurrency
-
opens multiple GridFTP control channel sessions, each running in a separate process. Files transfers are distributed among the processes, so this works best if there are many files in a transfer task. If an endpoint is hosted by multiple data transfer nodes, the processes may be distributed among them. Concurrency is best for driving more processes, CPU cores, and even machine nodes.
- parallelism
-
is a network level optimization. GridFTP can send portions of the data over multiple separate TCP connections using a single process on each endpoint. Parallelism is best for transferring large files over high bandwidth, high latency networks.
Globus automatically selects appropriate settings for concurrency and parallelism on every transfer task - these may be overridden on per transfer or collection level.
Are there any limits on using the file transfer service?
We enforce the following limits on usage in order to ensure acceptable quality of service for all users and protect against abuse. Globus currently imposes the following limits:
-
3 active transfer tasks per Globus user
-
100 pending transfer tasks per Globus user
-
100,000,000 files in a single transfer task
-
5,000,000 files in a single directory
-
100,000 files for a directory listing / ls operation
-
10,000 Globus endpoints and collections owned by a single user - this total includes Globus Connect endpoints, and mapped and guest collections. This limit applies to the GCS Manager client identity, limiting the total number of collections on a Globus Connect Server endpoint to 9,999.
-
100 effective permissions per user on a guest collection [1]
-
1,000 total permissions per guest collection
-
10,000 roles per Globus user [2]
-
100 roles per endpoint or collection [3]
In addition, the Globus service will retain task details about events and completed files for up to 31 days, and the retention period for task summary data is 90 days.
The above limits are set based on our experience to-date and should accommodate the needs of most transfer users. If you have requirements that are likely to exceed these limits, please contact us to discuss.
Can I use Globus to manage identified or sensitive data?
The security and privacy of identified data is regulated by a variety of mechanisms, such as the Health Insurance Portability and Accountability Act (HIPAA), state privacy laws, Institutional Review Board (IRB) requirements, National Institutes of Health (NIH) security best practices, and institutional policies, depending on the nature of the identified data and how the data are used, for example, research versus health care use. The suitability of Globus for managing sensitive data varies on a case by case basis, as determined by the organization charged with protecting the data, after consideration of a number of factors, including an analysis of risk and applicable regulatory laws and policies. Please contact us if you would like to use Globus to manage sensitive data of any type. We are happy to help you determine if Globus can meet your needs for sensitive data management.
Does Globus store or access my files?
Files managed by Globus never reside on Globus infrastructure. Files are located on storage resources that are provided and controlled by system owners and administrators, who also implement the file access control policies and permissions used by Globus. Files do not transiently reside on, or pass through, Globus infrastructure during transfer. Files are transferred through a channel that is established directly between Globus collections on the source and destination storage systems, and the Globus service does not have access to this channel. Globus does access file metadata, such as filename and size, for the purposes of monitoring transfer progress, security, and integrity.
What is the Effective Transfer Rate reported by Globus?
The "Effective Transfer Rate" included in e-mail notifications and reported by the details command is the ratio of number of kilobytes transferred to the *total time taken to complete the transfer request*. Here, the "kilo" prefix refers to decimal multiples (103) rather than binary multiples (210). The total time is calculated from the time the transfer request is submitted to Globus to the time the transfer is completed. It includes retry time, downtime on the Globus collections, time that the transfer is paused for credential renewal, and time for checksum calculations. Hence, the Effective Transfer Rate indicates the time taken for reliable file transfer and should not be interpreted as raw bandwidth or throughput information.
For example, if your Globus Connect Personal collection is not connected for a portion of the task, that idle time is included in the transfer rate calculation and can result in relatively low Effective Transfer Rates even though the actual end-to-end throughput on the network is relatively high.
It is also worth noting that Globus allows each user to have up to three simultaneous transfers in progress, with additional transfers queued. If you submit more than three simultaneous transfer requests, the additional requests are queued while the three active requests are completed, and this queue time is also included in the Effective Transfer Rate calculation for those requests.
How do I control file permissions during transfers?
Globus does not preserve file permissions when performing a transfer. When you transfer files with Globus, their permissions are determined entirely by the destination Globus collection’s configuration. There are still ways that you can control the permissions of the files created by Globus, on a destination Globus collection, but they do not operate on information about the original file permissions.
Why We Don’t Preserve Permissions
An obvious question that arises is "Why doesn’t Globus preserve permissions?" This behavior is an unfortunate result of the fact that it is not entirely clear what preserving permissions means for some transfer tasks.
The Ideal Treatment of Permissions
Ideally, given collections A
and B
, with files in
A
, then transferring those files back and forth between
A
and B
would not alter the permissions of those files.
So, if we submitted a transfer task, copy A:/p/q/r
to
B:/x/y/z
, the file at B:/x/y/z
will have exactly
the same permissions as the original at A:/p/q/r
. Consider a second
transfer in the other direction, copy B:/x/y/z
to
A:/p/q/r_prime
.
Since this should share the same property as the previous transfer,
A:/p/q/r
and A:/p/q/r_prime
should be completely
indistinguishable — there should be no way to tell which one is the original by
content or permissions.
The Problem With Ownership
But what if the user authenticates to collection A
as a user with read permissions to A:/p/q/r
, but not ownership? Then when the file is transferred back to A:/p/q/r_prime
, the ownership will have changed. On most systems, only the superuser can change the owner of /p/q/r_prime
to match /p/q/r
. This is the basic issue with attempting to preserve ownership for files.
The Problem With Permissions Bits
Not all permissions settings are supported on all platforms. Consider what happens if A:/p/q/r
has UNIX octal permissions 0111 — anyone can execute the file, but no one can read or write it — and B
is a Windows GCP collection. When the file is stored in Windows as B:/x/y/z
, it can’t be given these same permissions because Windows does not support execute-only files. When B:/x/y/z
is transferred to A:/p/q/r_prime
, the only way for the transfer task to know to restore the original permissions is to keep track of all permissions of files transferred by Globus in case they are transferred again. Even with that extra information, it is difficult to know exactly what to do: what should Globus do if the file has been altered, or had permissions added or removed?
What if the file is moved with scp from B:/x/y/z
to C:/w/t/u
and with Globus from C:/w/t/u
to A:/p/q/r_prime
? Because permissions schemes are not uniform across all platforms, and files may move locally or remotely by means other than the Globus service, we cannot guarantee the transitivity of permissions across a series of transfers.
What Can You Do?
Having stated that the permissions of your files cannot be consistently preserved by Globus for technical reasons, what recourse do you, as a user or administrator, have? Our team is always looking to improve Globus, and better permissions handling is on the To Do List. In the meantime, however, you can make some steps to better control your file permissions.
Further Restricting Permissions for Globus Connect Server
By default, the GridFTP server uses the system umask setting to determine the permissions of all files that it creates. There is an option, passed either through the command line as "-perms", or through the config file (by default, placed in /etc/gridftp.conf ) as a line "perms <value>", which can be used to further restrict the permissions of new files. The option is specified as a three digit octal integer, as typical UNIX permissions are, and is documented in the Globus Toolkit 5.2 release here.
"perms" does not override the umask, but is applied additively. Note that the "perms" option is written as a positive set of permissions bits, which are desired for new files, while the umask is a negative set of bits, which are forbidden. Since the GridFTP server attempts to create files with the "perms" permissions, the effective permissions of a new file are <PERMS> AND (NOT <UMASK>) for regular files, rather than the default of 0666 AND (NOT <UMASK>). "perms" will not alter directory permissions, so those should still be 0777 AND (NOT <UMASK>).
Because the "perms" value is ANDed together with the inverted umask, it cannot be used to apply wider permissions than the umask allows, but it can be used to further restrict access. For example, if the system umask is set to 0002, but you want to forbid world read access and group write access to files, you could set "perms" to 0042. The resulting permissions, in this case, would be 0042 AND (NOT 0002) = 0042 AND (0775) = 0040, as the umask forbids the world write permission granted by "perms".
Using Filesystem ACLs
Because Globus delegates operations to the Globus collection’s filesystem without inspecting ACLs on the source or destination, you can leverage your collection’s support of ACLs to control permissions tightly. By setting ACLs on the destination such that they are applied to all new files in a directory tree, you can effectively set ACLs on the files created by the GridFTP server. GridFTP and Globus will never attempt to explicitly get or set the filesystem ACLs, effectively leaving their application up to the destination Globus collection’s filesystem implementation. Since different filesystems and operating systems may implement ACLs differently, we do not provide explicit instructions for any particular local ACL setup.
Setting the umask
Setting the umask explicitly is the only way to increase the permissions offered on files created by the GridFTP server. The most consistent and successful way to do this is to alter the Globus Connect Server init script to set the umask immediately before launching the GridFTP server. Most typically, the script is found in /etc/init.d/globus-connect-server
If you do not feel comfortable modifying the init script, this option is likely a bad choice for you. The init script is the only supported way of launching GridFTP for a Globus Connect Server installation, so damaging alterations to the script could prevent you from launching Globus Connect Server altogether. (In other words, choose this option at your own peril.)
Controlling Permissions for Globus Connect Personal
The above techniques can be applied to Globus Connect Personal, but there are some caveats. Most notably, we do not officially support modified versions of the Globus Connect Personal client, so if you alter any files or configuration within the client application in order to achieve your desired permissions scheme, your Globus Connect Personal collection will not necessarily qualify for support from Globus staff. At present, none of the forms of Globus Connect Personal support specifying "perms" to the GridFTP server.
If you are running Globus Connect Personal for Linux, you may have some success altering your personal umask setting before launching the application, as your umask should propagate down the process tree to the GridFTP server process. Likewise, if you are running Globus Connect Personal on Mac OS X, you may be successful setting your umask before launching the Globus Connect Personal app through the command line. These actions are not guaranteed to be successful based on the exact behavior of your platform. Because Windows does not support a umask equivalent, there is no way to replicate this behavior in Globus Connect Personal for Windows.
When supported by your platform, filesystem ACLs are respected, but they are not an option for all users.
Use Guest Collections
In many situations, restricting read or write access to a file can be handled correctly using Globus controlled Read and Write permissions on a guest collection. This does not alter the underlying permissions of the files, but restricts permissions when using a Globus account to access the guest collection. Globus will deny users without the Read permission the rights to copy files or list directory contents, and denies users without the Write permission the rights to copy a file to the specified path or directory.
These permissions settings do not alter the underlying guest collection’s permissions scheme in any way, so users with local access to the guest collection may be able to bypass these permissions settings by accessing files directly. If you know that your files are only exposed via Globus, then this option may be right for you.
How Does Globus Handle Symlinks?
At present, Globus skips symlinks in a wide class of transfers. The reason for this is that there are several notions of correct behavior for transfers of symlinks, especially with respect to their interaction with path restrictions in a Globus collection’s configuration. However, symlinks are not uniformly ignored, and in some actions, for which the behavior on symlinks is unambiguous, they will be followed.
This behavior is identical between Globus Connect Personal and Globus Connect Server.
Directory Listing
When listing the contents of a directory, if the path includes symlinks, those symlinks will be followed. However, when the links are followed, they do not receive special treatment — to Globus, they are considered indistinguishable from the directories to which they are links. This is very similar to the treatment of symlinks when doing local directory listings (i.e. ls in most shells), in which the fact that a directory is a link is not necessarily exposed.
So, if you have a symlink /tmp/myhome → /home/username/
, then when you attempt to list the contents of /tmp/myhome/Desktop/
, Globus will return a list of contents of /home/username/Desktop/
. Globus will not give any indication that /tmp/myhome is a symlink; there is no path rewriting or other indication that /tmp/myhome
is anything but an ordinary directory whose contents happen to be identical to /home/username/
.
Recursive Directory Transfers
When doing a recursive directory transfer, all symlinks in the directory tree are ignored. The one and only exception to this rule is the root of a directory transfer.
Consider the previous example, /tmp/myhome → /home/username/
. Doing a recursive directory transfer with a root directory of /tmp/myhome
will transfer all of the contents of /home/username/
, following the symlink /tmp/myhome
. However, a recursive directory transfer on /tmp
will skip /tmp/myhome
, not creating it as a directory, link, or file on the destination. Furthermore, this skipping behavior does not trigger any errors, faults, or warnings in the transfer history, as it is not considered an error condition.
Single File Transfers
Single file transfers follow the same basic rules that directory transfers do, in that they dereference symlinks to their destination files, and create the link on the destination as an ordinary file.
If I have a link, /a/b/c → /p/q/r
on my filesystem to an ordinary file, then transferring /a/b/c
to another Globus collection will behave as though the contents of /p/q/r
were stored in /a/b/c
, not giving any special treatment to /a/b/c
or /p/q/r
on account of its status as a link.
File and Directory Deletion
Globus does not follow symlinks when doing file or directory deletions. However, following the semantics of a typical UNIX rm command, Globus will unlink symbolic links by deleting them during a directory or file removal.
Symlinks and Path Restrictions
Globus collection configuration supports restricting the parts of the filesystem that can be accessed via Globus. In Globus Connect Server and GridFTP this corresponds to the RestrictPaths and SharingRestrictPaths options. By default, these settings apply to non-symlinked files and directories, not allowing access when a symlink points outside of the explicitly allowed components of the filesystem.
This behavior prevents abusive symlinks from breaking out of the path restrictions. Consider the case of a Globus collection which only allows access to /p/q/r/
, and a symlink /p/q/r/root → /
. If symlinks are followed irrespective of the path settings, /p/q/r/root/home/
would be accessible, even though /home/
is not included.
If you trust users with access to a Globus collection not to create this kind of exploitative symlink, you can override this behavior on Globus Connect Server collections with the rp-follow-symlinks option to the GridFTP server. This option is not readily available with Globus Connect Personal installations.
How do I link directly to Globus application pages?
Many of the pages in the Globus application can be linked with parameters that allow the page to open pre-configured for your needs. When you link to a Globus application page, the application will ensure that the user is logged in to the Globus Web App (and prompt the user to authenticate if the user is not already logged in). If you have a specific use case or application feature that isn’t covered in this document please let us know at support@globus.org.
What follows is a list of pages that are configurable.
File Manager (https://app.globus.org/file-manager)
The File Manager page provides the following parameters to preselect the two sides of the File Manager page:
-
origin_id - represents the id of the Globus endpoint or collection for the left hand side of the page.
-
origin_path - represents the path of the Globus endpoint or collection for the left hand side of the page.
-
destination_id - represents the id of the Globus endpoint or collection and path for the left hand side of the page.
-
destination_path - represents the path of the Globus endpoint or collection for the right hand side of the page.
Some examples:
-
https://app.globus.org/file-manager?origin_id=6c54cade-bde5-45c1-bdea-f4bd71dba2cc
-
Sends the user to the File Manager page with the Globus collection
Globus Tutorial Collection 1
selected on the left side.
-
-
-
Sends the user to the File Manager page with the Globus collection
Globus Tutorial Collection 1
selected on the left side andGlobus Tutorial Collection 2
on the right.
-
-
-
Sends the user to the File Manager page with the Globus collection
Globus Tutorial Collection 1
and the path/share/
selected on the left side.
-
Groups (https://app.globus.org/groups)
The groups page can be set up to automatically view a specific group by specifying the group id in the URL. For example the "Demo Group" group has an id of 7e183f30-e177-11e5-ba63-22000ab80e73
, so the URL to automatically link to this group is "https://app.globus.org/groups/7e183f30-e177-11e5-ba63-22000ab80e73". Groups with visibility policies that prevent visibility to non Globus members will display an error page if the user utilizing the link does not meet the visibility requirements. You can determine the group id and link by selecting the desired group on the groups page and clicking the "Get Shareable Link" button. After clicking the "Get Shareable Link" button, the URL for the Shareable Link will be copied to the system clipboard and available for use.
When accessed without parameters, the groups page (https://app.globus.org/groups) displays a list of the groups that the user is a member of, has been invited to, or has requested membership in. The groups page will also show a notification icon for groups that are awaiting action by the user.
Why is my transfer stalled?
There are several reasons a transfer may stall. You may want to start by seeing what Globus is doing with the request.
-
On the web, go to the "View Activity" page
-
Click on ID of specific transfer (probably the one at the top of the list)
-
Click "View Event Log".
-
Look through the progress messages and see if any of them indicate an error, such as an expired credential or quota exceeded.
In the CLI, the globus task show command will give you a high level overview of the progress of the task. If you run globus task show and find the task is not progressing, run the globus task event-list command. The following will show the most recent 5 events for the specified task:
$ globus task event-list --limit 5 <TaskID>
Why did my transfer expire?
Globus will make every attempt to complete a Transfer request but sometimes may be unable to do so, based on factors outside of our control. For example, occasionally, a Globus endpoint or collection may stop responding (due to server failure, network issues, etc.). In this case, you will receive notifications about the error and should follow up with your system administrator.
How do I resubmit a failed transfer?
If your transfer task has failed you should first look at the last few events in the event log to identify any problems needing human intervention (quota exceeded, out-of-disk space, etc.)
After fixing problems you can resubmit your task as follows: . Using the Web GUI, go to the File Manager page, pick your source and destination Globus endpoints or collections, select the "Transfer & Timer Options" menu in the middle of the page, then select the "sync - only transfer new or changed files" checkbox - selecting the 'where the checksum is different' option for this item. . Using the CLI, rerun the transfer command with the --verify-checksum and --sync-level checksum options
What characters should I avoid in filenames/paths?
On different filesystems, directory names and filenames may be restricted to certain characters. For instance, following character are reserved on ext2, ext3, ext4, hfs, FAT, NTFS:
ext2, ext3, ext4 (Linux) | |
---|---|
NULL | |
/ | forward slash |
hfs (Mac OS X) | |
---|---|
: | colon |
/ | forward slash |
FAT, NTFS (MS Windows) | |
---|---|
< | less than |
> | greater than |
: | colon |
" | double quote |
/ | forward slash |
\ | backslash |
| | vertical bar or pipe |
? | question mark |
* | asterisk |
If a file or a directory with one of the characters <>:"\|?* is copied from Linux to MS Windows, then MS Windows will return the error message "The filename, directory name, or volume label syntax is incorrect" and refuse to create the file or the directory.
Also the filesystems FAT, NTFS, hfs are not case sensitive. It means that if two files on ext2, ext3, ext4 are different by case and they are transferred to a non-case sensitive filesystem they will be copied into one file.
How does Globus handle performance tuning on transfers?
Globus tunes transfers based on the number of servers of the source and destination Globus endpoints, the location of the endpoints, and the performance options configured for each endpoint associated with your subscription. We discuss how the Globus service arrives at the values for these performance parameters for transfers between two given endpoints in our GCS Installation Guide. For a discussion on how to control these settings on your endpoint, see the CLI globus endpoint update man page for more details on the --network-use performance option.
How does load balancing work in Globus Connect Server endpoints with multiple servers?
Globus load balances across data transfer nodes when a transfer contains multiple files. This is only done on transfer requests that contain more than 10 files. Also, multiple concurrent transfer requests are load balanced across servers using a round robin algorithm.
How long does Globus store transfer task history?
Summary task information is stored for 90 days, including Task ID, Task Status, Endpoint Display Names, and Task Transfer Settings. Detailed task information is stored for 30 days, including the faults logged for the task as well as the status information for individual files and directories.
From the web interface, users can access their task history at the following URL:
The task details under the "Event Log" tab for a given task are those that will expire after 30 days. All other details for a task will remain available for 90 days.
From the Globus CLI, users can access their task history with the globus task show command like so:
$ globus task show TASK_ID
The above will show the basic details for the task that will be preserved for 90 days. Users can see more detailed information on their task by using the globus task event-list
command like so:
$ globus task event-list TASK_ID
The details returned by the globus task event-list
command show the job details that are preserved for only 30 days, and thus this command will return nothing for tasks more than 30 days old.
Why does my transfer produce only empty (0 byte) files on the destination Globus collection?
This is often a symptom of a firewall policy problem that results in interference with data channel traffic between the source and destination Globus collections involved in the transfer. A good first step in troubleshooting this sort of issue is to ensure that firewall policy for both the source and destination Globus collections is consistent with what Globus shows in the firewall policy requirements for the software that is being used to run that collection. The firewall policy requirements for Globus Connect Server can be found here. The firewall policy requirements for Globus Connect Personal can be found here.
What happens if my transfer exceeds my disk quota?
By default, when the storage quota is exceeded during your transfer, Globus will periodically retry the transfer. The Event Log on the web app Activity page will warn "storage quota exceeded". If no progress on your transfer can be made in three days, the transfer will fail and you will then be notified via email. However, if you select the option to fail on quota error, your transfer will fail immediately rather than retry for three days, and you will be notified immediately via email.
What happens to my transfer if I don’t have permission to read a file or directory or if a file or directory cannot be found?
There may be cases where your transfer includes source files or directories that you are not permitted to read or that have been deleted after the transfer has been submitted but before the transfer has completed. By default, the transfer will pause on the first occurrence of a permission denied or file not found error. Globus will then periodically retry the transfer. If no transfer progress can be made in three days, the transfer will fail, and you will then be notified via email.
Alternatively, you can select the option to skip over source files and directories with permission denied or file not found errors. If the source is GoogleDrive, this option will also skip over source files and directories with ambiguous path errors, which occur with duplicate file names and directories. Be aware that when this option is selected, a successfully completed transfer request does not necessarily mean all files and directories have been transferred. The status of the transfer task and the email notification will specify the number of paths skipped due to permission denied or file not found errors. If that number is greater than zero, always review the list of skipped paths, which is available via the API or the Globus Web App.
What factors affect transfer performance?
The transfer speed between two Globus collections is determined by a number of factors, such as:
-
the speed of the network between the source and destination collections
-
the performance of the source and destination storage systems
-
the load / available resources (RAM, CPU, etc.) of the source and destination collections
The speed of a transfer will be limited by whichever factor presents as the bottleneck.
To get an idea of maximum expected speed for traffic between two DTNs, tools such as iPerf3 can be used.
ESnet has articles on performance testing that may prove valuable to sites wanting to to test or troubleshoot transfer performance.
How can I automate or schedule transfers with the Globus service?
Globus provides multiple methods to automate and schedule transfers.
It is possible to automate transfers with the Globus service using scripts that make use of the Globus API or the Globus CLI. We offer examples of such scripts here. These scripts can also be run on a schedule using cron or some other system scheduler service.
The Globus Timer service can also be used to create scheduled or recurring transfers, using either the Globus Web App or the Globus CLI.
When you start a transfer in the Globus Web App, you can use the Transfer and Timer options of the File Manager to select a transfer start time, choose a repeat interval, and configure an end date or finite number of repeats. You may need to provide consent after selecting the start button.
You can view and manage your recurring or scheduled transfers on the Activity page under the Timers tab. The list of Timer requests can be found under the Timer Log tab. The corresponding transfer task of a successful Timer request is located under the Tasks tab of the Activity page. Please note that a successful request does not necessarily mean that the corresponding transfer task completed successfully. For example, a successful Timer request may have produced a transfer task that encountered a fatal error, such as a quota error. You should check the status of your transfer tasks under the Tasks tab to confirm that your recurring and scheduled transfers are successful.
You can also use the Globus CLI to submit recurring or scheduled transfer tasks. Once you install Globus CLI and authenticate, you can submit a transfer command that defines your source and destination, the files you wish to transfer, and a time interval. Optionally, you can submit the list of files and directories you’d like to transfer as a CSV formatted file, set a delayed start time, sync files, or set other transfer options, such as encryption.
You may not submit a Timer that would result in running an average of more than one repeating transfer per minute or more than 100 Timers per user.
Timers are not supported for high assurance collections at this time.
I know my collaborator’s email address. Will I be able to share high assurance data with them?
Yes, but only if your collaborator’s email address is the same as their username from an identity provider that is recognized by Globus.
Why did my collaborator not receive email notification that I shared data with them?
Your collaborator will not be notified that you shared data with them if your collaborator has never logged into Globus and their username is not the same as their email address.
Can I set permissions on guest collections to automatically expire? Can I limit how long everyone can share data from my collection?
Yes. Owners, administrators, and access managers of high assurance guest collections can use the web application to set or update a permission expiration time. Sharing notification emails will include a permission expiration time. Expired permissions are immediately deleted.
Administrators of high assurance mapped collections can configure a maximum allowable expiration time
for any permission. The maximum can be configured using the Globus Connect Server CLI
with the --acl-expiration-mins
option or using the web application.
Existing permissions with an expiration time longer than the maximum
will automatically be updated with the maximum expiration time.
For example, if a sharing permission does not have an expiration time
when a mapped collection administrator sets the maximum,
the permission will automatically be set to expire at the maximum expiration time.