- What are Globus collections?
- What’s the difference between Globus and GridFTP?
- Which browsers does the Globus web user interface support?
- How do I "refresh my credentials" (or "activate an endpoint")?
- What does the error "Credentials are needed" mean?
- What is "activation"?
- How do I see error logs for my transfers?
- How do I check the status of my transfers?
- How can I transfer files to and from my laptop or desktop?
- How will I know when my transfer is completed?
- Which protocols does Globus support?
- What does "Beta" mean in the Globus context?
- What are GridFTP concurrency, pipelining, parallelism, and striping?
- Are there any limits on using the file transfer service?
- How can I activate an endpoint for the maximum amount of time?
- Can I use Globus to manage identified data?
- What is the Effective Transfer Rate reported by Globus?
- How do I control file permissions during transfers?
- How Does Globus Handle Symlinks?
- How do I link directly to Globus application pages?
- Why is my transfer stalled?
- What does the error "Directory contents cannot be found" mean?
- Why did my transfer expire?
- How do I resubmit a failed transfer?
- What characters should I avoid in filenames/paths?
- How does Globus handle performance tuning on transfers?
- How does load balancing work in endpoints with multiple servers?
- Can I use Globus to transfer data to/from any anonymous FTP server?
- How long does Globus store transfer task history?
- Why does my transfer produce only empty (0 byte) files on the destination endpoint?
- How can I automate transfers with the Globus service?
- How do I transfer between two endpoints that require different institutional credentials via CILogon?
- What happens if my transfer exceeds my disk quota?
- What happens to my transfer if I don’t have permission to read a file or directory or if a file or directory cannot be found?
- How can I submit recurring or scheduled transfers?
Globus collections provide the interfaces required to access data. This is new terminology introduced with Globus Connect version 5, which has several enhancements and a new architecture. In previous versions, endpoints provided interfaces for (a) secure data access, and (b) administration of the endpoint for server management and configuration. In the new architecture these two functions are cleanly separated with collections used for data access, and endpoints used for management.
Globus is software-as-a-service (SaaS) for file transfer, and sharing. It is a cloud-hosted service, operated by the Globus development team, that acts as a third-party mediator/facilitator for managing data on storage systems (or endpoints) that are owned and managed by their respective owners.
GridFTP is user-installable software that is part of the Globus Toolkit and is used by grid installers and developers building large-scale data movement solutions. Globus uses the GridFTP protocol for transferring files (HTTP support is planned for a future release).
The Globus web user interface is tested and verified to work on the most recent desktop version of Chrome, Edge, Firefox, and Safari.
Earlier versions of these browsers, as well as other desktop browsers (e.g. Opera) and mobile/touch-based browsers, may experience loss of functionality and/or distorted visual effects. Limited or no support may be available for such browsers.
Because Globus does not store usernames/passwords for your endpoints, periodic re-authentication is required. This means you need to log on with the appropriate username and password for the given endpoint (this is called endpoint "activation"). There are three ways to refresh your credentials in Globus:
Go to the Endpoints page. Find the endpoint for which you need to renew credentials and click the "activate" link to re-authenticate.
Go to the File Manager page, and enter the endpoints you need. This should prompt you to log in so you can access the endpoint. If you had already begun a transfer when you discovered the need to renew credentials, you do not need to start a new transfer; just logging into the endpoint will be sufficient to refresh your credentials.
You can refresh any credential from any network-connected computer at any time using the above mechanisms.
Globus Connect Personal endpoints do not need to be explicitly activated. If a Globus Connect Personal endpoint is part of an active transfer request, its credentials will be automatically refreshed as needed.
You do not need to resubmit a previously inactive transfer after you’ve reactivated the relevant endpoints; Globus will automatically resume transferring files when both the source and destination endpoints are activated.
It means Globus no longer has access to the endpoint(s) you are using because your credentials have expired. See the instructions above for ways to re-authenticate on the endpoint and refresh your credentials.
Activation is the mechanism Globus uses to authenticate a user on an endpoint. It enables users to specify which identity should be used to authenticate with an endpoint. For example if I wanted to transfer a file from NERSC to ALCF, I could sign in to Globus using my Globus account name and password, and activate the
nersc#dtn endpoint using my NERSC username and password, and finally activate the
alcf#dtn endpoint using my ALCF username and pin+OTP.
You can access a detailed list of errors (and other status messages) for any Globus Online task using both the Web interface and the command line interface.
Using the Web interface:
Click on the Activity page.
Click on the label of the transfer you would like to examine. You will see a popup with the transfer details.
Click on the "Event Log" tab to see a complete list of status messages. Note: this may be a very long list (potentially thousands of pages for a large transfer).
Using the command line interface (CLI):
All of the above commands have options that control the amount and format of the output returned.
Use Globus Connect Personal, which provides a point-and-click interface for configuring and operating a Globus endpoint on your local machine. After installing Globus Connect Personal, your computer looks just like any other Globus endpoint, so all the standard Globus web and command line interface features will work. Globus Connect Personal runs behind NATs and firewalls, as long as it can make an outbound connection. Click the link for your operating system to get started with Globus Connect Personal: Mac OS X, Windows, Linux.
Globus sends a notification message to the e-mail address stored in your account profile.
Globus uses the GridFTP and UDT protocols to transfer data. Direct uploads and downloads to and from endpoints via HTTPS (using a web browser, for example) is available in Globus Connect Server version 5.
We use "Beta" as an indicator of what an end user may expect from a newly-released Globus feature. We will usually tag something as "Beta" when we have a new feature that we want to expose to users for feedback, but that we know is not fully tested and quality assured. When a Globus feature/page is tagged as "Beta" it means that we expect it to work as intended, but that you may experience some issues in your context. This means that we cannot provide performance guarantees and suggest that you not rely on the feature for production use.
Globus optimizes GridFTP transfers by choosing performance optimizations based on the number and sizes of files in the workload. Thus, users do not have to be GridFTP and globus-url-copy experts to get high performance transfers. If you are an advanced user or resource owner, read on…
opens multiple login sessions (also known as control channel sessions). Each login session starts a GridFTP process on the server, usually via xinetd. Thus, a concurrency (cc) of 4 would drive 4 GridFTP processes, so you have 4 processes driving IO (each one working on a different file). Files are divided among the sessions, so this only works if you have multiple files in a job (most users do). Also note that each session may be to a different server if you had DNS round robining, a load balancer, or multiple physical endpoints defined in Globus. So concurrency is great for driving more filesystem processes, CPU cores, and even machine nodes, in addition to opening more network data streams.
speeds up lots of tiny files by stuffing multiple FTP commands into each login session back-to-back without waiting for the first command’s response. This reduces latency and keeps the GridFTP server constantly busy; it is never idle waiting for the next command. Note that a GridFTP server process currently only works on one command at a time (future protocol enhancements are planned to drive threaded, out of order processing of commands).
is a network level optimization. Regular FTP sends a file over one TCP stream, which isn’t ideal for high latency, high throughput links. Parallelism can divide and send a file’s data blocks over multiple TCP streams, however, all the TCP streams have the same source and destination GridFTP server process. Large files over high latency links can benefit from higher parallelism.
splits a single file’s data blocks across multiple servers. Globus does not support striping, based on the observation that most users are actually transferring more than one file and that an endpoint often serves multiple users concurrently. Striping can actually be counter productive in these cases, since it adds additional overhead and complexity, and the other options listed above deliver excellent performance.
Globus automatically selects appropriate settings for concurrency, pipelining, and parallelism on every transfer task - these may be overridden on per transfer or endpoint level.
We enforce some limits on usage in order to provide reasonable performance to all users and protect against abuse. A Globus user is currently subject to the following limits:
3 active transfer tasks
100 pending transfer tasks
100,000,000 files in a single transfer task
5,000,000 files in a single directory
100 effective ACLs per user on an endpoint or guest collection 
1,000 total ACLs per endpoint or collection
1,000 roles per user 
100 roles per endpoint or collection 
20 transfer API requests per second per authenticated user. Metering is done over 10 second periods to allow for bursts. 
In addition, the Globus service will retain task details about events and completed files for up to 31 days.
The above limits are set based on our experience to-date and should accommodate the needs of most transfer users. If you have requirements that are likely to exceed these limits, please contact us to discuss.
By default, the MyProxy service that ships with Globus Connect Server is configured to grant credentials with a maximum lifetime of 7 days. However, there are many endpoints that are configured to use values other than this default, and so such endpoints may have a maximum credential lifetime that is more or less than 7 days. For example, CILogon identity providers will typically grant credentials that have a maximum life time of 10 days.
It is not possible for an end user to be issued credentials that will last longer than the maximum credential lifetime for which an identity provider is configured. When authenticating on an endpoint that uses a MyProxy identity provider, you can specify the lifetime of the credential you are requesting. You can do this on the Globus command line interface by using the globus endpoint activate command with the --proxy-lifetime option, or in the web interface by clicking on the "advanced" link on the web page when prompted to authenticate to the endpoint. However, no matter what credential lifetime you request, you will never be issued a credential that is longer than the maximum credential lifetime than the MyProxy server is configured to issue. When activating on endpoints that use a CILogon identity provider there is no way to request a specific credential lifetime. Rather, you will simply be issued a credential with the default lifetime that the identity provider is configured to issue - typically 10 days.
The security and privacy of identified data is regulated by a variety of mechanisms, such as the Health Insurance Portability and Accountability Act (HIPAA), state privacy laws, Institutional Review Board (IRB) requirements, National Institutes of Health (NIH) security best practices, and institutional policies, depending on the nature of the identified data and how the data are used, for example, research versus health care use. The suitability of Globus for managing sensitive data varies on a case by case basis, as determined by the organization charged with protecting the data, after consideration of a number of factors, including an analysis of risk and applicable regulatory laws and policies. We do not currently have the capacity to act as a Business Associate of a Covered Entity, however, we seek partners interested in working with us towards a common goal of a Business Associate Agreement.
Files managed by Globus never reside on Globus infrastructure. Files are located on storage resources that are provided and controlled by system owners and administrators, who also implement the file access control policies and permissions used by Globus. Files do not transiently reside on, or pass through, Globus infrastructure during transfer. Files are transferred through a channel that is established directly between Globus endpoints on the source and destination storage systems, and the Globus service does not have access to this channel. Globus does access file metadata, such as filename and size, for the purposes of monitoring transfer progress, security, and integrity. When transferring sensitive data, Globus users should opt to encrypt the transfer channel.
Please contact us if you would like to use Globus to manage sensitive data of any type. We are happy to help you determine if Globus can meet your needs for sensitive data management.
The "Effective Transfer Rate" included in e-mail notifications and reported by the details command is the ratio of number of bits transferred to the *total time taken to complete the transfer request*. The total time is calculated from the time the transfer request is submitted to Globus to the time the transfer is completed. It includes retry time, downtime on the endpoints, time that the transfer is paused for credential renewal, and time for checksum calculations. Hence, the Effective Transfer Rate indicates the time taken for reliable file transfer and should not be interpreted as raw bandwidth or throughput information.
For example, if your credentials on either endpoint expire and it takes you a few hours to renew them, that idle time is included in the transfer rate calculation and can result in relatively low Effective Transfer Rates even though the actual end-to-end throughput on the network is relatively high.
It is also worth noting that Globus allows each user to have up to three simultaneous transfers in progress, with additional transfers queued. If you submit more than three simultaneous transfer requests, the additional requests are queued while the three active requests are completed, and this queue time is also included in the Effective Transfer Rate calculation for those requests.
Globus does not preserve file permissions when performing a transfer. When you transfer files with Globus, their permissions are determined entirely by the destination endpoint’s configuration. There are still ways that you can control the permissions of the files created by Globus, on a destination endpoint, but they do not operate on information about the original file permissions.
An obvious question that arises is "Why doesn’t Globus preserve permissions?" This behavior is an unfortunate result of the fact that it is not entirely clear what preserving permissions means for some transfer tasks.
Ideally, given endpoints
user#B, with files in
user#A, then transferring those files back and forth between
user#B would not alter the permissions of those files.
So, if we submitted a transfer task, copy
user#B:/x/y/z, the file at
user#B:/x/y/z will have exactly
the same permissions as the original at
user#A:/p/q/r. Consider a second
transfer in the other direction, copy
Since this should share the same property as the previous transfer,
user#A:/p/q/r_prime should be completely
indistinguishable — there should be no way to tell which one is the original by
content or permissions.
But what if the user authenticates to
user#A as a user with read permissions to
user#A:/p/q/r, but not ownership? Then when the file is transferred back to
user#A:/p/q/r_prime, the ownership will have changed. On most systems, only the superuser can change the owner of
/p/q/r_prime to match
/p/q/r. This is the basic issue with attempting to preserve ownership for files.
Not all permissions settings are supported on all platforms. Consider what happens if
user#A:/p/q/r has UNIX octal permissions 0111 — anyone can execute the file, but no one can read or write it — and
user#B is a Windows endpoint. When the file is stored in Windows as
user#B:/x/y/z, it can’t be given these same permissions because Windows does not support execute-only files. When
user#B:/x/y/z is transferred to
user#A:/p/q/r_prime, the only way for the transfer task to know to restore the original permissions is to keep track of all permissions of files transferred by Globus in case they are transferred again. Even with that extra information, it is difficult to know exactly what to do: what should Globus do if the file has been altered, or had permissions added or removed?
What if the file is moved with scp from
user#C:/w/t/u and with Globus from
user#A:/p/q/r_prime? Because permissions schemes are not uniform across all platforms, and files may move locally or remotely by means other than the Globus service, we cannot guarantee the transitivity of permissions across a series of transfers.
Having stated that the permissions of your files cannot be consistently preserved by Globus for technical reasons, what recourse do you, as a user or endpoint administrator, have? Our team is always looking to improve Globus, and better permissions handling is on the To Do List. In the meantime, however, you can make some steps to better control your file permissions.
By default, the GridFTP server uses the system umask setting to determine the permissions of all files that it creates. There is an option, passed either through the command line as "-perms", or through the config file (by default, placed in /etc/gridftp.conf ) as a line "perms <value>", which can be used to further restrict the permissions of new files. The option is specified as a three digit octal integer, as typical UNIX permissions are, and is documented in the Globus Toolkit 5.2 release here.
"perms" does not override the umask, but is applied additively. Note that the "perms" option is written as a positive set of permissions bits, which are desired for new files, while the umask is a negative set of bits, which are forbidden. Since the GridFTP server attempts to create files with the "perms" permissions, the effective permissions of a new file are <PERMS> AND (NOT <UMASK>) for regular files, rather than the default of 0666 AND (NOT <UMASK>). "perms" will not alter directory permissions, so those should still be 0777 AND (NOT <UMASK>).
Because the "perms" value is ANDed together with the inverted umask, it cannot be used to apply wider permissions than the umask allows, but it can be used to further restrict access. For example, if the system umask is set to 0002, but you want to forbid world read access and group write access to files, you could set "perms" to 0042. The resulting permissions, in this case, would be 0042 AND (NOT 0002) = 0042 AND (0775) = 0040, as the umask forbids the world write permission granted by "perms".
Because Globus delegates operations to the endpoint’s filesystem without inspecting ACLs on the source or destination, you can leverage your endpoint’s support of ACLs to control permissions tightly. By setting ACLs on the destination such that they are applied to all new files in a directory tree, you can effectively set ACLs on the files created by the GridFTP server. GridFTP and Globus will never attempt to explicitly get or set the filesystem ACLs, effectively leaving their application up to the destination endpoint’s filesystem implementation. Since different filesystems and operating systems may implement ACLs differently, we do not provide explicit instructions for any particular local ACL setup.
Setting the umask explicitly is the only way to increase the permissions offered on files created by the GridFTP server. The most consistent and successful way to do this is to alter the Globus Connect Server init script to set the umask immediately before launching the GridFTP server. Most typically, the script is found in /etc/init.d/globus-connect-server
If you do not feel comfortable modifying the init script, this option is likely a bad choice for you. The init script is the only supported way of launching GridFTP for a Globus Connect Server installation, so damaging alterations to the script could prevent you from launching Globus Connect Server altogether. (In other words, choose this option at your own peril.)
The above techniques can be applied to Globus Connect Personal, but there are some caveats. Most notably, we do not officially support modified versions of the Globus Connect Personal client, so if you alter any files or configuration within the client application in order to achieve your desired permissions scheme, your endpoint will not necessarily qualify for support from Globus staff. At present, none of the forms of Globus Connect Personal support specifying "perms" to the GridFTP server.
If you are running Globus Connect Personal for Linux, you may have some success altering your personal umask setting before launching the application, as your umask should propagate down the process tree to the GridFTP server process. Likewise, if you are running Globus Connect Personal on Mac OS X, you may be successful setting your umask before launching the Globus Connect Personal app through the command line. These actions are not guaranteed to be successful based on the exact behavior of your platform. Because Windows does not support a umask equivalent, there is no way to replicate this behavior in Globus Connect Personal for Windows.
When supported by your platform, filesystem ACLs are respected, but they are not an option for all users.
In many situations, restricting read or write access to a file can be handled correctly using Globus controlled Read and Write permissions on a Shared Endpoint. This does not alter the underlying permissions of the files, but restricts permissions when using a Globus account to access the endpoint. Globus will deny users without the Read permission the rights to copy files or list directory contents, and denies users without the Write permission the rights to copy a file to the specified path or directory.
These permissions settings do not alter the underlying endpoint’s permissions scheme in any way, so users with local access to the endpoint may be able to bypass these permissions settings by accessing files directly. If you know that your files are only exposed via Globus, then this option may be right for you.
At present, Globus skips symlinks in a wide class of transfers. The reason for this is that there are several notions of correct behavior for transfers of symlinks, especially with respect to their interaction with path restrictions in an endpoint’s configuration. However, symlinks are not uniformly ignored, and in some actions, for which the behavior on symlinks is unambiguous, they will be followed.
This behavior is identical between Globus Connect Personal and Globus Connect Server.
When listing the contents of a directory, if the path includes symlinks, those symlinks will be followed. However, when the links are followed, they do not receive special treatment — to Globus, they are considered indistinguishable from the directories to which they are links. This is very similar to the treatment of symlinks when doing local directory listings (i.e. ls in most shells), in which the fact that a directory is a link is not necessarily exposed.
So, if you have a symlink
/tmp/myhome → /home/username/, then when you attempt to list the contents of
/tmp/myhome/Desktop/, Globus will return a list of contents of
/home/username/Desktop/. Globus will not give any indication that /tmp/myhome is a symlink; there is no path rewriting or other indication that
/tmp/myhome is anything but an ordinary directory whose contents happen to be identical to
When doing a recursive directory transfer, all symlinks in the directory tree are ignored. The one and only exception to this rule is the root of a directory transfer.
Consider the previous example,
/tmp/myhome → /home/username/. Doing a recursive directory transfer with a root directory of
/tmp/myhome will transfer all of the contents of
/home/username/, following the symlink
/tmp/myhome. However, a recursive directory transfer on
/tmp will skip
/tmp/myhome, not creating it as a directory, link, or file on the destination. Furthermore, this skipping behavior does not trigger any errors, faults, or warnings in the transfer history, as it is not considered an error condition.
Single file transfers follow the same basic rules that directory transfers do, in that they dereference symlinks to their destination files, and create the link on the destination as an ordinary file.
If I have a link,
/a/b/c → /p/q/r on my filesystem to an ordinary file, then transferring
/a/b/c to another endpoint will behave as though the contents of
/p/q/r were stored in
/a/b/c, not giving any special treatment to
/p/q/r on account of its status as a link.
Globus does not follow symlinks when doing file or directory deletions. However, following the semantics of a typical UNIX rm command, Globus will unlink symbolic links by deleting them during a directory or file removal.
Globus endpoint configuration supports restricting the parts of the filesystem that can be accessed via Globus. In Globus Connect Server and GridFTP this corresponds to the RestrictPaths and SharingRestrictPaths options. By default, these settings apply to non-symlinked files and directories, not allowing access when a symlink points outside of the explicitly allowed components of the filesystem.
This behavior prevents abusive symlinks from breaking out of the path restrictions. Consider the case of an endpoint which only allows access to
/p/q/r/, and a symlink
/p/q/r/root → /. If symlinks are followed irrespective of the path settings,
/p/q/r/root/home/ would be accessible, even though
/home/ is not included.
If you trust users with access to an endpoint not to create this kind of exploitative symlink, you can override this behavior on Globus Connect Server endpoints with the rp-follow-symlinks option to the GridFTP server. This option is not readily available with Globus Connect Personal installations.
Many of the pages in the Globus application can be linked with parameters that allow the page to open pre-configured for your needs. When you link to a Globus application page, the application will ensure that the user is logged in to the Globus website (and prompt the user to authenticate if the user is not already logged in). If you have a specific use case or application feature that isn’t covered in this document please let us know at firstname.lastname@example.org.
What follows is a list of pages that are configurable.
File Manager (https://app.globus.org/file-manager)
The File Manager page provides the following parameters to preselect the two sides of the File Manager page:
origin_id - represents the id of the endpoint for the left hand side of the page.
origin_path - represents the path of the endpoint for the left hand side of the page.
destination_id - represents the id of the endpoint and path for the left hand side of the page.
destination_path - represents the path of the endpoint for the right hand side of the page.
Endpoints which are specified on the URL in this fashion will require activation if they are not already activated by the user. You can determine the parameters and link by setting the desired endpoint(s) on the File Manager page and selecting "get link" from the function menu.
Sends the user to the File Manager page with the endpoint
Globus Tutorial Endpoint 1selected on the left side.
Sends the user to the File Manager page with the endpoint
Globus Tutorial Endpoint 1selected on the left side and
Globus Tutorial Endpoint 2on the right.
Sends the user to the File Manager page with the endpoint
Globus Tutorial Endpoint 1and the path
/share/selected on the left side.
The groups page can be set up to automatically view a specific group by specifying the group id in the URL. For example the "Demo Group" group has an id of
7e183f30-e177-11e5-ba63-22000ab80e73, so the URL to automatically link to this group is "https://app.globus.org/groups/7e183f30-e177-11e5-ba63-22000ab80e73". Groups with visibility policies that prevent visibility to non Globus members will display an error page if the user utilizing the link does not meet the visibility requirements. You can determine the group id and link by selecting the desired group on the groups page and clicking the "Get Shareable Link" button. After clicking the "Get Shareable Link" button, the URL for the Shareable Link will be copied to the system clipboard and available for use.
When accessed without parameters, the groups page (https://app.globus.org/groups) displays a list of the groups that the user is a member of, has been invited to, or has requested membership in. The groups page will also show a notification icon for groups that are awaiting action by the user.
For a specific endpoint, bring the endpoint up in the File Manager. If you do not have a current activation on the endpoint selected, you will be prompted to activate on it. To see all endpoints that the user has credentials that are being used for active transfers and about to expire, go to https://app.globus.org/endpoints?reactivatable=1&scope=in-use.
There are several reasons a transfer may stall. You may want to start by seeing what Globus is doing with the request.
On the web, go to the "View Activity" page
Click on ID of specific transfer (probably the one at the top of the list)
Click "View Event Log".
Look through the progress messages and see if any of them indicate an error, such as an expired credential or quota exceeded.
In the CLI, the globus task show command will give you a high level overview of the progress of the task. If you run globus task show and find the task is not progressing, run the globus task event-list command. The following will show the most recent 5 events for the specified task:
$ globus task event-list --limit 5 <TaskID>
If the globus task show command shows the task is inactive, you may need to refresh your credentials. See this FAQ item for information on how to do this.
Globus Connect Personal for Windows may have a problem automatically locating your user directory. Try manually entering "/" (just forward slash, no quotes) as the directory, and then you should be able to browse. Your c: drive would be "/cygdrive/c", and your user directory would be either "/cygdrive/c/users/<username>" or "/cygdrive/c/documents" and "settings/<username>".
Globus will make every attempt to complete a Transfer request but sometimes may be unable to do so, based on factors outside of our control. For example, occasionally, an endpoint may stop responding (due to server failure, network issues, etc.). In this case, you will receive notifications about the error and should follow up with your system administrator.
A more common cause is "credential expiration". This means that Globus is no longer authorized to access the endpoint on your behalf to execute/manage your transfer. When this happens, Globus will send you an e-mail notification and suspend the transfer task until you renew your credentials for the endpoint. If the credentials are not renewed (i.e. the endpoint is not reactivated) within 3 days of the notification being sent, the transfer task will automatically expire.
Instructions for renewing your credentials are available here.
If your transfer task has failed you should first look at the last few events in the event log to identify any problems needing human intervention (quota exceeded, out-of-disk space, etc.)
After fixing problems you can resubmit your task as follows: . Using the Web GUI, go to the Start Transfer page, select the "more options" link at the bottom, and select the "only transfer new or changed files where the checksum is different" and "verify file integrity after transfer" checkboxes. . Using the CLI, rerun the transfer command with the --verify-checksum and --sync-level checksum options
On different filesystems, directory names and filenames may be restricted to certain characters. For instance, following character are reserved on ext2, ext3, ext4, hfs, FAT, NTFS:
|ext2, ext3, ext4 (Linux)|
|hfs (Mac OS X)|
|FAT, NTFS (MS Windows)|
||||vertical bar or pipe|
If a file or a directory with one of the characters <>:"\|?* is copied from Linux to MS Windows, then MS Windows will return the error message "The filename, directory name, or volume label syntax is incorrect" and refuse to create the file or the directory.
Also the filesystems FAT, NTFS, hfs are not case sensitive. It means that if two files on ext2, ext3, ext4 are different by case and they are transferred to a non-case sensitive filesystem they will be copied into one file.
Globus tunes transfers based on the number of servers of the source and destination endpoints, the location of the endpoints, and the performance options configured for each managed endpoint in your provider subscription. See the CLI globus endpoint update man page for more details on the --network-use performance option.
Globus can load balance across servers when a task has more than one file (you might need about 10 files or so in a task.) Also, multiple tasks running at once might be load balanced across servers; the first task begins on a random server, with subsequent tasks assigned to other servers in a round robin fashion.
Globus can be used to transfer data to/from anonymous GridFTP servers (but not anonymous FTP servers). The service supports transfers between two anonymous GridFTP servers or between an anonymous GridFTP server and a standard GridFTP server. Follow the instructions below for your particular case:
Between two anonymous GridFTP servers:
$ endpoint-add anon-endpoint1 -p ftp://host1:port $ endpoint-add anon-endpoint2 -p ftp://host2:port $ endpoint-activate anon-endpoint* -m myproxy.globusonline.org $ scp anon-endpoint1:/path_to_src_file anon-endpoint2:/path_to_dst_file
Between a anonymous GridFTP server and a standard GridFTP server:
$ endpoint-add anon-endpoint -p ftp://host:port $ endpoint-add std-endpoint -p gsiftp://host:port $ endpoint-activate anon-endpoint -m myproxy.globusonline.org $ endpoint-activate std-endpoint -m myproxy-server-associated-with-std-gridftp-server $ scp anon-endpoint:/path_to_src_file std-endpoint:/path_to_dst_file
Globus will store a basic summary of a user’s task history indefinitely. Task details such as Task ID, Task Status, Source and Destination Endpoint Display Names for the task, Task Transfer Settings, and many other details are preserved indefinitely. Certain specific details for tasks are only stored for 30 days and then these details expire. These task details include the faults logged for the task as well as the status information concerning the individual files and directories involved in the transfer task.
From the web interface, users can access their task history at the following URL:
The task details under the "Event Log" tab for a given task are those that will expire after 30 days. All other details for a task will remain available indefinitely.
$ globus task show TASK_ID
The above will show the basic details for the task that will be preserved indefinitely. Users can see more detailed information on their task by using the
globus task event-list command like so:
$ globus task event-list TASK_ID
The details returned by the
globus task event-list command show the job details that are preserved for only 30 days, and thus this command will return nothing for tasks more than 30 days old.
This is often a symptom of a firewall policy problem that results in interference with data channel traffic between the source and destination endpoints involved in the transfer. A good first step in troubleshooting this sort of issue is to ensure that firewall policy for both the source and destination endpoints is consistent with what Globus shows in the firewall policy requirements for the software that is being used to run that endpoint. The firewall policy requirements for Globus Connect Server can be found here. The firewall policy requirements for Globus Connect Personal can be found here.
How do I transfer between two endpoints that require different institutional credentials via CILogon?
When a user activates an endpoint with an institutional credential that uses CILogon, the credential is automatically associated with all endpoints that require a credential that uses CILogon. When a user attempts to transfer to or from an endpoint that requires a different CILogon institutional credential, an error will occur on this endpoint.
To mitigate this issue, the user must find the endpoint with the error at https://app.globus.org/endpoints and choose the "Extend Activation" option on the Overview tab. This will prompt a login flow, allowing the user to choose the correct institution.
For example, Endpoints A and B require institutional credentials using CILogon from Institutions A and B, respectively. If a user has activated Endpoint A and then attempts a transfer to/from Endpoint B, an error will be reported indicating that Endpoint B has incorrect credentials or that the user is not found. The user can then use the "Extend Activation" option for Endpoint B in order to login with credentials from Institution B to associate the correct credentials with Endpoint B.
By default, when the storage quota is exceeded during your transfer, Globus will periodically retry the transfer. The Event Log on the web app Activity page will warn "storage quota exceeded". If no progress on your transfer can be made in three days, the transfer will fail and you will then be notified via email. However, if you select the option to fail on quota error, your transfer will fail immediately rather than retry for three days, and you will be notified immediately via email.
What happens to my transfer if I don’t have permission to read a file or directory or if a file or directory cannot be found?
There may be cases where your transfer includes source files or directories that you are not permitted to read or that have been deleted after the transfer has been submitted but before the transfer has completed. By default, the transfer will pause on the first occurrence of a permission denied or file not found error. Globus will then periodically retry the transfer. If no transfer progress can be made in three days, the transfer will fail, and you will then be notified via email.
Alternatively, you can select the option to skip over source files and directories with permission denied or file not found errors. Be aware that when this option is selected, a successfully completed transfer request does not necessarily mean all files and directories have been transferred. The status of the transfer task and the email notification will specify the number of paths skipped due to permission denied or file not found errors. If that number is greater than zero, always review the list of skipped paths, which is available via the API or the web app.
You can use globus-timer-cli, which is a command line client in beta release, to submit a job that will reliably request periodic transfers with a defined start time. For example, you can set up a daily, automatic sync to your back-up system or delay the start of a big transfer until midnight on Saturday or move data to archival storage once a week. Status of such scheduled jobs can be monitored and managed as well, and, unlike a cron job, your recurring transfers don’t depend on the availability of your system.
Once you install globus-timer-cli and authenticate, you can submit a transfer command that defines your source and destination, the files you wish to transfer, and a time interval. Optionally, you can submit the list of files and directories you’d like to transfer as a CSV formatted file, set a delayed start time, sync files, or set other transfer options, such as encryption.