transfer - Advanced file and directory transfer command
transfer [OPTIONS] — SOURCE-ENDPOINT/FILE DESTINATION-ENDPOINT/FILE
transfer [OPTIONS] — SOURCE-ENDPOINT/DIRECTORY/ DESTINATION-ENDPOINT/DIRECTORY/ -r
transfer [OPTIONS] < INPUT-LINE(s)…
transfer creates a task that copies files and/or directories between endpoints. If multiple input lines are given over stdin, all lines must have the same source and destination endpoint.
The transfer command will always place the destination files in a consistent, deterministic location. The contents of a source directory will be placed inside the destination directory. A source file will be copied to the destination file path, which must not be an existing directory. All intermediate / parent directories on the destination will be automatically created if they don’t exist.
Paths relative to a user’s home directory may be specified by using "
If the files or directories given as input are symbolic links, they are followed. However, no other symbolic links are followed and no symbolic links are ever created on the destination.
transfer will print its task id in this format: "Task ID: <taskid>". This task id may be passed to other task management commands.
Globbing ("*") is not supported.
Automatic Endpoint Activation
If an endpoint is not activated, this command will attempt to auto-activate it. Otherwise, the command will fail, and direct the user to run endpoint-activate and/or display a web URL for OAuth activation.
This command supports the following endpoint identification formats:
"^test_bookmark". The bookmark name must be prepended with a
"^"character. The bookmark name may have URL-escaped characters. If the bookmark name contains a "/", it must be URL-escaped as "%2F".
"bob#test_endpoint". This contains the user who owns the endpoint and the endpoint legacy name.
Unqualified Legacy Name:
"test_endpoint". The user name defaults to the current logged in user.
A single input line may be given on the command line, after a "--" option break. Alternatively, one or more lines may be written to standard input as a "batch input" method.
An input line has the following syntax:
source-endpoint/dir/ dest-endpoint/dir/ -r
An input line must contain a source and destination endpoint and path. An input line may also contain the following options:
Recursively transfer directory. The contents of the source directory, including all subdirectories, are transferred to the destination directory. The destination directory (and its parent directories) will be created if it does not exist. Both the source and destination paths must end with a slash ("/") character.
Handling Special Characters
The simple (single line) input method assumes paths given on the command line are URL-encoded, which is the common convention for Globus CLI commands.
The batch input method supports the following:
- URL Escaped
If an endpoint/path token does not begin with a double quote ("), it is assumed to have URL-escaped bytes. Thus, sequences like "%20" and "%25" will be transformed to a space (" ") and literal "%" byte, respectively.
- Quoted Bytes
If an endpoint/path token begins with a double quote ("), it is assumed to be a sequence of raw bytes terminated by a final double quote ("). Embedded spaces and "%" characters are allowed and preserved. An embedded double quote (") can not be used in this format.
These options apply to the transfer command itself:
- -s SYNC-LEVEL
Synchronize, i.e. only copy new files or files that have been changed. Note that this option alone does not turn on deletion of any files. (See the --delete option for that).
Valid sync levels are:
Copy files that do not exist at the destination
Copy files if the size of the destination does not match the size of the source
Copy files if the timestamp of the destination is older than the timestamp of the source
Copy files if checksums of the source and destination do not match
Each sync level includes the comparison criteria from lower-numbered levels. For instance, if level 3 is specified and the source and destination file sizes are different or the source file has a newer timestamp, the file will be transfered and a checksum operation is not even attempted.
- --label LABEL
Set the task’s label. The label can be displayed and searched for in commands such as status and details. See labels(7) for more details on labels.
- -d DEADLINE
Set the task’s deadline. If the task has not completed by the deadline it will be automatically canceled. A suffix of "m", "h", or "d" may be specified to indicate minutes, hours, or days. If a unit suffix is not given it is assumed to be minutes. Example: "30m", "4h", "1d".
If a deadline is not explicitly set, Globus Online will give the task a flexible deadline that is automatically extended as long as it is making progress.
Encrypt the data channel during file transfer. This requires gsiftp and will not work on anonymous GridFTP servers. By default, data channel connections are securely authenticated during initialization, but the file data is sent directly (unencrypted).
Preserve modification time of files that were transferred.
After each file is transferred, verify that the source and destination checksums match. If they don’t, retransfer the entire file and try checksum verification again. Keep trying until verification succeeds. This is currently the default.
Disables the --verify-checksum option; no checksum verification is performed.
Delete extraneous files (and directories) in the destination directory. Only applies to recursive directory transfers. For example, a transfer of /some/source/dir/ to /tmp/mydir/ will ensure /tmp/mydir/ is a perfect clone of /some/source/dir/. All other files in /tmp outside of /tmp/mydir/ are left alone.
These options are intended for idempotent request submission in the presence of network failures. For example, an automated script should call --generate-id once and then call transfer --taskid=TASKID in a loop, retrying any SSH failures (exit code 255).
- --taskid TASKID
Specify a taskid to use instead of having one automatically created. TASKID must be an id previously returned by --generate-id.
If TASKID has already been successfully submitted, transfer will do nothing and return successfully. If TASKID has not yet been submitted successfully, transfer will delete any previous partial request and replace it with the new input.
If this option is present, transfer will print a new task id that may be used as an argument to --taskid and exit.
These options are for advanced users who want to override the automatic tuning algorithm.
- --perf-p NUM
Set parallelism level (Data streams per control channel).
- --perf-cc NUM
Set concurrency level (Number of control channels).
- --perf-pp NUM
Set pipeline depth (Files in flight per control channel). Allowed values are 1-32. This is useful for small files and/or high latency connections.
Transfer a file to a user’s home directory, and assign a task label:
$ transfer --label 'demo 1' -- go#ep1/share/godata/file1.txt xsede#forge/~/mynewfile.txt
Transfer a directory, recursively, using encrypted data channels:
$ transfer --encrypt -- go#ep1/share/godata/ go#ep2/~/godatacopy/ -r
Use the batch input method to submit multiple lines. Synchronize a directory and a file, using checksums (sync level 3) for comparison:
$ transfer -s 3 demo#tiger/etc/forest/ demo#bear/etc/woods/ -r demo#tiger/etc/specialfile demo#bear/etc/specialfile-copy <EOF>