NAME

transfer - Advanced file and directory transfer command

SYNOPSIS

transfer [OPTIONS] — SOURCE-ENDPOINT/FILE DESTINATION-ENDPOINT/FILE

transfer [OPTIONS] — SOURCE-ENDPOINT/DIRECTORY/ DESTINATION-ENDPOINT/DIRECTORY/ -r

transfer [OPTIONS] < INPUT-LINE(s)

transfer --generate-id

DESCRIPTION

transfer creates a task that copies files and/or directories between endpoints. If multiple input lines are given over stdin, all lines must have the same source and destination endpoint.

The transfer command will always place the destination files in a consistent, deterministic location. The contents of a source directory will be placed inside the destination directory. A source file will be copied to the destination file path, which must not be an existing directory. All intermediate / parent directories on the destination will be automatically created if they don’t exist.

Paths relative to a user’s home directory may be specified by using "/~/path" or "/~someuser/path" syntax.

If the files or directories given as input are symbolic links, they are followed. However, no other symbolic links are followed and no symbolic links are ever created on the destination.

transfer will print its task id in this format: "Task ID: <taskid>". This task id may be passed to other task management commands.

Globbing ("*") is not supported.

Automatic Endpoint Activation

If an endpoint is not activated, this command will attempt to auto-activate it. Otherwise, the command will fail, and direct the user to run endpoint-activate and/or display a web URL for OAuth activation.

Endpoint Naming

This command supports the following endpoint identification formats:

  • UUID: "7865988a-aeb3-4e55-b8cd-938c258e7854"

  • Bookmark: "^test_bookmark". The bookmark name must be prepended with a "^" character. The bookmark name may have URL-escaped characters. If the bookmark name contains a "/", it must be URL-escaped as "%2F".

    Note:When a bookmark is used, the bookmark’s path is prepended to the input path.
  • Legacy Name: "bob#test_endpoint". This contains the user who owns the endpoint and the endpoint legacy name.

  • Unqualified Legacy Name: "test_endpoint". The user name defaults to the current logged in user.

Input Lines

A single input line may be given on the command line, after a "--" option break. Alternatively, one or more lines may be written to standard input as a "batch input" method.

An input line has the following syntax:

File

source-endpoint/file dest-endpoint/file

Directory

source-endpoint/dir/ dest-endpoint/dir/ -r

An input line must contain a source and destination endpoint and path. An input line may also contain the following options:

-r

Recursively transfer directory. The contents of the source directory, including all subdirectories, are transferred to the destination directory. The destination directory (and its parent directories) will be created if it does not exist. Both the source and destination paths must end with a slash ("/") character.

Handling Special Characters

The simple (single line) input method assumes paths given on the command line are URL-encoded, which is the common convention for Globus CLI commands.

The batch input method supports the following:

URL Escaped

If an endpoint/path token does not begin with a double quote ("), it is assumed to have URL-escaped bytes. Thus, sequences like "%20" and "%25" will be transformed to a space (" ") and literal "%" byte, respectively.

Quoted Bytes

If an endpoint/path token begins with a double quote ("), it is assumed to be a sequence of raw bytes terminated by a final double quote ("). Embedded spaces and "%" characters are allowed and preserved. An embedded double quote (") can not be used in this format.

OPTIONS

These options apply to the transfer command itself:

-s SYNC-LEVEL

Synchronize, i.e. only copy new files or files that have been changed. Note that this option alone does not turn on deletion of any files. (See the --delete option for that).

Valid sync levels are:

0

Copy files that do not exist at the destination

1

Copy files if the size of the destination does not match the size of the source

2

Copy files if the timestamp of the destination is older than the timestamp of the source

3

Copy files if checksums of the source and destination do not match

Each sync level includes the comparison criteria from lower-numbered levels. For instance, if level 3 is specified and the source and destination file sizes are different or the source file has a newer timestamp, the file will be transfered and a checksum operation is not even attempted.

--label LABEL

Set the task’s label. The label can be displayed and searched for in commands such as status and details. See labels(7) for more details on labels.

-d DEADLINE

Set the task’s deadline. If the task has not completed by the deadline it will be automatically canceled. A suffix of "m", "h", or "d" may be specified to indicate minutes, hours, or days. If a unit suffix is not given it is assumed to be minutes. Example: "30m", "4h", "1d".

If a deadline is not explicitly set, Globus Online will give the task a flexible deadline that is automatically extended as long as it is making progress.

--encrypt

Encrypt the data channel during file transfer. This requires gsiftp and will not work on anonymous GridFTP servers. By default, data channel connections are securely authenticated during initialization, but the file data is sent directly (unencrypted).

--preserve-mtime

Preserve modification time of files that were transferred.

Note:The destination server must be a recent GridFTP version, otherwise the transfer will have errors and not complete.
--verify-checksum

After each file is transferred, verify that the source and destination checksums match. If they don’t, retransfer the entire file and try checksum verification again. Keep trying until verification succeeds. This is currently the default.

--no-verify-checksum

Disables the --verify-checksum option; no checksum verification is performed.

--delete

Delete extraneous files (and directories) in the destination directory. Only applies to recursive directory transfers. For example, a transfer of /some/source/dir/ to /tmp/mydir/ will ensure /tmp/mydir/ is a perfect clone of /some/source/dir/. All other files in /tmp outside of /tmp/mydir/ are left alone.

Reliable Submission

These options are intended for idempotent request submission in the presence of network failures. For example, an automated script should call --generate-id once and then call transfer --taskid=TASKID in a loop, retrying any SSH failures (exit code 255).

--taskid TASKID

Specify a taskid to use instead of having one automatically created. TASKID must be an id previously returned by --generate-id.

If TASKID has already been successfully submitted, transfer will do nothing and return successfully. If TASKID has not yet been submitted successfully, transfer will delete any previous partial request and replace it with the new input.

--generate-id

If this option is present, transfer will print a new task id that may be used as an argument to --taskid and exit.

Performance Tuning

These options are for advanced users who want to override the automatic tuning algorithm.

--perf-p NUM

Set parallelism level (Data streams per control channel).

Note:If this value exceeds the maximum parallelism limit set on the source or destination endpoint, the lowest limit will be used.
--perf-cc NUM

Set concurrency level (Number of control channels).

Note:If this value exceeds the maximum concurrency limit set on the source or destination endpoint, the lowest limit will be used.
--perf-pp NUM

Set pipeline depth (Files in flight per control channel). Allowed values are 1-32. This is useful for small files and/or high latency connections.

EXAMPLES

Transfer a file to a user’s home directory, and assign a task label:

$ transfer --label 'demo 1' -- go#ep1/share/godata/file1.txt xsede#forge/~/mynewfile.txt

Transfer a directory, recursively, using encrypted data channels:

$ transfer --encrypt -- go#ep1/share/godata/  go#ep2/~/godatacopy/ -r

Use the batch input method to submit multiple lines. Synchronize a directory and a file, using checksums (sync level 3) for comparison:

$ transfer -s 3
demo#tiger/etc/forest/ demo#bear/etc/woods/ -r
demo#tiger/etc/specialfile demo#bear/etc/specialfile-copy
<EOF>

© 2010- The University of Chicago Legal