1. Overview

Transfer and delete are asynchronous operations, and result in a background task being created. Both require a submission_id when submitted, and return a task_id on successful submission. The submission_id is used to safely re-submit in case of a network failure, and the task_id is used to monitor the progress of the task (see Task Monitoring API). The submission id and task id are not the same.

Before a transfer or delete task is submitted, the source (and destination for transfers) must be "activated" with user credentials, so the transfer service can perform operations on their behalf. See the Endpoint Activation API for details.

2. Document Types

2.1. Transfer and Delete Documents

The "transfer" and "delete" documents share several common fields and have similar structure. Both contain a submission id, an endpoint or endpoints to operate on, a list of options, and a list of paths. The key difference is that a transfer submission contains a pair of endpoints and a list of pairs of source destination paths, while a delete submission contains a single endpoint and a list of paths to delete. There are also several options that only make sense for one or the other, like encrypt_data which is only available for transfers.

2.1.1. Endpoint Format

Endpoints should be specified using the endpoint id, but for backward compatibility the endpoint canonical_name is also accepted. Using the canonical_name is deprecated, and will be removed in a future release. All new applications must use endpoint id, and old applications should be updated to use the id.

2.1.2. Path Format

Paths must use the path component separator forward slash (/), commonly used on Unix and Mac OSX, not backslash. This is true even if one of the endpoints happens to be a GridFTP server running on a windows machine, because the GridFTP protocol has standardized on forward slash. Directory paths must end in forward slash, and file paths must not end in forward slash.

For transfers, both source and destination must be directories or both must be files. The destination must be the exact path to create or overwrite on the destination, not the parent directory. This is unlike the unix cp and scp commands, which will either copy into the destination directory or create the directory, depending on whether or not it already exists.

See also Path Encoding.

2.1.3. Common Transfer and Delete fields

Field Name JSON Type Description

DATA_TYPE

string

Either "transfer" or "delete" to indicate the submission type.

submission_id

string

Id acquired from /submission_id.

label

string

Optional user specified string to help identify the Transfer or delete task. See the CLI label reference for details. Note that label can no longer be updated on completed tasks, it can only be set when submitting the task and while the task is still running.

notify_on_succeeded

boolean

If true and the user has notification enabled, send a notification email when the transfer completes with status SUCCEEDED. Default true.

notify_on_failed

boolean

If true and the user has notification enabled, send a notification email when the transfer completes with status FAILED. Default true.

notify_on_inactive

boolean

If true and the user has notification enabled, send a notification email when the transfer enters status INACTIVE, e.g. from activation credentials expiring. Default true.

2.2. Transfer specific fields

Field Name JSON Type Description

source_endpoint

string

Id of the endpoint to transfer data from.

destination_endpoint

string

Id of the endpoint to transfer data to.

DATA

list

List of transfer_item documents containing source and destination paths.

encrypt_data

boolean

If true, encrypt the data channel. Default false. If either the source or destination endpoint, or for shared endpoints the source or destination host endpoint, has force_encryption set, the data channel will be encrypted even if this is set to false.

sync_level

integer

If non-null, synchronize files, i.e. copy only files that are new or have been changed. The level must be an integer in the 0-3 range, and it controls what checks are performed to determine if a file should be copied. Higher levels include the checks from lower levels:

0

Copy files that do not exist at the destination.

1

Copy files if the size of the destination does not match the size of the source.

2

Copy files if the timestamp of the destination is older than the timestamp of the source.

3

Copy files if checksums of the source and destination do not match. Files on the destination are never deleted.

verify_checksum

boolean

After transfer, verify that the source and destination file checksums match. If they don’t, re-transfer the entire file and keep trying until it succeeds. Default false.

preserve_timestamp

boolean

Preserve file modification time. Default false.

delete_destination_extra

boolean

Delete extraneous files in the destination directory. Only applies for recursive directory transfers. Default false.

2.3. Transfer item fields

Field Name JSON Type Description

DATA_TYPE

string

Always has value "transfer_item" to indicate this document type.

source_path

string

Source path of file or directory to transfer.

destination_path

string

Distination path of file or directory to transfer.

recursive

boolean

Must be true for directory paths, false for files.

2.4. Delete specific fields

Field Name JSON Type Description

endpoint

string

Id of the endpoint to delete data on.

DATA

list

List of delete_item documents containing paths to delete.

recursive

boolean

Delete directory contents recursively. Required if any of the delete items point to a directory. Note that unlike transfer submissions, this is a top level field and can’t be specified per item. Default false.

ignore_missing

boolean

Don’t generate errors for non existent files and directories. Default false.

interpret_globs

boolean

If false (the default), every character in the entire path is treated literally.

If true, shell glob characters (*, ?, [, and ]) in the last path component are interpreted as a pattern, unless they are escaped by a preceding backslash (\). For consistency, the rest of the path (e.g. everything but the final component) also has any backslash escapes removed, so a literal backslash anywhere in the path must be escaped. Like UNIX, glob characters will not match files or directories that start with a period (.); a literal period character is required in such cases. The special directory entries . and .. are never matched.

2.5. Delete item fields

Field Name JSON Type Description

DATA_TYPE

string

Always has value "delete_item" to indicate this document type.

path

string

Path of file or directory to delete. Directory paths must end in forward slash "/".

3. Operations

3.1. Get a submission id

Get a submission id, required when submitting transfer and delete tasks. Note that this is different than the task id returned by the submit operations.

URL

/submission_id

Method

GET

Response Body

{
  "value": "55379aa2-d9a2-11e5-976c-22000b9da45e",
  "DATA_TYPE": "submission_id"
}

3.2. Submit a transfer task

URL

/transfer

Method

POST

Request Body

Transfer document

Response Body

{
  "DATA_TYPE": "transfer_result",
  "task_id": "994f289b-d9a2-11e5-976c-22000b9da45e",
  "submission_id": "f5cc79fd-dfc8-475e-b726-b96c734a484d",
  "code": "Accepted",
  "message": "The task was submitted successfully",
  "resource": "/transfer",
  "request_id": "ABCdef789",
}

3.2.1. Result codes

Code HTTP Status Description

Accepted

202

The transfer or delete submission has been accepted and a task has been created and queued for execution

Duplicate

200

A transfer or delete submission with the same submission id has already been accepted. In case of a network error, the client may not know whether the submission was successful, and won’t know the task id if it was successful. If the client re-submits and gets this code, it means the initial request was successful, and the task id in the response can be used. If this is received on a request that is not a retry, the client is likely not getting a submission id correctly for each submission.

3.2.2. Errors

Code HTTP Status Description

ClientError.BadRequest or BadRequest

400

There is some problem in the request document, see the message for details.

PermissionDenied

403

The user does not have permission to use one of the endpoints in the request.

ServiceUnavailable

503

The service is down for maintenance.

NoCredException

409

One or more endpoints in the request are not activated or have expired activation. Activate the affected endpoint(s) and retry the operation.

Conflict

409

The source or destination endpoint in the request is a shared endpoint and its host endpoint has been deleted.

3.3. Submit a delete task

Response and error codes are the same as for transfer submission.

URL

/delete

Method

POST

Request Body

Delete document

Response Body

{
  "DATA_TYPE": "delete_result",
  "task_id": "cfec1219-3f78-4c02-8fe6-83d5ba01f3a2",
  "submission_id": "499f18bf-9ea6-48e6-a030-98e0f29512b5",
  "code": "Accepted",
  "message": "The task was submitted successfully",
  "resource": "/delete",
  "request_id": "ABCdef789",
}

© 2010- The University of Chicago Legal