Task Submission
1. Overview
Transfer and delete are asynchronous operations, and result in a background task being created. Both require a submission_id when submitted, and return a task_id on successful submission. The submission_id is used to safely re-submit in case of a network failure, and the task_id is used to monitor the progress of the task (see Task Monitoring API). The submission id and task id are not the same.
By default, before a transfer or delete task is submitted, the source (and destination for transfers) must be "activated" with user credentials, so the transfer service can perform operations on their behalf. See the Endpoint Activation API for details.
2. Document Types
2.1. Transfer and Delete Documents
The "transfer" and "delete" documents share several common fields and have similar structure. Both contain a submission id, an endpoint or endpoints to operate on, a list of options, and a list of paths. The key difference is that a transfer submission contains a pair of endpoints and a list of pairs of source destination paths, while a delete submission contains a single endpoint and a list of paths to delete. There are also several options that only make sense for one or the other, like encrypt_data which is only available for transfers.
2.1.1. Endpoint Format
Endpoints should be specified using the endpoint id, but for backward compatibility the endpoint canonical_name is also accepted. Using the canonical_name is deprecated, and will be removed in a future release. All new applications must use endpoint id, and old applications should be updated to use the id.
2.1.2. Path Format
Paths must use the path component separator forward slash (/), commonly used on Unix and Mac OSX, not backslash. This is true even if one of the endpoints happens to be a GridFTP server running on a windows machine, because the GridFTP protocol has standardized on forward slash.
See also Path Encoding.
2.2. Common Transfer and Delete fields
Field Name | JSON Type | Description |
---|---|---|
DATA_TYPE |
string |
Either "transfer" or "delete" to indicate the submission type. |
submission_id |
string |
Id acquired from /submission_id. |
label |
string |
Optional user specified string to help identify the task. The label is opaque; no specific form is required. The maximum length is 128 characters. The allowed characters are:
|
notify_on_succeeded |
boolean |
If true and the user has notification enabled, send a notification email when the transfer completes with status SUCCEEDED. Default true. |
notify_on_failed |
boolean |
If true and the user has notification enabled, send a notification email when the transfer completes with status FAILED. Default true. |
notify_on_inactive |
boolean |
If true and the user has notification enabled, send a notification email when the transfer enters status INACTIVE, e.g. from activation credentials expiring. Default true. |
skip_activation_check |
boolean |
If true, allow task submission if the endpoint(s) aren’t currently activated. Default false. |
deadline |
string |
ISO 8601 datetime string. If non-null, cancel the task if it’s not complete at the specified time. The default (null deadline) behavior for transfer tasks is to have a flexible deadline, which allows the task to continue indefinitely, as long as it is making progress. Delete tasks default to a deadline of 24 hours after submission. |
2.3. Transfer specific fields
Field Name | JSON Type | Description |
---|---|---|
source_endpoint |
string |
Id of the endpoint to transfer data from. |
destination_endpoint |
string |
Id of the endpoint to transfer data to. |
DATA |
list |
One or more items to transfer. See Transfer Submission for details. |
filter_rules |
list |
One or more filter_rules to apply during recursive expansion. See filter_rules for details. |
encrypt_data |
boolean |
If true, encrypt the data channel. Default false. If either the source or destination endpoint, or for shared endpoints the source or destination host endpoint, has force_encryption set, the data channel will be encrypted even if this is set to false. |
sync_level |
integer |
If non-null, synchronize files, i.e. copy only files that are new or have been changed. The level must be an integer in the 0-3 range, and it controls what checks are performed to determine if a file should be copied. Higher levels include the checks from lower levels:
|
verify_checksum |
boolean |
After transfer, verify that the source and destination file checksums match. If they don’t, re-transfer the entire file and keep trying until it succeeds. Default false. |
preserve_timestamp |
boolean |
Preserve modification time of files and symlinks. Default false. Directory times are not preserved because filesystems typically override them. |
delete_destination_extra |
boolean |
Delete files, directories, and symlinks on the destination endpoint which don’t exist on the source endpoint or are a different type. Only applies for recursive directory transfers. Default: false. |
recursive_symlinks |
string |
(beta) This field is only supported by certain endpoints. How to process symbolic links (symlinks) found in the contents of recursive directory transfer_items. The default is "ignore". Valid values are:
|
skip_source_errors |
boolean |
Skip files or directories on the source endpoint that hit PERMISSION_DENIED or FILE_NOT_FOUND errors. |
fail_on_quota_errors |
boolean |
Cancel the task if QUOTA_EXCEEDED errors are hit. |
perf_cc, perf_p, perf_pp |
integer |
If non-null, a hint for what concurrency, parallelism, and pipelining value to use for the task. (experimental) |
perf_udt |
boolean |
If true, use the UDT transfer protocol. (experimental) |
2.4. transfer_item
Field Name | JSON Type | Description |
---|---|---|
DATA_TYPE |
string |
Always the value "transfer_item" |
source_path |
string |
Source path of file or directory |
destination_path |
string |
Destination path of file or directory |
recursive |
boolean |
Must be true for directory paths, false for files. |
external_checksum |
string |
Checksum for validating the integrity of the source file. Assumed to be MD5 unless checksum_algorithm is set. Cannot be used for directories. |
checksum_algorithm |
string |
Algorithm to be used for checksum validation of this transfer_item. Accepts arbitrary strings as algorithms, but a string that does not match an algorithm supported by both endpoints will cause the task to fail at submission or execution. |
2.5. transfer_symlink_item
Field Name | JSON Type | Description |
---|---|---|
DATA_TYPE |
string |
Always the value "transfer_symlink_item" |
source_path |
string |
Source path of symlink to keep |
destination_path |
string |
Destination path of symlink |
2.6. Delete specific fields
Field Name | JSON Type | Description |
---|---|---|
endpoint |
string |
Id of the endpoint to delete data on. |
DATA |
list |
List of delete_item documents containing paths to delete. Note that unlike transfer tasks, directories do not need to end with ("/"). File or directory symlinks are deleted and never followed. |
recursive |
boolean |
Delete directory contents recursively. Required if any delete_item (or any path matched by globbing) is a directory. Symlinks found during recursive expansion are deleted and never followed. Note that unlike transfer submissions, this is a top level field and can’t be specified per item. Default false. |
ignore_missing |
boolean |
Don’t generate errors for non existent files and directories. Default false. |
interpret_globs |
boolean |
If false (the default), every character in the entire path is treated literally. If true, shell glob characters (*, ?, [, and ]) in the last path component are interpreted as a pattern, unless they are escaped by a preceding backslash (\). For consistency, the rest of the path (e.g. everything but the final component) also has any backslash escapes removed, so a literal backslash anywhere in the path must be escaped. Like UNIX, glob characters will not match files or directories that start with a period (.); a literal period character is required in such cases. The special directory entries . and .. are never matched. The second-to-last path component must be a directory (or symlink-to-directory, in which case it is followed). Symlinks found during glob expansion are deleted and never followed. |
3. Operations
3.1. Get a submission id
Get a submission id, required when submitting transfer and delete tasks. Note that this is different than the task id returned by the submit operations.
URL |
/submission_id |
---|---|
Method |
GET |
Response Body |
|
3.2. Submit a transfer task
The transfer submission document contains task-level options and one or more items to transfer in the "DATA" field. This API is relatively low-level and explicit, but allows renaming and is idempotent. There are three types of items:
3.2.1. Recursive Directory
To copy a directory, use a transfer_item with the recursive option set to True. The source_path must be a directory or symlink-to-directory. The source_path and destination_path may or may not end with slash ("/"); both forms are equivalent. A trailing slash will be automatically added if it is not present.
The contents of source_path are placed inside destination_path on the destination endpoint. The destination_path directory and all its parent subdirectories are created automatically. The recursive_symlinks option determines what happens to symlinks found in the contents of source_path. The delete_destination_extra option may be used to provide "mirroring". The filter_rules option can be used to specify which items will and won’t be included in the recursive transfer.
3.2.2. File
To copy a file, use a transfer_item with the recursive option set to False. The source_path and destination_path must not end with slash ("/"). The source_path must be a file or symlink-to-file.
The file at source_path is copied to destination_path. The destination_path is the full path of the destination file, and all parent directories are created automatically.
3.2.3. Symlink
To keep a specific symlink (keep the symlink itself, and not dereference its target), use a transfer_symlink_item. The source_path and destination_path must not end with slash ("/"). The source_path must be a symlink of any type (file, directory, or invalid).
The symlink at source_path is duplicated at destination_path (the target is not dereferenced or copied). The destination_path is the full path of the destination symlink, and all parent directories are created automatically. The symlink target path name is preserved identically; there is no conversion between absolute or relative paths, or chroot paths.
3.2.4. Filter Rules
To control which items are and aren’t transferred when recursively transferring a directory, one or more filter_rule documents can be given. If an item matches every specified field in a filter rule, the rule’s method will be applied on that item.
Filter rules will not be applied to items given as input in the DATA field, and up to 20 filter rules may be given with any transfer.
Field Name | JSON Type | Description |
---|---|---|
DATA_TYPE |
string |
Always the value "filter_rule". |
method |
string |
Currently only "exclude" is supported. Any item discovered during the recursive transfer that matches this filter rule won’t be transferred to the destination endpoint. |
type |
string |
Either "file" or "dir". If left as null or not given, this filter rule will apply to all item types, otherwise it will only apply to the specified type. |
name |
string |
A pattern to match against the item name. This field is currently required for all filter rules. Unix style wildcards are supported. * matches everything. ? matches any single character. [] matches any singe character within the brackets. [!] matches any single character not within the brackets. |
Example Filter Rules
Exclude any files ending with the .git extension
{
"DATA_TYPE": "filter_rule",
"method": "exclude",
"type": "file",
"name": "*.git"
}
Exclude any directory named "Private" or "private"
{
"DATA_TYPE": "filter_rule",
"method": "exclude",
"type": "dir",
"name": "[Pp]rivate"
}
Exclude all hidden items
{
"DATA_TYPE": "filter_rule",
"method": "exclude",
"name": ".*"
}
URL |
/transfer |
---|---|
Method |
POST |
Request Body |
|
Response Body |
|
3.2.5. Result codes
Code | HTTP Status | Description |
---|---|---|
Accepted |
202 |
The transfer or delete submission has been accepted and a task has been created and queued for execution |
Duplicate |
200 |
A transfer or delete submission with the same submission id has already been accepted. In case of a network error, the client may not know whether the submission was successful, and won’t know the task id if it was successful. If the client re-submits and gets this code, it means the initial request was successful, and the task id in the response can be used. If this is received on a request that is not a retry, the client is likely not getting a submission id correctly for each submission. |
3.2.6. Errors
Code | HTTP Status | Description |
---|---|---|
ClientError.BadRequest or BadRequest |
400 |
There is some problem in the request document, see the message for details. |
PermissionDenied |
403 |
The user does not have permission to use one of the endpoints in the request. |
ConsentRequired |
403 |
The source and/or destination requires a data_access scope that the user has not consented to. See the ConsentRequired error for details. |
ServiceUnavailable |
503 |
The service is down for maintenance. |
NoCredException |
409 |
One or more endpoints in the request are not activated or have expired activation. Activate the affected endpoint(s) and retry the operation. |
Conflict |
409 |
A transfer with identical paths has not yet completed, |
3.3. Submit a delete task
Response and error codes are the same as for transfer submission.
URL |
/delete |
---|---|
Method |
POST |
Request Body |
|
Response Body |
|