Transfer API
  • Transfer API Documentation
  • API Overview
  • Task Submission
  • Task Management
  • File Operations
  • Endpoints and Collections
  • Globus Connect Personal Management
  • Endpoint and Collection Search
  • Roles
  • Collection Bookmarks
  • Guest Collection Permission Management
  • Advanced Collection Management
  • Transfer Action Providers
    • Migrating Transfer Action Providers
    • Transfer Action Provider: Transfer
    • Transfer Action Provider: Delete
    • Transfer Action Provider: Manage Permission
    • Transfer Action Provider: List Directory Contents
    • Transfer Action Provider: Stat File or Directory
    • Transfer Action Provider: Make Directory
    • Transfer Action Provider: Collection Info
    • Transfer Action Provider: Create GCP Guest Collection
    • Transfer Action Provider: Create GCSv5 Guest Collection
Skip to main content
Globus Docs
  • APIs
    Auth Flows Groups Search Timers Transfer Globus Connect Server Compute Helper Pages
  • Applications
    Globus Connect Personal Globus Connect Server Premium Storage Connectors Compute Command Line Interface Python SDK JavaScript SDK
  • Guides
  • Support
    FAQs Mailing Lists Contact Us Check Support Tickets
  1. Home
  2. Globus Services
  3. Transfer API Documentation
  4. Task Submission

Task Submission

Table of Contents
  • 1. Overview
  • 2. Document Types
    • 2.1. Transfer and Delete Documents
      • 2.1.1. Path Format
    • 2.2. Common Transfer and Delete fields
    • 2.3. Transfer specific fields
    • 2.4. transfer_item
    • 2.5. transfer_symlink_item
    • 2.6. Delete specific fields
    • 2.7. delete_item
  • 3. Operations
    • 3.1. Get a submission id
    • 3.2. Submit a transfer task
      • 3.2.1. Recursive Directory
      • 3.2.2. File
      • 3.2.3. Symlink
      • 3.2.4. Filter Rules
      • 3.2.5. Result codes
      • 3.2.6. Errors
    • 3.3. Submit a delete task

1. Overview

Transfer and delete are asynchronous operations, and result in a background task being created. Both require a submission_id requested from the get submission id resource and return a task_id on successful submission. The submission_id is used to safely re-submit tasks without risk of duplicate submission in the case of a network failure, and the task_id can be used with the task resources for managing and monitoring the task after submission. The submission id and task id are not the same.

2. Document Types

2.1. Transfer and Delete Documents

The "transfer" and "delete" documents share several common fields and have similar structure. Both contain a submission id, a collection or collections to operate on, a list of options, and a list of paths. The key difference is that a transfer submission contains a pair of collections and a list of pairs of source destination paths, while a delete submission contains a single collection and a list of paths to delete. There are also several options that only make sense for one or the other, like encrypt_data which is only available for transfers.

2.1.1. Path Format

Paths must use the path component separator forward slash (/), commonly used on Unix and Mac OSX, not backslash. This is true even if one of the collections happens to be running on a windows machine, because the GridFTP protocol has standardized on forward slash.

See also Path Encoding.

Note

Paths are always relative to the root of the collection, which may not be the root of the filesystem.

2.2. Common Transfer and Delete fields

Field Name JSON Type Description

DATA_TYPE

string

Either "transfer" or "delete" to indicate the submission type.

submission_id

string

Id acquired from /submission_id. If a submission fails due to network error, resubmit the task with the same submission_id to avoid risk of duplicate submission.

label

string

Optional user specified string to help identify the task. The label is opaque; no specific form is required. The maximum length is 128 characters. Newline characters (\r or \n) and trailing or leading whitespace are not allowed.

notify_on_succeeded

boolean

If true and the user has notification enabled, send a notification email when the task completes with status SUCCEEDED. Default true.

notify_on_failed

boolean

If true and the user has notification enabled, send a notification email when the task completes with status FAILED. Default true.

notify_on_inactive

boolean

[DEPRECATED] Activation is no longer used by any supported entity types.

skip_activation_check

boolean

[DEPRECATED] Activation is no longer used by any supported entity types.

deadline

string

ISO 8601 datetime string. If non-null, cancel the task if it’s not complete at the specified time. The default (null deadline) behavior for transfer tasks is to have a flexible deadline, which allows the task to continue indefinitely, as long as it is making progress. Delete tasks default to a deadline of 24 hours after submission.

store_base_path_info

boolean

If true, stores parent directories alongside a task document so that they can be viewed there by a user. For transfer requests, both the source and destination paths are stored. For delete requests, the supplied path is stored as the source path in the task document. The parent directories are derived from the source and destination paths of the first supplied item in the task. If the supplied path is a root or home directory, the stored path will be null. Default false.

2.3. Transfer specific fields

Field Name JSON Type Description

source_endpoint[1]

string

Id of the collection to transfer data from.

destination_endpoint[1]

string

Id of the collection to transfer data to.

DATA

list

One or more items to transfer. See Transfer Submission for details.

filter_rules

list

One or more filter_rules to apply during recursive expansion. See filter_rules for details.

encrypt_data

boolean

If true, encrypt the data channel. Default false. If either the source or destination collections has force_encryption set or inherited from a parent, the data channel will be encrypted even if this is set to false.

sync_level

integer

If non-null, synchronize files, i.e. copy only files that are new or have been changed. The level must be an integer in the 0-3 range, and it controls what checks are performed to determine if a file should be copied. Higher levels include the checks from lower levels:

0

Copy files that do not exist at the destination.

1

Copy files if the size of the destination does not match the size of the source.

2

Copy files if the timestamp of the destination is older than the timestamp of the source.

3

Copy files if checksums of the source and destination do not match.

verify_checksum

boolean

After transfer, verify that the source and destination file checksums match. If they don’t, re-transfer the entire file and keep trying until it succeeds. Default false.

preserve_timestamp

boolean

Preserve modification time of files and symlinks. Default false. Directory times are not preserved because filesystems typically override them.

delete_destination_extra

boolean

Delete files, directories, and symlinks in the destination directory which don’t exist on the source directory or are a different type. Only applies for recursive directory transfers. Default: false.

recursive_symlinks

string

(beta) This field is only supported by certain collections.

How to process symbolic links (symlinks) found in the contents of recursive directory transfer_items. The default is "ignore". Valid values are:

ignore

Symlinks in the source directory are ignored. If delete_destination_extra is true, all symlinks in the destination directory are removed, because they are presumed not to exist on the source.

keep

Symlinks in the source directory are created as symlinks in the destination directory. The symlink target path name is preserved identically; there is no conversion between absolute or relative paths, or chroot paths.

copy

Symlinks in the source directory are followed (subject to permissions) and created as files or directories in the destination directory. The task will fail if any symlink in the source directory is invalid (no permission to the target, or the target does not exist). If delete_destination_extra is true, all symlinks in the destination directory are removed.

skip_source_errors

boolean

Skip files or directories on the source collection that hit PERMISSION_DENIED, FILE_NOT_FOUND, or AMBIGUOUS_PATH errors.

fail_on_quota_errors

boolean

Cancel the task if QUOTA_EXCEEDED errors are hit.

perf_cc, perf_p, perf_pp

integer

If non-null, a hint for what concurrency, parallelism, and pipelining value to use for the task. (experimental)

perf_udt

boolean

If true, use the UDT transfer protocol. (experimental)

source_local_user

string

Optional value passed to the source’s identity mapping specifying which local user account to map to. Only usable with Globus Connect Server v5 mapped collections.

destination_local_user

string

Optional value passed to the destination’s identity mapping specifying which local user account to map to. Only usable with Globus Connect Server v5 mapped collections.

2.4. transfer_item

Field Name JSON Type Description

DATA_TYPE

string

Always the value "transfer_item"

source_path

string

Source path of file or directory

destination_path

string

Destination path of file or directory

recursive

boolean

Can optionally be set to true for directory paths or false for file paths.

If unset, the item’s type will be automatically discovered at task runtime and directories will be recursively transferred, but options incompatible with the item’s type will not be detected at submission and instead cause task errors.

external_checksum

string

Checksum for validating the integrity of the source file. Assumed to be MD5 unless checksum_algorithm is set. Cannot be used for directories.

checksum_algorithm

string

Algorithm to be used for checksum validation of this transfer_item. Accepts arbitrary strings as algorithms, but a string that does not match an algorithm supported by both collections will cause the task to fail at submission or execution.

2.5. transfer_symlink_item

Field Name JSON Type Description

DATA_TYPE

string

Always the value "transfer_symlink_item"

source_path

string

Source path of symlink to keep

destination_path

string

Destination path of symlink

2.6. Delete specific fields

Field Name JSON Type Description

endpoint[1]

string

Id of the collection to delete data on.

DATA

list

List of delete_item documents containing paths to delete.

File or directory symlinks are deleted and never followed.

recursive

boolean

Delete directory contents recursively. Required if any delete_item (or any path matched by globbing) is a directory. Symlinks found during recursive expansion are deleted and never followed. Note that unlike transfer submissions, this is a top level field and can’t be specified per item. Default false.

ignore_missing

boolean

Don’t generate errors for non existent files and directories. Default false.

interpret_globs

boolean

If false (the default), every character in the entire path is treated literally.

If true, shell glob characters (*, ?, [, and ]) in the last path component are interpreted as a pattern, unless they are escaped by a preceding backslash (\). For consistency, the rest of the path (e.g. everything but the final component) also has any backslash escapes removed, so a literal backslash anywhere in the path must be escaped.

Like UNIX, glob characters will not match files or directories that start with a period (.); a literal period character is required in such cases. The special directory entries . and .. are never matched.

The second-to-last path component must be a directory (or symlink-to-directory, in which case it is followed). Symlinks found during glob expansion are deleted and never followed.

local_user

string

Optional value passed to identity mapping specifying which local user account to map to. Only usable with Globus Connect Server v5 mapped collections.

2.7. delete_item

Field Name JSON Type Description

DATA_TYPE

string

Always has value "delete_item" to indicate this document type.

path

string

Path of file or directory to delete.

3. Operations

3.1. Get a submission id

Get a submission id, required when submitting transfer and delete tasks. Note that this is different than the task id returned by the submit operations.

URL

/submission_id

Method

GET

Response Body

{
  "value": "55379aa2-d9a2-11e5-976c-22000b9da45e",
  "DATA_TYPE": "submission_id"
}

3.2. Submit a transfer task

The transfer submission document contains task-level options and one or more items to transfer in the "DATA" field. This resource is relatively low-level and explicit, but allows renaming and is idempotent. There are three types of items:

3.2.1. Recursive Directory

To copy a directory, use a transfer_item with recursive optionally set to true. The source_path must be a directory or symlink-to-directory. The source_path and destination_path may or may not end with slash ("/"); both forms are equivalent. A trailing slash will be automatically added if it is not present.

The contents of source_path are placed inside destination_path on the destination collection. The destination_path directory and all its parent subdirectories are created automatically. The recursive_symlinks option determines what happens to symlinks found in the contents of source_path. The delete_destination_extra option may be used to provide "mirroring" by deleting any files in the destination directory not in the source. The filter_rules option can be used to specify which items will and won’t be included in the recursive transfer.

3.2.2. File

To copy a file, use a transfer_item with recursive optionally set to false. The source_path must be a file or symlink-to-file.

The file at source_path is copied to destination_path. The destination_path is the full path of the destination file, and all parent directories are created automatically.

3.2.3. Symlink

To keep a specific symlink (keep the symlink itself, and not dereference its target), use a transfer_symlink_item. The source_path and destination_path must not end with slash ("/"). The source_path must be a symlink of any type (file, directory, or invalid).

The symlink at source_path is duplicated at destination_path (the target is not dereferenced or copied). The destination_path is the full path of the destination symlink, and all parent directories are created automatically. The symlink target path name is preserved identically; there is no conversion between absolute or relative paths, or chroot paths.

3.2.4. Filter Rules

To control which items are and aren’t transferred when recursively transferring directories, one or more filter_rule documents can be given in a list. Each item that is found during recursive directory traversal is matched against the filter rules in the order they are listed. When the first filter rule that matches an item is applied (either "include" or "exclude"), filter rule matching stops. If no rules match, the item is included in the transfer. Notably, this makes "include" filter rules only useful when overriding more general "exclude" filter rules later in the list.

Filter rules will not be applied to items given as input in the DATA field, and up to 20 filter rules may be given with any transfer.

Field Name JSON Type Description

DATA_TYPE

string

Always the value "filter_rule".

method

string

Either "include" or "exclude." Items that first match an "include" filter rule will be transferred. Items that first match an "exclude" filter rule will not be transferred.

type

string

Either "file" or "dir". If omitted, this filter rule will apply to all item types. Otherwise it will only apply to the specified type.

If a "dir" is excluded then it, and all items within it, will be excluded, even if items in the directory might have matched an "include" filter rule.

name

string

A pattern to match against the item name. This field is currently required for all filter rules. Unix style wildcards are supported. * matches everything. ? matches any single character. [] matches any single character within the brackets. [!] matches any single character not within the brackets.

Example Filter Rules

Exclude any files ending with the ".git" extension.

[
  {
    "DATA_TYPE": "filter_rule",
    "method": "exclude",
    "type": "file",
    "name": "*.git"
  }
]

Only transfer files with the ".txt" extension.

Note that the second filter rule has a type of "file" to allow directories to be recursively traversed.

[
  {
    "DATA_TYPE": "filter_rule",
    "method": "include",
    "type": "file",
    "name": "*.txt"
  },
  {
    "DATA_TYPE": "filter_rule",
    "method": "exclude",
    "type": "file",
    "name": "*"
  }
]

Exclude any directories named "Private" or "private".

[
  {
    "DATA_TYPE": "filter_rule",
    "method": "exclude",
    "type": "dir",
    "name": "[Pp]rivate"
  }
]

Exclude all hidden items other than ".specific_hidden_file".

Note that if ".specific_hidden_file" is within a hidden directory found during recursive directory traversal, it will not be transferred with these filter rules — as the directory and all its contents would be excluded before the file was found.

[
  {
    "DATA_TYPE": "filter_rule",
    "method": "include",
    "name": ".specific_hidden_file"
  },
  {
    "DATA_TYPE": "filter_rule",
    "method": "exclude",
    "name": ".*"
  }
]

Only transfer files with "data" in their names, with the exception of "unwanted_data.csv".

[
  {
    "DATA_TYPE": "filter_rule",
    "method": "exclude",
    "type": "file",
    "name": "unwanted_data.csv"
  },
  {
    "DATA_TYPE": "filter_rule",
    "method": "include",
    "type": "file",
    "name": "*data*"
  },
  {
    "DATA_TYPE": "filter_rule",
    "method": "exclude",
    "type": "file",
    "name": "*"
  }
]

URL

/transfer

Method

POST

Request Body

Transfer document

Response Body

{
  "DATA_TYPE": "transfer_result",
  "task_id": "994f289b-d9a2-11e5-976c-22000b9da45e",
  "submission_id": "f5cc79fd-dfc8-475e-b726-b96c734a484d",
  "code": "Accepted",
  "message": "The task was submitted successfully",
  "resource": "/transfer",
  "request_id": "ABCdef789"
}

3.2.5. Result codes

Code HTTP Status Description

Accepted

202

The transfer or delete submission has been accepted and a task has been created and queued for execution

Duplicate

200

A transfer or delete submission with the same submission id has already been accepted. In case of a network error, the client may not know whether the submission was successful, and won’t know the task id if it was successful. If the client re-submits and gets this code, it means the initial request was successful, and the task id in the response can be used. If this is received on a request that is not a retry, the client is likely not getting a submission id correctly for each submission.

3.2.6. Errors

Code HTTP Status Description

ClientError.BadRequest or BadRequest

400

There is some problem in the request document, see the message for details.

PermissionDenied

403

The user does not have permission to use one of the collections in the request.

ConsentRequired

403

The source and/or destination requires a data_access scope that the user has not consented to. See Data Access Consent for details.

ServiceUnavailable

503

The service is down for maintenance.

Conflict

409

An existing transfer with the same user, collections, and paths has not yet completed.

3.3. Submit a delete task

Response and error codes are the same as for transfer submission.

URL

/delete

Method

POST

Request Body

Delete document

Response Body

{
  "DATA_TYPE": "delete_result",
  "task_id": "cfec1219-3f78-4c02-8fe6-83d5ba01f3a2",
  "submission_id": "499f18bf-9ea6-48e6-a030-98e0f29512b5",
  "code": "Accepted",
  "message": "The task was submitted successfully",
  "resource": "/delete",
  "request_id": "ABCdef789"
}

1. This use of the term "endpoint" is a case of legacy endpoint terminology and can also/exclusively refer to collections
  • Transfer API Documentation
  • API Overview
  • Task Submission
  • Task Management
  • File Operations
  • Endpoints and Collections
  • Globus Connect Personal Management
  • Endpoint and Collection Search
  • Roles
  • Collection Bookmarks
  • Guest Collection Permission Management
  • Advanced Collection Management
  • Transfer Action Providers
    • Migrating Transfer Action Providers
    • Transfer Action Provider: Transfer
    • Transfer Action Provider: Delete
    • Transfer Action Provider: Manage Permission
    • Transfer Action Provider: List Directory Contents
    • Transfer Action Provider: Stat File or Directory
    • Transfer Action Provider: Make Directory
    • Transfer Action Provider: Collection Info
    • Transfer Action Provider: Create GCP Guest Collection
    • Transfer Action Provider: Create GCSv5 Guest Collection
© 2010- The University of Chicago Legal Privacy Accessibility