1. Overview

Transfer and delete are asynchronous operations, and result in a background task being created. The task id is returned from successful submission, and can be used to monitor the progress of the task.

Note:Detailed per-file status, errors, and information events are retained for 30 days and then deleted. When this happens, the history_deleted flag will be set on the task document.

1.1. Task Visibility

A task will be visible if the current primary identity is the same as the primary identity when the task was submitted. The only time this will not be the case is when two separate identities with existing task history are linked. When that happens, the task history for the linked identity that is no longer primary will not be visible. It can still be recovered and viewed by unlinking the identity and signing in to that identity separately.

2. Document Types

2.1. Task Document

The "task" document type represents a single transfer or delete submission. Other task types may be added in the future.

2.1.1. Task Fields

Field Name JSON Type Description

DATA_TYPE

string

Always has value "task" to indicate this document type.

task_id

string

Globally unique uuid for this task.

type

string

The type of task - either "TRANSFER" or "DELETE".

status

string

One word indicating the progress of the task:

"ACTIVE"

The task is in progress.

"INACTIVE"

The task has been suspended and will not continue without intervention. Currently, only credential expiration will cause this state.

"SUCCEEDED"

The task completed successfully.

"FAILED"

The task or one of its subtasks failed, expired, or was canceled.

label

string

User defined label to make finding tasks simpler.

username

string

[DEPRECATED] Use owner_id instead. The globus username of the user who created the task. For non-globusid.org identities, this will be a meaningless string starting with u_, and for globusid.org identities it will be the globusid.org username.

owner_id

string

Identity id of the task owner. If the task was submitted with multiple linked identities, the owner will be the primary identity.

request_time

string

The date and time the task was created, in ISO 8601 format.

completion_time

string

The date and time the task was completed or failed, in ISO 8601 format. If the task is still in progress (status "ACTIVE" or "INACTIVE"), this will be null.

deadline

string

If set when the task is created, cancel the task if it’s not finished when this date and time is reached. By default tasks will not be canceled as long as they are making progress.

source_endpoint

string

[DEPRECATED] Use source_endpoint_id instead. Canonical name of the source endpoint for type "TRANSFER" and the name of the only endpoint for type "DELETE".

source_endpoint_id

string

id of the source endpoint.

source_endpoint_display_name

string

display_name of the source endpoint at time of submit, or the canonical_name if display_name is not set. For tasks submitted prior to August 2015 (version 4.4), this will be the same as source_endpoint.

destination_endpoint

string

[DEPRECATED] Use destination_endpoint_id instead. Canonical name of the destination endpoint for type "TRANSFER", null for type "DELETE".

destination_endpoint_id

string

id of the destination endpoint if any (null for "DELETE" tasks).

destination_endpoint_display_name

string

Like source_endpoint_display_name but for the destination, or null for "DELETE" tasks.

sync_level

integer

The sync level used for the transfer, or null if not used. Always null for delete tasks.

encrypt_data

boolean

For transfer tasks, this will be true if the data channel is encrypted. This can happen in two ways: if one of the endpoints or host endpoints involved in the transfer has force_encryption set to true, or if the transfer request has the encrypt_data option set to true. Always false for delete tasks.

verify_checksum

boolean

The verify_checksum option used for the transfer. Always false for delete tasks.

delete_destination_extra

boolean

The delete_destination_extra option used for the transfer. Always false for delete tasks.

recursive_symlinks

string

The recursive_symlinks option used for transfer tasks. Always null for delete tasks.

preserve_timestamp

boolean

The preserve_timestamp option used for the transfer. Always false for delete tasks.

command

string

If the task was submitted via the CLI, this field will contain the original command line, including options, that created the task. If submitted via the Globus website or directly via the Transfer API, this will contain the string "API", followed by the API version, and optionally followed by a short string representing the client that submitted the request. The format of this field is subject to change and should not be relied upon.

history_deleted

boolean

This flag will be set for tasks older than one month, and indicates that details of the task are no longer available. In particular, subtasks and successful transfers will not be available for the task if this is true, so clients should always check this before querying for subtasks and successful transfers.

faults

int

The number of errors this task encountered. Note that certain types of faults are not fatal (for example, network communication errors) and can be successfully retried. A CANCELED or EXPIRED event is not included in this fault count.

files

int

The total number of files in a transfer task. This can grow as the task recursively scans directories.

directories

int

The total number of directories in a transfer task. This can grow as the task recursively scans directories.

symlinks

int

The total number of kept symlinks (not copied or ignored) in a transfer task. This can grow as the task recursively scans directories.

files_skipped

int

The number of files skipped because no changes were detected. This will always be zero for non-sync transfer tasks (with null sync_level) and for all delete tasks.

files_transferred

int

The number of files actually transferred over the network. For a successful transfer, files = files_skipped + files_transferred. Not used for delete tasks.

subtasks_total

int

Total number of subtasks. This includes file transfer or delete subtasks and helper subtasks such as directory expansion, so is not a reliable measure of the number of files being transferred. It can also grow over time as new files and directories are discovered in directory expansions.

subtasks_pending

int

Number of subtasks which are still in progress.

subtasks_retrying

int

The number of pending subtasks that have had one or more faults while processing. This is a subset of subtasks_pending.

subtasks_succeeded

int

Number of subtasks which have completed successfully.

subtasks_expired

int

Number of subtasks which expired and were not completed.

subtasks_canceled

int

Number of subtasks which were canceled.

subtasks_failed

int

Number of subtasks which failed for reasons other than expiring or being cancelled.

bytes_transferred

int

The total number of bytes transferred summed across all subtasks.

bytes_checksummed

int

If sync level 3 is used, the number of bytes checksummed while determining which files need to be transferred.

effective_bytes_per_second

int

A simplistic calculation of bytes/second based on the start time of the task and its completion time, if applicable, or the current time. Valid for transfer tasks. Always 0 for other task types.

nice_status

string

[ALPHA] For tasks with status ACTIVE or INACTIVE, a string indicating a more detailed status of the task. For completed tasks, this will always be empty or null. If it has value OK or Queued, the task is waiting for other tasks to complete or proceeding normally, and no intervention should be required. All other values indicate that some error is being encountered, which may or may not resolve on it’s own. Examples: Creds Expired, PERMISSION_DENIED, ENDPOINT_ERROR, CONNECT_FAILED, PAUSED_BY_ADMIN.

nice_status_details

string

[ALPHA] Raw error output

nice_status_short_description

string

[ALPHA] 3-4 word description of nice_status code

nice_status_expires_in

string

[ALPHA] Seconds until any credential required for the task expires (-1=never, 0=expired)

canceled_by_admin

string

[ALPHA] If the task completes successfully, is canceled by the task owner using the standard cancel resource, or hits the deadline before completing, this will be null. It is set only for tasks canceled by an endpoint administrator using the Advanced Endpoint Management API. For such tasks, it will have one of the following values:

"SOURCE"

An administrator of the source endpoint of the task canceled the task. Also set for "DELETE" type tasks which involve only one endpoint

"DESTINATION"

An administrator of the destination endpoint of the task canceled the task

"BOTH"

An administrator of both the source and destination endpoint canceled the task.

canceled_by_admin_message

string

[ALPHA] For tasks with canceled_by_admin set to a non-null value, this will contain a message from the administrator who canceled the task.

is_paused

boolean

"true" if the task is in progress (status "ACTIVE" or "INACTIVE") and has been paused by the administrator of the source or destination endpoint, "false" if the task has not been paused or is complete (status "SUCCEEDED" or "FAILED"). Use Get task pause info to get information about why the task is paused.

2.2. Event Document

Events are logged as a task makes progress or runs into errors.

Event Document Example
{
    "DATA_TYPE": "event",
    "code": "PERMISSION_DENIED",
    "description": "Permission denied",
    "details": "Error (transfer)\nServer: ballen#uc-laptop (Globus Connect)\nFile: /~/Downloads/plus-plan-exposure.png\nCommand: STOR ~/Downloads/plus-plan-exposure.png\nMessage: Fatal FTP response\n---\n500 Command failed. : Path not allowed.\n",
    "is_error": True,
    "time": "2014-07-08 18:50:18+00:00"
}

2.2.1. Event Fields

Field Name JSON Type Description

DATA_TYPE

string

Always has value "event" to indicate this document type.

code

string

A code indicating the type of the event.

is_error

boolean

true if event is an error event

description

string

A description of the event.

details

string

Type specific details about the event.

time

string

The date and time the event occurred, in ISO 8601 format (YYYY-MM-DD HH:MM:SS) and UTC.

2.3. Limited pause rule document

The limited pause rule document is a subset of the full pause rule document. It does not contain sensitive fields, in particular modified_by, which is only viewable by endpoint administrators using the Advanced Endpoint Management API. The DATA_TYPE is "pause_rule_limited" instead of "pause_rule", to indicate the exclusions. See pause_rule document for details.

3. Path Arguments

Name Type Description

task_id

string

Unique id string of a task.

4. Common Query Parameters

Name Type Description

fields

string

Comma separated list of fields to include in the response. This can be used to save bandwidth on large list responses when not all fields are needed.

limit

int

For paged resources, change the page size. For event_list and task_list, the default is 10 and the max is 1000.

offset

int

For paged resources, specify an offset within the full result set. Typically a fixed page size is specified with limit, and offset is incremented by the page size to fetch each page.

orderby

string

For paged resources, a comma separated list of order by options. Each order by option is either a field name, or a field name followed by space and ASC or DESC for ascending and descending; ascending is the default. Note that only certain fields are supported for ordering; see the specific operation documentation for details.

filter

string

For paged resources, return only resources that match all of the specified filter criteria. The value must be a "/" separated list of of "FIELD_NAME:FIELD_FILTER" strings. See the event_list and task_list documentation for details on what fields are supported for each.

5. Common Errors

Code HTTP Status Description

TaskNotFound

404

If task specified by <task_id> is not found

Conflict

409

If task is complete and can’t be updated.

ServiceUnavailable

503

If the service is down for maintenance.

6. Operations

6.1. Get task list

Get a paged list of tasks submitted by the current user. Returns only returns TRANSFER tasks by default; clients should pass query parameter filter=type:TRANSFER,DELETE to get all tasks.

Note:In the next release, this will be changed so all tasks of any type will be returned by default. The explicit filter including both types will continue to work.

URL

/task_list?filter=type:TRANSFER,DELETE

Method

GET

Response Body

{
    "DATA_TYPE": "task_list",
    "length": 2,
    "limit": 10,
    "offset": 20,
    "total": 125,
    "DATA": [
        {
            "username": "jsmith",
            "bytes_transferred": 10240,
            "faults": 0,
            "DATA_TYPE": "task",
            "sync_level": null,
            "completion_time": "2000-01-02 03:45:06+00:00",
            "deadline": "2000-01-02 03:45:06+00:00",
            "type": "TRANSFER",
            "destination_endpoint": "go#ep1",
            "files": 10,
            "delete_destination_extra": false,
            "nice_status_details": "Error from gridftp server foo.com: 500 I am too busy",
            "request_time": "2000-01-02 03:45:06+00:00",
            "nice_status": "OK",
            "subtasks_expired": 10,
            "subtasks_canceled": 10,
            "label": null,
            "subtasks_total": 10,
            "nice_status_expires_in": 123,
            "status": "ACTIVE",
            "bytes_checksummed": 10,
            "subtasks_failed": 10,
            "history_deleted": false,
            "files_skipped": 3,
            "subtasks_retrying": 10,
            "nice_status_short_description": null,
            "preserve_timestamp": false,
            "task_id": "12345678-9abc-def0-1234-56789abcde03",
            "encrypt_data": false,
            "source_endpoint": "bob#laptop",
            "subtasks_succeeded": 10,
            "command": "transfer",
            "subtasks_pending": 10,
            "verify_checksum": false,
            "directories": 10
        }
    ]

}

6.1.1. Filter and Order By Options

Fields allowed in the filter and orderby query parameters.

Task List Filter Example
GET /task_list?filter=status:ACTIVE,INACTIVE/label:~experiment1*

Get tasks that are still running (status ACTIVE or INACTIVE), and have a label that begins with the string "experiment1".

Name Type Description

task_id

string list

Comma separated list of UUID strings. Return only tasks with the specified task ids. Only returns tasks owned by the current user.

type

string list

Comma separated list of type names (TRANSFER, DELETE). Return only tasks of the specified type(s). The default is currently TRANSFER, but this will be changing in the next release so both TRANSFER and DELETE tasks are included by default.

status

string list

Comma separated list of status codes (ACTIVE, INACTIVE, FAILED, SUCCEEDED). Return only tasks with one of the specified statuses.

label

pattern list

Comma separated list of patterns to match against the label field. Returns tasks that match any of the patterns. Each pattern is an operator, followed by a string. The operator is one of !, =, , or !. The = operator requires an exact match, and ! matches anything other than the specified string. The operator is a case-insentive match and allows the use of as a wildcard. The ! operator matches anything that does not match the pattern. For both ~ operators, a literal can be escaped as \*, and a literal backslash can be escaped as \\. As a shortcut, if no operator is specified, = (exact match) is assumed.

request_time

datetime range

Accepts a time range, specified by a comma separated list of two ISO 8601 date/time strings. If one of the dates is ommited, it forms an open range, so "dt," returns all records with date greator or equal to dt, and ",dt" returns all records with dates less than or equal to dt. If there is no comma, it is treated in the same way as "dt,". If the time is omitted from a date/time, it’s assumed to be 00:00.

completion_time

datetime range

Like the request time filter, but for the completion time.

6.2. Get task by id

Get a single task by task id. All fields are included by default, but the fields query parameter can be used to fetch only specific fields.

URL

/task/<task_id>

Method

GET

Response Body

Task document

6.3. Update task by id

Update a single task by task id. Only the label and deadline fields can be updated, and only on tasks that are still running. If the task is complete a "Conflict" error will be returned. A copy of the task body with one of those fields modified can be used, or a partial document containing only DATA_TYPE and the fields to be modified.

URL

/task/<task_id>

Method

PUT

Response Body

Result resource

6.4. Cancel task by id

Submit a cancel request for an active task, by id. Cancel requests are processed asynchronously, but this API call will wait up to 10 seconds for the cancel request to be completed before returning a response. If the task was already complete, result code "TaskComplete" is returned. If the cancel request is processed within 10 seconds, result code "Canceled" is returned. Note that when "Canceled" is returned, it’s still possible that the task completed successfully just as the request was processed but after the check was made to see if the task was already complete. Clients should always check the status field of the task to verify what happened if they care about whether the task succeeded or failed. What will always be true when "Canceled" or "TaskComplete" is returned is that the task is no longer active. If the cancel request can’t be processed in 10 seconds, code "CancelAccepted" is returned, and the client can use "Get task by id" to fetch the task and see when it’s status changes from "ACTIVE" to "FAILED" or "SUCCEEDED".

Only the owner of a task can cancel it via this API resource. If the owner is an administrator on one of the endpoints involved in the task, tasks canceled with this resource will still NOT be marked as as canceled_by_admin. This resource is designed for when the user is acting as a normal user, regardless of any higher level authority they have been granted.

URL

/task/<task_id>/cancel

Method

POST

Response Body

{
    "DATA_TYPE": "result",
    "code": "Canceled",
    "message": "The task has been cancelled successfully.",
    "resource": "/endpoint/user#ep1/cancel",
    "request_id": "ABCdef789"
}

6.5. Get event list

Get a paged list of all events, including error and info events. The results are ordered by time descending (newest first).

URL

/task/<task_id>/event_list

Method

GET

Response Body

{

    "DATA_TYPE": "event_list",
    "length": 2,
    "limit": 2,
    "offset": 0,
    "total": 125,
    "DATA": [
        {
            "code": "PROGRESS",
            "description": null,
            "DATA_TYPE": "event",
            "is_error": false,
            "details": null,
            "time": "2000-01-02 03:45:06+00:00"
        },
        {
            "code": "FILE_NOT_FOUND",
            "description": "No such file or directory",
            "DATA_TYPE": "event",
            "is_error": true,
            "details": "Error (transfer)\nServer: go#ep1 (endpoint1.tutorial.globusonline.org:2811)\nFile: /~/doesnotexist\nCommand: MLST ~/doesnotexist\nMessage: Fatal FTP response\n---\n500-Command failed : System error in stat: No such file or directory\r\n500-A system call failed: No such file or directory\r\n500 End.\n",
            "time": "2014-07-28 02:35:10+00:00"
        }
    ]
}

6.5.1. Filter Options

Fields allowed in the filter query parameter.

Event List Filter Example
GET /event_list?filter=is_error:1

Get only error events.

Name Type Description

is_error

boolean

"1" for true, "0" for false. If true, return only events that are classified as errors. If false, return only events that are classified as non-error (informational). By default, returns all events.

6.5.2. Errors

Code HTTP Status Description

HistoryDeleted

409

After 30 days, the event list for a task is no longer available and this error will be returned.

6.6. Get task successful transfers

For "TRANSFER" tasks that are completed (have status "SUCCEEDED" or "FAILED"), get a list of files that were successfully transferred. The list includes the source and destination paths of each file. Note that this does not include files that were checked but skipped as part of a sync transfer, only files that were actually transferred, and does not include any directories.

A result set may be too large to fit into a single response. In this case, the response will have the next_marker field set to a non-empty, opaque integer token. Pass this token as the marker query parameter in another request to get the next chunk of data. A client must not attempt to generate or assume knowledge of a marker’s format. Note that this differs from the offset based paging supported by some other resources.

Returns 404 "ClientError.NotFound" if history has been deleted (history is kept for only one month ). Returns 400 "ClientError.BadRequest" if the task is not yet complete or is not of type "TRANSFER".

Note:The paths are relative to the shared endpoint root, as submitted in the transfer request, not relative to the host endpoint root.

URL

/task/<task_id>/successful_transfers [?marker=MARKER]

Method

GET

Response Body

{
  "DATA_TYPE": "successful_transfers",
  "marker": 0
  "next_marker": null,
  "DATA": [
    {
      "DATA_TYPE": "successful_transfer",
      "source_path": "/share/godata/file1.txt"
      "destination_path": "/~/test_godata_copy/file1.txt",
    },
    {
      "DATA_TYPE": "successful_transfer",
      "source_path": "/share/godata/file2.txt"
      "destination_path": "/~/test_godata_copy/file2.txt",
    },
    {
      "DATA_TYPE": "successful_transfer",
      "source_path": "/share/godata/file3.txt"
      "destination_path": "/~/test_godata_copy/file3.txt",
    }
  ],
}

6.7. Get task pause info

Get details about why a task is paused (or possibly about to be paused). This incudes pause rules on source and destination endpoints (both host and shared endpoint) that affect the owner of the task, and per-task pause flags set by source endpoint and destination endpoint or source and destination shared endpoint administrators. Any pause rules that have been overridden by an administrator are not listed.

If the task is not paused, this may still return pause rules that have been created but not yet applied to the task. This is because pause rules are processed asynchronously.

If the task is complete, this will return an empty result set, meaning that pause_rules list will be empty and both pause messages will be null.

Authorization

Requires the user to be the owner of the task. To access pause info as an administrator with endpoint manager privileges, use the endpoint manager pause info operation.

Pause Rule

A pause rule is set by the administrator of an endpoint and causes all matching tasks to or from that endpoint to be paused. The rules returned by this operation have some sensitive fields removed, see the pause_rule_limited document.

URL

/task/<task_id>/pause_info

Method

GET

Response Body

{
    "DATA_TYPE": "pause_info_limited",
    "pause_rules": [... list of pause_rule_limited documents...],
    "source_pause_message": null,
    "destination_pause_message": "Disk problems, pausing all tasks until we resolve",
    "source_pause_message_share": null,
    "destination_pause_message_share": null
}

© 2010- The University of Chicago Legal