Transfer API Menu
  • Transfer API Documentation
  • API Overview
  • Endpoint Activation
  • Task Submission
  • Task Monitoring
  • File Operations
  • Endpoint Management
  • Endpoint Search
  • Endpoint Roles
  • Endpoint Bookmarks
  • Endpoint ACL Management
  • Advanced Endpoint Management
  • Change History
Skip to main content
Globus Docs
  • APIs
    Auth Compute Flows Globus Connect Server Groups Search Transfer Python SDK Helper Pages
  • How To
  • Guides
    Globus Connect Server High Assurance Collections for Protected Data Command Line Interface Premium Storage Connectors Security Modern Research Data Portal
  • Support
    FAQs Mailing Lists Contact Us Check Support Tickets
  1. Home
  2. Globus APIs
  3. Transfer API Documentation

API Overview

Table of Contents
  • 1. Overview
  • 2. Terminology
    • 2.1. Transfer Terminology
    • 2.2. API Terminology
  • 3. Making Requests
    • 3.1. Base URL
    • 3.2. Document Formats
    • 3.3. Authentication
    • 3.4. Errors
  • 4. Activation and Consent
    • 4.1. Activation
    • 4.2. Data Access Consent
  • 5. Examples
    • 5.1. Conventions
    • 5.2. Monitoring
    • 5.3. Endpoint Management
      • 5.3.1. Public Endpoints
    • 5.4. Endpoint Directory Listing
      • 5.4.1. Directory Listing
    • 5.5. Creating Directories
    • 5.6. Transfer Submission
    • 5.7. Task Monitoring
    • 5.8. Delete Submission
  • 6. Common Query Parameters
    • 6.1. Paging
      • 6.1.1. Offset Paging
      • 6.1.2. Marker Paging
      • 6.1.3. Last Key Paging
      • 6.1.4. Next Token Paging
    • 6.2. Filtering
    • 6.3. Sorting
    • 6.4. Limiting Result Fields
  • 7. Request Rate Limiting

1. Overview

The Transfer API provides remote data access via GridFtp to file-systems that have been configured with collections within the Transfer service. Collections are generally created using the Globus Connect Server or Globus Connect Personal software packages, but the Transfer service also currently maintains legacy support for other GridFtp collections.

Most requests to the Transfer API fall under three categories: configuring and managing endpoints and collections, synchronously accessing data on a collection such as directory listings, and submitting and managing asynchronous tasks between collections such as file transfers.

This documentation assumes a basic familiarity with HTTP, including the GET, POST, PUT, and DELETE request methods, Content-Type and Accepts headers, and standard status codes.

Python users may wish to use the Globus Python SDK when making requests to the Transfer API as it handles authentication and exposes methods for most Transfer API functionally.

2. Terminology

2.1. Transfer Terminology

  • task - a batch of asynchronous file transfer or delete operations that were submitted together, identified by a unique ID.

  • collection - a definition for a source / destination for data access with a convenient name and unique ID.

  • endpoint - a definition for an installation of Globus Connect Server used as an interface for management and creation of collections. Note that before Globus Connect Server v5, there was no split between the endpoint and its collections, so much of this documentation and the Transfer API itself still refers to collections as endpoints.

  • activation - delegating a temporary x509 credential to the Globus transfer service to access data on behalf of the user.

  • consent - a string associated with an OAuth2 token that tells services what access the user has agreed to. In addition to the transfer scope that users consent to in order make requests to the Transfer API, Globus Connect Server v5 data access requires dependent consents to access data on some endpoints.

2.2. API Terminology

  • resource - a URL addressable part of the API, which can be interacted with using a subset of the GET, POST, PUT, and DELETE HTTP methods.

  • document - a representation of data, returned by resources as output and accepted by resources as input. There are several standard document types, and some types include sub-documents (for example, the task_list type is a container for many documents of type task).

3. Making Requests

3.1. Base URL

All the URLs in the Transfer documentation are relative to a base url for the transfer service.

https://transfer.api.globusonline.org/v0.10

so the full URL to /task_list will be:

https://transfer.api.globusonline.org/v0.10/task_list

Clients should store the base URL in one place and use it when constructing resource URLs, to simplify changing versions.

3.2. Document Formats

The API uses json for all input and output, including error documents.

Note that application/x-www-form-urlencoded is not supported. The body should contain the actual JSON data, not a form encoded version of that data.

The json representation uses a "DATA_TYPE" key to specify the type of resource and a "DATA" key containing a list of sub-documents, if any. For example, the endpoint document type is described in detail here:

  • Endpoint Document

3.3. Authentication

Authentication to the Transfer API requires using Globus Auth to obtain an access token. The Globus Auth developer guide describes the steps to register a client and request tokens for that client to make requests to Globus services.

When requesting tokens, request Transfer all scope in order to get an access token scoped to make requests to the Transfer API:

urn:globus:auth:scope:transfer.api.globus.org:all

Once obtained, the access token needs to be passed to the Transfer API in the Authorization header with the method Bearer:

Authorization: Bearer TOKEN

You can test this by saving the token to a private temporary file, e.g. $HOME/tmp-globus-auth-token.txt, and then passing it to curl with the -H option:

TOKEN=$(cat $HOME/tmp-globus-auth-token.txt)
curl -H "Authorization: Bearer $TOKEN" \
    'https://transfer.api.globusonline.org/v0.10/task_list'

You could also paste the token directly but then it will be stored in your shell history, which is often not desirable.

3.4. Errors

When an error occurs an HTTP status code >=400 will be returned. The body of the response will be a JSON document with details about the error, including code and message fields. The error code will also be provided in the "X-Transfer-API-Error" header. Note that requests outside the API path version prefix may return an HTML or plaintext error body instead. Here is an example error returned when an endpoint is not found:

{
  "code": "EndpointNotFound",
  "message": "No such endpoint '23c1a962-7e68-11e5-ac37-f0def10a689e'",
  "request_id": "HrbjJy3QJ",
  "resource": "/endpoint/23c1a962-7e68-11e5-ac37-f0def10a689e"
}

A 404 status code is used for this response. The code field has the same value as the X-Transfer-API-Error header, for convenient access.

4. Activation and Consent

Endpoints and collections may require activation or additional consents before they can be used for data access.

4.1. Activation

Activation provides the Transfer service with an x509 credential required by some endpoints so the service can perform operations on behalf of the user on the file system. Generally, only Globus Connect Server v4 and legacy GridFTP endpoints will require activation.

If you are unsure if you need to activate an endpoint, the first step is auto-activation. Auto-activation will succeed if the endpoint already has a cached credential or if it never needed one in the first place. If it fails, it will return an activation requirements document that can be used for an activation request, however it is generally simpler to send users to the Globus Webapp for Web Activation.

4.2. Data Access Consent

Standard Globus Connect Server v5 mapped collections require users to consent to a collection specific dependent data_access scope in order to access data on that collection. Every such scope will be in the format:

urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/COLLECTION_UUID/data_access]

Where COLLECTION_UUID is the uuid of a collection that requires the data_access scope. You may request multiple dependent scopes by separating them with spaces:

urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/COLLECTION_UUID_1/data_access *https://auth.globus.org/scopes/COLLECTION_UUID_2/data_access]

If you know ahead of time that you will be accessing a mapped collection that requires a data_access dependent scope, you should request those consents when you first request tokens for authenticating with Transfer.

However, if you don’t know ahead of time which endpoints or collections you will be accessing, you can add error handling logic to handle ConsentRequired errors raised when Transfer is unable to get the dependent tokens needed to access data. The body of these error responses will contain a required_scopes field with a list of scopes to which the user must consent in order to complete the failing request.

{
  "code": "ConsentRequired",
  "message": "Missing required data_access consent",
  "request_id": "WmMV97A1w",
  "required_scopes": ["urn:globus:auth:scope:transfer.api.globus.org:all[*https://auth.globus.org/scopes/ea4c8cf2-d3a2-4c1a-92c2-59925fe7ac5b/data_access *https://auth.globus.org/scopes/0357da44-c908-47e5-8351-ba5c695d11a7/data_access]"],
  "resource": "/transfer"
}

5. Examples

5.1. Conventions

The convention used for examples in this document is similar to raw HTTP requests and responses, with the URL shortened and most headers omitted. As an example, to get a task_list for the logged in user, the request is described as:

GET /task_list

This means that a GET request must be made to the task_list resource, which actually has the URL https://transfer.api.globusonline.org/v0.10/task_list for version v0.10. This is BASE_URL + /task_list. As discussed above, the BASE_URL should be set in one place and re-used, not hard coded into each request. The actual raw HTTP request will typically include many headers:

GET /v0.10/task_list HTTP/1.1
Host: transfer.api.globusonline.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 Iceweasel/4.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: UTF-8,*
Keep-Alive: 115
Connection: keep-alive

Most of these headers were added by the browser (Firefox); the developer will not normally need to deal with them.

For examples that involve sending data, the body is included inline, just like it would be in an HTTP request. For example endpoint creation is described like this:

POST /endpoint
Content-Type: application/json

{
  "display_name": "ACME University shared storage",
  "DATA_TYPE": "endpoint",
  "description": "Example gridftp endpoint."
  "DATA": [
    {
      "DATA_TYPE": "server",
      "hostname": "gridftp.example.org",
      "scheme": "gsiftp",
      "port": 2811,
    }
  ],
}

This means that to create an endpoint, a request using method POST can be made to BASE_URL + /endpoint, with header content-type set to "application/json", and having as the request body the JSON data describing the endpoint. Other headers are required for authentication, but they are not specific to this request.

This format is used to provide a quick description of how to make a request, independent of the client used. The Python and Java examples hide many of the details involved in accessing the API; this document is focused on describing the API itself including those details.

5.2. Monitoring

  • Paged task list with sorting and field selection. (Reference)

    GET /task_list?offset=0&limit=10&fields=task_id,request_time&orderby=request_time

    Lists the first 10 tasks belonging to the currently logged in user, showing only the task_id and request_time fields, ordered by request_time (ascending/oldest first).

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "DATA_TYPE": "task_list",
      "length": 3,
      "limit": "10",
      "offset": "0",
      "total": "3",
      "DATA": [
        {
          "task_id": "3949cec8-7cc8-11e0-82be-12313932c1e0",
          "DATA_TYPE": "task",
          "request_time": "2011-05-12 18:49:22"
        },
        {
          "task_id": "edebec3a-7cc8-11e0-82be-12313932c1e0",
          "DATA_TYPE": "task",
          "request_time": "2011-05-12 18:52:11"
        },
        {
          "task_id": "35115208-7cc9-11e0-82be-12313932c1e0",
          "DATA_TYPE": "task",
          "request_time": "2011-05-12 18:54:34"
        },
      ]
    }
  • Event list. (Reference)

    GET /task/3949cec8-7cc8-11e0-82be-12313932c1e0/event_list

    List all events associated with a task. Events include starting and finishing the transfer, cancellation, progress reports of bytes transferred so far, and any errors encountered.

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "DATA_TYPE": "event_list",
      "length": 2,
      "limit": "10",
      "offset": "0",
      "total": "2",
      "DATA": [
        {
          "code": "SUCCEEDED",
          "description": "The operation succeeded",
          "DATA_TYPE": "event",
          "parent_task_id": "8cb34a9e-7cc8-11e0-82be-12313932c1e0",
          "details": "bytes=3103 mbps=0.000",
          "time": "2011-05-12 18:49:25"
        },
        {
          "code": "STARTED",
          "description": "The operation was started or restarted",
          "DATA_TYPE": "event",
          "parent_task_id": "8cb34a9e-7cc8-11e0-82be-12313932c1e0",
          "details": "Starting at offset 0",
          "time": "2011-05-12 18:49:25"
        }
      ]
    }

5.3. Endpoint Management

  • Endpoint search (Reference)

    GET /endpoint_search?filter_scope=my-endpoints
    GET /endpoint_search?filter_scope=recently-used
    GET /endpoint_search?filter_scope=all&filter_fulltext=xsede+gordon

    List all endpoints owned by the current user, used recently by the user in transfer or delete tasks, or containing the specified search terms. The results for the "XSEDE gordon" search are shown below:

200 OK
X-Transfer-API-KOA-Version: 4.5
Content-Type: application/json

{
  "DATA_TYPE": "endpoint_list",
  "has_next_page": false,
  "limit": 3,
  "offset": 0,
  "DATA": [
    {
      "_rank": 0.421588,
      "acl_available": false,
      "acl_editable": false,
      "activated": false,
      "canonical_name": "arnoldg#gordon",
      "contact_email": null,
      "contact_info": null,
      "default_directory": "/~/",
      "department": null,
      "description": "Mirrors xsede#gordon",
      "disable_verify": false,
      "display_name": null,
      "expire_time": null,
      "expires_in": 0,
      "force_encryption": false,
      "gcp_connected": null,
      "gcp_paused": null,
      "globus_connect_setup_key": null,
      "host_endpoint": null,
      "host_endpoint_display_name": null,
      "host_endpoint_id": null,
      "host_path": null,
      "id": "cbfb19f5-6d04-11e5-ba46-22000b92c6ec",
      "in_use": false,
      "info_link": null,
      "is_globus_connect": false,
      "is_go_storage": false,
      "keywords": null,
      "location": "Automatic",
      "max_concurrency": 4,
      "max_parallelism": 8,
      "my_effective_roles": [],
      "myproxy_dn": null,
      "myproxy_server": null,
      "name": "gordon",
      "network_use": "normal",
      "oauth_server": "cilogon.org",
      "organization": null,
      "preferred_concurrency": 2,
      "preferred_parallelism": 4,
      "public": true,
      "s3_owner_activated": false,
      "s3_url": null,
      "shareable": true,
      "sharing_target_endpoint": null,
      "sharing_target_root_path": null,
      "subscription_id": null,
      "username": "arnoldg"
    },
    {
      "_rank": 0.421588,
      "acl_available": false,
      "acl_editable": false,
      "activated": false,
      "canonical_name": "vyekkirala#gordon",
      "contact_email": null,
      "contact_info": null,
      "default_directory": null,
      "department": null,
      "description": "Mirrors xsede#gordon except that this uses test-oa4mp.iu.xsede.org for authentication/delegation.",
      "disable_verify": false,
      "display_name": null,
      "expire_time": null,
      "expires_in": 0,
      "force_encryption": false,
      "gcp_connected": null,
      "gcp_paused": null,
      "globus_connect_setup_key": null,
      "host_endpoint": null,
      "host_endpoint_display_name": null,
      "host_endpoint_id": null,
      "host_path": null,
      "id": "cf08f264-6d04-11e5-ba46-22000b92c6ec",
      "in_use": false,
      "info_link": null,
      "is_globus_connect": false,
      "is_go_storage": false,
      "keywords": null,
      "location": "Automatic",
      "max_concurrency": 4,
      "max_parallelism": 8,
      "my_effective_roles": [],
      "myproxy_dn": null,
      "myproxy_server": null,
      "name": "gordon",
      "network_use": "normal",
      "oauth_server": "test-oa4mp.iu.xsede.org",
      "organization": null,
      "preferred_concurrency": 2,
      "preferred_parallelism": 4,
      "public": true,
      "s3_owner_activated": false,
      "s3_url": null,
      "shareable": true,
      "sharing_target_endpoint": null,
      "sharing_target_root_path": null,
      "subscription_id": null,
      "username": "vyekkirala"
    },
    {
      "_rank": 0.396413,
      "acl_available": false,
      "acl_editable": false,
      "activated": false,
      "canonical_name": "xsede#gordon",
      "contact_email": null,
      "contact_info": null,
      "default_directory": null,
      "department": null,
      "description": null,
      "disable_verify": false,
      "display_name": null,
      "expire_time": "2015-08-25T21:14:17+00:00",
      "expires_in": 0,
      "force_encryption": false,
      "gcp_connected": null,
      "gcp_paused": null,
      "globus_connect_setup_key": null,
      "host_endpoint": null,
      "host_endpoint_display_name": null,
      "host_endpoint_id": null,
      "host_path": null,
      "id": "c5e7e362-6d04-11e5-ba46-22000b92c6ec",
      "in_use": false,
      "info_link": null,
      "is_globus_connect": false,
      "is_go_storage": false,
      "keywords": null,
      "location": "Automatic",
      "max_concurrency": 4,
      "max_parallelism": 8,
      "my_effective_roles": [],
      "myproxy_dn": null,
      "myproxy_server": "myproxy.xsede.org",
      "name": "gordon",
      "network_use": "normal",
      "oauth_server": "oa4mp.xsede.org",
      "organization": null,
      "preferred_concurrency": 2,
      "preferred_parallelism": 4,
      "public": true,
      "s3_owner_activated": false,
      "s3_url": null,
      "shareable": true,
      "sharing_target_endpoint": null,
      "sharing_target_root_path": null,
      "subscription_id": "1813a867-5f94-11e4-b64e-12313940394d",
      "username": "xsede"
    }
  ]
}
  • Single endpoint. (Reference)

    GET /endpoint/ddb59aef-6d04-11e5-ba46-22000b92c6ec

    The value 'ddb59aef-6d04-11e5-ba46-22000b92c6ec' is the id of "Globus Tutorial Endpoint 1", owned by user "go", with legacy canonical name "go#ep1". Note that using the legacy canonical name will work in place of the id (GET /endpoint/go%23ep1), but this is deprecated and will be removed in the future. Use GET /endpoint_search to find endpoints and determine their id.

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "DATA": [
        {
          "DATA_TYPE": "server",
          "hostname": "ep1.transfer.globus.org",
          "id": 207976,
          "port": 2811,
          "scheme": "gsiftp",
          "subject": null,
          "uri": "gsiftp://ep1.transfer.globus.org:2811"
        }
      ],
      "acl_available": false,
      "acl_editable": false,
      "activated": false,
      "canonical_name": "go#ep1",
      "contact_email": null,
      "contact_info": null,
      "default_directory": null,
      "department": null,
      "description": null,
      "disable_verify": false,
      "display_name": "Globus Tutorial Endpoint 1",
      "expire_time": "2015-10-24T21:50:16+00:00",
      "expires_in": -1,
      "force_encryption": false,
      "gcp_connected": null,
      "gcp_paused": null,
      "globus_connect_setup_key": null,
      "host_endpoint": null,
      "host_endpoint_display_name": null,
      "host_endpoint_id": null,
      "host_path": null,
      "id": "ddb59aef-6d04-11e5-ba46-22000b92c6ec",
      "in_use": false,
      "info_link": null,
      "is_globus_connect": false,
      "is_go_storage": false,
      "keywords": null,
      "location": "Automatic",
      "max_concurrency": 4,
      "max_parallelism": 8,
      "my_effective_roles": [],
      "myproxy_dn": null,
      "myproxy_server": "myproxy.globusonline.org",
      "name": "ep1",
      "network_use": "normal",
      "oauth_server": null,
      "organization": null,
      "preferred_concurrency": 2,
      "preferred_parallelism": 4,
      "public": true,
      "s3_owner_activated": false,
      "s3_url": null,
      "shareable": true,
      "sharing_target_endpoint": null,
      "sharing_target_root_path": null,
      "subscription_id": "964be8f5-5f9b-11e4-b64e-12313940394d",
      "username": "go"
    }
  • Endpoint create. (Reference)

    POST /endpoint
    Content-Type: application/json
    
    {
      "display_name": "Big data storage at acme university",
      "oauth_server": "oauth.acme.edu",
      "DATA_TYPE": "endpoint",
      "description": "Example gridftp endpoint."
      "DATA": [
        {
          "DATA_TYPE": "server",
          "hostname": "gridftp.example.org",
          "scheme": "gsiftp",
          "port": 2811,
        }
      ],
    }

    Note the content-type header; this is required whenever POSTing or PUTing data to the API.

    201 Created
    X-Transfer-API-KOA-Version: 4.5
    Location: https://transfer.test.api.globusonline.org/v0.10/endpoint/testuser%23testep.json
    Content-Type: application/json
    
    {
      "code": "Created",
      "resource": "/endpoint",
      "DATA_TYPE": "endpoint_create_result",
      "id": "d9a5511e-687f-4e5a-9019-afe73b861199",
      "globus_connect_setup_key": null,
      "request_id": "6UKB1S7iV",
      "message": "Endpoint created successfully"
    }
  • Globus Connect Personal endpoint create. (Reference)

    POST /endpoint
    Content-Type: application/json
    
    {
      "DATA_TYPE": "endpoint",
      "description": "My work laptop running globus connect personal"
      "display_name": "Work Laptop",
      "public": false,
      "is_globus_connect": true
    }

    To complete installation of Globus Connect Personal, you must enter the setup key, which you get from the create response:

    201 Created
    Content-Type: application/json
    Location: https://transfer.api.globusonline.org/v0.10/endpoint/USERNAME%23ENDPOINT_NAME.json
    
    {
      "globus_connect_setup_key": "5c93772f-98f3-4173-bd22-5ea405177af8",
      "resource": "/endpoint",
      "DATA_TYPE": "endpoint_create_result",
      "id": "a98d9e2d-19b4-4335-a067-932157d2b339",
      "code": "Created",
      "request_id": "NwfXW3WNZ",
      "message": "Endpoint created successfully"
    }

    The globus_connect_setup_key will also be available in the endpoint representation until it is used to complete setup. It is deleted after first use.

  • Endpoint update. (Reference)

    PUT /endpoint/ID
    Content-Type: application/json
    
    {
      "DATA_TYPE": "endpoint",
      "display_name": "New name for my endpoint"
    }

    Note that the id is in the URL, not the representation itself.

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "message": "Endpoint updated successfully",
      "code": "Updated",
      "resource": "/endpoint/ENDPOINT_ID",
      "DATA_TYPE": "result",
      "request_id": "GCgXqTE9n"
    }

5.3.1. Public Endpoints

Globus users can share endpoints with one another by making the endpoint public. This can be done by setting the public property to true on an endpoint document when creating or updating the endpoint.

Globus also maintains several sets of commonly used endpoints under special usernames:

  • Globus Tutorial Endpoint 1, Globus Tutorial Endpoint 2 - These endpoints can be used by any Globus user without authenticating. They have limited disk quota, and should only be used for basic testing.

5.4. Endpoint Directory Listing

5.4.1. Directory Listing

Directory listing on an endpoint is exposed as a sub-resource of the endpoint:

GET /endpoint/ID/ls?path=/~/directory

If the endpoint connection succeeds and the path is a valid directory with appropriate permission for the user, a file_list is returned.

/\~/ is an alias for the users' home directory on the server. path can be an empty string, in which case the "default" directory is used, currently /~/.

Note that only directory listing is supported - if path points to a file, an error will be returned. Paging, filtering, ordering, and field selection are supported. Unlike most paged resources, all records are returned by default. This is because the gsiftp protocol does not support partial listing, so the entire list is always fetched.

5.5. Creating Directories

To create a directory on an endpoint, submit a mkdir document to POST /operation/endpoint/ID/mkdir

{
  "path": "/~/newdir",
  "DATA_TYPE": "mkdir"
}

If the path field does not contain an absolute path, it’s assumed to be relative to the user’s home directory (~).

A standard error document is returned on failure; on success a "mkdir_result" is returned, with status 202 and code DirectoryCreated:

{
  "message": "The directory was created successfully",
  "code": "DirectoryCreated",
  "resource": "/operation/endpoint/427a0454-77dd-45d4-89d3-282c431c6bfe/mkdir",
  "DATA_TYPE": "mkdir_result",
  "request_id": "abc123"
}

Note that recursive transfers implicitly create directories as needed at the destination; the purpose of the mkdir resource is to provide explicit creation.

5.6. Transfer Submission

A transfer is a request to copy files and directories from a source endpoint to a destination endpoint. The request document is essentially a list of transfer items containing source / destination path pairs.

To fulfill the request, the service creates a task, which can be monitored using the task_id.

For directory transfer items, the contents of the source directory are recursively copied to the destination directory, including any subdirectories. Any intermediate/parent directories that don’t exist on the destination will be created.

For file transfer items, the source file is copied to the file path specified as the destination. The destination path can’t be a directory, This is to avoid inconsistent behavior depending on whether or not the destination exists, so when run repeatedly (for example to keep two copies in sync) it performs the same operation each time.

Both endpoints need to be activated before the transfer is submitted. If an endpoint expires before the transfer is complete, the endpoints can be re-activated to allow it to continue, up until the deadline (which defaults to 24 hours after the request time).

When submitting a transfer, you must first get a submission_id:

GET /submission_id

The submission id should be saved in case the submission is interrupted before a result is received from the server. The transfer can then be resubmitted, and if the original request was successful it will not double submit, it will simply return a result indicating that it’s a duplicate id, with the id of the task created to fulfill the request.

The transfer itself is submitted via POST /transfer:

{
  "DATA_TYPE": "transfer",
  "submission_id": "VAwPR1dFRhAHQn93dmd3EkETBSs2ejJnVQRWIyp6YytFUl8O",
  "source_endpoint": "d561f96b-6161-4abd-96ad-2b14612f9fe6",
  "destination_endpoint": "e0d7e8a7-6347-40af-b5bb-df0c84731dd4",
  "label": "example transfer label",
  "sync_level": null,
  "DATA": [
    {
      "source_path": "/~/file1.txt",
      "destination_path": "/~/dir1/file1copy.txt",
      "recursive": false,
      "DATA_TYPE": "transfer_item"
    },
    {
      "source_path": "/~/some_directory/",
      "destination_path": "/~/some_directory_copy/",
      "recursive": true,
      "DATA_TYPE": "transfer_item"
    }
  ]
}

and returns a transfer_result:

{
  "submission_id": "UAlfRFdDQEsHQn8tJGd3EkETBStoemJnVQRWIyp6YytFUl8O",
  "code": "Accepted",
  "resource": "/transfer",
  "task_id": "5f63266a-f6ba-11e0-a861-f0def10a689e",
  "DATA_TYPE": "transfer_result",
  "request_id": "abc123",
  "message": "Transfer submission accepted."
}

sync_level can be used to request that only modified files are transferred, using different mechanisms to determine modification. See the transfer document type for details on the different sync levels. If sync_level is not included or null, all files will be transferred.

5.7. Task Monitoring

To track the progress of a newly submitted task, use the task_id field of the returned result document.

GET /task/TASK_ID

This returns a task document.

A request to cancel the task can be submitted like this:

POST /task/TASK_ID/cancel

It is possible that the transfer will finish before the cancellation goes through; a result document type is returned with a message describing what happened.

5.8. Delete Submission

Remote files and directories can be deleted on an endpoint by submitting a delete document to POST /delete:

{
  "submission_id": "AA1bFgMUEBgHQn8ufWd3EkETBSgzdGZnAgYBd39zYn0RCANT",
  "endpoint": "ddb59af0-6d04-11e5-ba46-22000b92c6ec",
  "recursive": false,
  "DATA_TYPE": "delete",
  "label": "example delete label",
  "length": 2,
  "ignore_missing": false,
  "DATA": [
    {
      "path": "/~/bashrc_copy_example",
      "DATA_TYPE": "delete_item"
    }
  ]
}

The submission_id, label, and deadline fields behave just like the same fields in a transfer document, and the delete_result returned after submission is the same as a transfer_result.

If any of the paths point to a directory, recursive must be set to true and the entire directory contents will be deleted. Deleting a directory only if it is empty is not supported.

If ignore_missing is not set, the job will fail and stop deleting paths if one of the paths does not exist.

To avoid breaking backward compatibility in 0.10, delete tasks are not included by default in task_list. To include delete tasks, use filter=type:TRANSFER,DELETE.

6. Common Query Parameters

Most resources support field selection using the fields parameter. Most list resources support pagination using one of the paging methods described below. Some list resources also support filtering on certain fields using a filter parameter, and sorting on certain fields using orderby.

6.1. Paging

The Transfer service uses multiple types of paging for controlling results depending on the resource used. Resource documentation will specify which type of paging that resource uses.

6.1.1. Offset Paging

Resources which use this type of paging use the offset and limit query parameters to guide pagination. The default offset is 0, while the default limit and maximum offset and limit vary among resources. Typical usage involves starting with offset 0, choosing a page size (limit=PAGE_SIZE), and incrementing offset by PAGE_SIZE to display successive pages. The limit and offset values will be echoed in the response body, and some resources will also include a has_next_page value which will be True if making a query at the next offset would yield more results.

For example, with a page size of 50:

# page 1
GET /task_list?offset=0&limit=50

# page 2
GET /task_list?offset=50&limit=50

# page 3
GET /task_list?offset=100&limit=50

6.1.2. Marker Paging

Resources which use this type of paging use the marker query parameter and the marker and next_marker response body fields to guide pagination. If the next_marker field in a response is not null, it can be passed as the marker query parameter on the next request to fetch the next page. If the next_marker field in a response is null, there are no further results. The marker query parameter used for that request will be echoed in the response body. Page size cannot be controlled for resources using this paging method.

For example:

# page 1
GET task/<task_id>/successful_transfers
{
  "marker": null,
  "next_marker": 123,
  ...
}

# page 2
GET task/<task_id>/successful_transfers?marker=123
{
  "marker": 123,
  "next_marker": 456,
  ...
}

# page 2
GET task/<task_id>/successful_transfers?marker=456
{
  "marker": 456,
  "next_marker": null,
  ...
}

6.1.3. Last Key Paging

Resources which use this type of paging use the last_key query parameter along with the last_key and has_next_page response body fields to guide pagination. If has_next_page is true, the last_key response field can be passed as a query parameter to fetch the next page. If has_next_page is false, there are no more results at the time of the request. Page size can be controlled using the limit query parameter which will be echoed in the response body.

For example:

# page 1
GET /endpoint_manager/task_list?limit=100
{
    "has_next_page": true,
    "last_key": "abc",
    "limit": 100,
    ...
}

# page 2
GET /endpoint_manager/task_list?limit=100&last_key="abc"
{
    "has_next_page": false,
    "last_key": null,
    "limit": 100,
    ...
}

6.1.4. Next Token Paging

Resources which use this type of paging use the next_token query parameter to guide pagination. Responses will include a next_token value that will either be null if there are no more results, or should be passed as the next_token query parameter to fetch the next page of results. Page size can be controlled using the max_results query parameter.

For example:

# page 1
GET /endpoint/<endpoint_xid>/shared_endpoint_list?max_results=100
{
    "next_token": "abc",
    ...
}

# page 2
GET /endpoint/<endpoint_xid>/shared_endpoint_list?max_results=100&next_token=abc
{
    "next_token": null,
    ...
}

6.2. Filtering

Only certain fields support filtering; see the reference documentation for a full list. There are several types of filters, including date range, a single value, or a list of values. See the field documentation for descriptions and examples.

This example for the task list returns ACTIVE and SUCCESSFUL tasks submitted before December 20 2010:

GET /task_list?filter=status:ACTIVE,SUCCESSFUL/request_time:,2010-12-20 00:00:00

The new convention for filters is to use separate parameters for each, of the form filter_NAME - see Endpoint Search for an example.

6.3. Sorting

The orderby parameter sets a sort field and direction. Only fields which support filtering are sortable. The value is a comma separated list of field names, with an optional direction specifier. For example:

GET /task_list?orderby=status,request_time desc

returns tasks first ordered by status, in ascending alphabetical order, then within tasks with the same status sorts by request_time, with newer tasks first (descending).

6.4. Limiting Result Fields

The fields query parameter can be used to limit which fields are included in the response, for example:

GET /task_list?fields=task_id,status

will return a task list with only task_id and status fields in each task. This can save bandwidth and parsing time if you know you only need certain fields.

Field selection can also be done on sub-documents, by prefixing the field name with the document type name. For example:

GET /endpoint_search?filter_scope=my-endpoints&fields=id,display_name

will include only the id and display_name of each endpoint.

7. Request Rate Limiting

This service will send an error if too many requests are being received. The HTTP status is 429 and the error document code is RateLimitExceeded. Upon receiving this error, a client MUST use an exponential backoff (delay). If a client does not use a delay before retrying, that client and/or user may be locked out from the service.

The rate limit is 20 requests per second, per effective identity. Metering is done on 10 second periods to allow for some bursts.

  • Transfer API Documentation
  • API Overview
  • Endpoint Activation
  • Task Submission
  • Task Monitoring
  • File Operations
  • Endpoint Management
  • Endpoint Search
  • Endpoint Roles
  • Endpoint Bookmarks
  • Endpoint ACL Management
  • Advanced Endpoint Management
  • Change History
© 2010- The University of Chicago Legal Privacy Accessibility