1. Overview

The Transfer API supports monitoring the progress of a user’s file transfer tasks, managing file transfer endpoints, listing remote directories, and submitting new transfer tasks.

This documentation assumes a basic familiarity with HTTP, including the GET, POST, PUT, and DELETE request methods, Content-Type and Accepts headers, and standard status codes.

2. Authentication

Authentication to the Transfer API requires using the Globus Auth API to obtain an access token. The Globus Auth developer guide describes the steps to obtain a token. The Globus Python SDK provides support for Transfer and Globus Auth and can be integrated with your application.

Authentication using X509 credentials is deprecated and will be removed by June 30, 2017.

2.1. Obtaining Tokens

2.1.1. Thick Clients

Native applications and thick clients should use the Native App flow in Globus Auth to obtain tokens. If automated or recurring access to the Transfer service is required, refresh tokens should be requested with the native app flow. Sample scripts that show native app flow and use of refresh tokens are provided here: https://github.com/globus/native-app-examples

2.1.2. Web Applications

Web applications should use the Authorization Code grant in Globus Auth to obtain tokens.

In this method, a web application redirects the user agent (user’s web browser) to Globus to authenticate, and provides a callback URL. Globus prompts the user to sign in, and after successful sign in confirms that the user would like to give the application access to their account. It then redirects the user agent back to the web application at the callback URL and passes it a "request token", which is different from an access token. The web application uses the request token and its own client credential to get an access token. This workflow allows the user to manage application permissions and revoke access on a per application basis in the future if needed, and prevents the user’s password from flowing through third party web applications.

2.2. Using Tokens

The Transfer token needs to be passed to the Transfer API in the Authorization header with custom method Bearer:

Authorization: Bearer TOKEN

You can test this by saving the token to a private temporary file, e.g. $HOME/tmp-globus-auth-token.txt, and then passing it to curl with the -H option:

TOKEN=$(cat $HOME/tmp-globus-auth-token.txt)
curl -H "Authorization : Bearer $TOKEN" \
    'https://transfer.api.globusonline.org/v0.10/task_list'

You could also paste the token directly but then it will be stored in your shell history, which is often not desirable.

2.3. Terminology

2.3.1. Transfer Terminology

  • task - a batch of file transfers operations that were submitted together, identified by an ID string.

  • endpoint - a definition for a gridftp server (or other file transfer source / destination), with a convenient name. Endpoints have a display name, which will not necessarily be unique, and a string id which is globally unique. For legacy usage, they also have a globally unique canonical name of the form USERNAME#NAME, where USERNAME is the user who created the endpoint, and NAME is the legacy endpoint name.

  • activation - delegating a temporary credential to the Globus transfer service to perform directory listing and transfers on behalf of the user.

2.3.2. API Terminology

  • resource - a URL addressable part of the API, which can be interacted with using a subset of the GET, POST, PUT, and DELETE HTTP methods.

  • document - a representation of data, returned by resources as output and accepted by resources as input. There are several standard document types, and some types include sub-documents (for example, the task_list type is a container for many documents of type task).

2.4. Base URL

All the URLs in the examples below should be taken relative to the Transfer API root:

https://transfer.api.globusonline.org/v0.10

so the full URL to /task_list will be:

https://transfer.api.globusonline.org/v0.10/task_list

Clients should store the base URL in one place and use it when constructing resource URLs, to simplify changing versions.

The API exposes lists of resources and allows fetching single resources by name. For example, a list of tasks is available at /task_list, and a task with id 123-abc is accessed with /task/123-abc. This convention is used for all resource URL patterns.

Many resources provide links to related resources; these can be used instead of hard-coding URL patterns, making the client more robust to changes in future versions of the API.

2.6. Paged Lists

The task_list, event_list, and endpoint_search resources support pagination via query parameters. By default you will only get the first 10 records. Getting all records is currently not supported. Different records can be selected using the limit and offset query parameters. See the Paging section for details.

2.7. Document Formats

The API uses json for all input and output, including error documents. Some resources have legacy support for html, but that is deprecated and will be removed in the future.

Note that application/x-www-form-urlencoded is not supported. The body should contain the actual JSON data, not a form encoded version of that data.

The json representation uses a "DATA_TYPE" key to specify the type of resource and a "DATA" key containing a list of sub-documents, if any. For example, the endpoint document type is described in detail here:

2.8. Errors

When an error occurs an HTTP status code >=400 will be returned. The body of the response will be a JSON document with details about the error, including a code field. The error code will also be provided in the "X-Transfer-API-Error" header. Note that requests outside the API path version prefix may return an HTML or plaintext error body instead. Here is an example error returned when an endpoint is not found:

{
  "code": "EndpointNotFound",
  "message": "No such endpoint '23c1a962-7e68-11e5-ac37-f0def10a689e'",
  "request_id": "HrbjJy3QJ",
  "resource": "/endpoint/23c1a962-7e68-11e5-ac37-f0def10a689e"
}

A 404 status code is used for this response. The code field has the same value as the X-Transfer-API-Error header, for convenient access.

3. Examples

3.1. Conventions

The convention used for examples in this document is similar to raw HTTP requests and responses, with the URL shortened and most headers omitted. As an example, to get a task_list for the logged in user, the request is described as:

GET /task_list

This means that a GET request must be made to the task_list resource, which actually has the URL https://transfer.api.globusonline.org/v0.10/task_list for version v0.10. This is BASE_URL + /task_list. As discussed above, the BASE_URL should be set in one place and re-used, not hard coded into each request. The actual raw HTTP request will typically include many headers:

GET /v0.10/task_list HTTP/1.1
Host: transfer.api.globusonline.org
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20100101 Firefox/4.0.1 Iceweasel/4.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: UTF-8,*
Keep-Alive: 115
Connection: keep-alive
X-Transfer-API-X509-User: testuser

Most of these headers were added by the browser (Firefox); the developer will not normally need to deal with them.

For examples that involve sending data, the body is included inline, just like it would be in an HTTP request. For example endpoint creation is described like this:

POST /endpoint
Content-Type: application/json

{
  "display_name": "ACME University shared storage",
  "DATA_TYPE": "endpoint",
  "description": "Example gridftp endpoint."
  "DATA": [
    {
      "DATA_TYPE": "server",
      "hostname": "gridftp.example.org",
      "scheme": "gsiftp",
      "port": 2811,
    }
  ],
}

This means that to create an endpoint, a request using method POST can be made to BASE_URL + /endpoint, with header content-type set to "application/json", and having as the request body the JSON data describing the endpoint. Other headers are required for authentication, but they are not specific to this request.

This format is used to provide a quick description of how to make a request, independent of the client used. The Python and Java examples hide many of the details involved in accessing the API; this document is focused on describing the API itself including those details.

3.2. Monitoring

  • Paged task list with sorting and field selection. (Reference)

    GET /task_list?offset=0&limit=10&fields=task_id,request_time&orderby=request_time

    Lists the first 10 tasks belonging to the currently logged in user, showing only the task_id and request_time fields, ordered by request_time (ascending/oldest first).

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "DATA_TYPE": "task_list",
      "length": 3,
      "limit": "10",
      "offset": "0",
      "total": "3",
      "DATA": [
        {
          "task_id": "3949cec8-7cc8-11e0-82be-12313932c1e0",
          "DATA_TYPE": "task",
          "request_time": "2011-05-12 18:49:22"
        },
        {
          "task_id": "edebec3a-7cc8-11e0-82be-12313932c1e0",
          "DATA_TYPE": "task",
          "request_time": "2011-05-12 18:52:11"
        },
        {
          "task_id": "35115208-7cc9-11e0-82be-12313932c1e0",
          "DATA_TYPE": "task",
          "request_time": "2011-05-12 18:54:34"
        },
      ]
    }
  • Event list. (Reference)

    GET /task/3949cec8-7cc8-11e0-82be-12313932c1e0/event_list

    List all events associated with a task. Events include starting and finishing the transfer, cancelation, progress reports of bytes transferred so far, and any errors encountered.

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "DATA_TYPE": "event_list",
      "length": 2,
      "limit": "10",
      "offset": "0",
      "total": "2",
      "DATA": [
        {
          "code": "SUCCEEDED",
          "description": "The operation succeeded",
          "DATA_TYPE": "event",
          "parent_task_id": "8cb34a9e-7cc8-11e0-82be-12313932c1e0",
          "details": "bytes=3103 mbps=0.000",
          "time": "2011-05-12 18:49:25"
        },
        {
          "code": "STARTED",
          "description": "The operation was started or restarted",
          "DATA_TYPE": "event",
          "parent_task_id": "8cb34a9e-7cc8-11e0-82be-12313932c1e0",
          "details": "Starting at offset 0",
          "time": "2011-05-12 18:49:25"
        }
      ]
    }

3.3. Endpoint Management

  • Endpoint search (Reference)

    GET /endpoint_search?filter_scope=my-endpoints
    GET /endpoint_search?filter_scope=recently-used
    GET /endpoint_search?filter_scope=all&filter_fulltext=xsede+gordon

    List all endpoints owned by the current user, used recently by the user in transfer or delete tasks, or containing the specified search terms. The results for the "XSEDE gordon" search are shown below:

200 OK
X-Transfer-API-KOA-Version: 4.5
Content-Type: application/json

{
 u'DATA_TYPE': u'endpoint_list',
 u'has_next_page': False,
 u'limit': 3,
 u'offset': 0
 u'DATA': [
   {u'_rank': 0.421588,
    u'acl_available': False,
    u'acl_editable': False,
    u'activated': False,
    u'canonical_name': u'arnoldg#gordon',
    u'contact_email': None,
    u'contact_info': None,
    u'default_directory': u'/~/',
    u'department': None,
    u'description': u'Mirrors xsede#gordon',
    u'disable_verify': False,
    u'display_name': None,
    u'expire_time': None,
    u'expires_in': 0,
    u'force_encryption': False,
    u'gcp_connected': None,
    u'gcp_paused': None,
    u'globus_connect_setup_key': None,
    u'host_endpoint': None,
    u'host_endpoint_display_name': None,
    u'host_endpoint_id': None,
    u'host_path': None,
    u'id': u'cbfb19f5-6d04-11e5-ba46-22000b92c6ec',
    u'in_use': False,
    u'info_link': None,
    u'is_globus_connect': False,
    u'is_go_storage': False,
    u'keywords': None,
    u'location': u'Automatic',
    u'max_concurrency': 4,
    u'max_parallelism': 8,
    u'my_effective_roles': [],
    u'myproxy_dn': None,
    u'myproxy_server': None,
    u'name': u'gordon',
    u'network_use': u'normal',
    u'oauth_server': u'cilogon.org',
    u'organization': None,
    u'preferred_concurrency': 2,
    u'preferred_parallelism': 4,
    u'public': True,
    u's3_owner_activated': False,
    u's3_url': None,
    u'shareable': True,
    u'sharing_target_endpoint': None,
    u'sharing_target_root_path': None,
    u'subscription_id': None,
    u'username': u'arnoldg'},
   {u'_rank': 0.421588,
    u'acl_available': False,
    u'acl_editable': False,
    u'activated': False,
    u'canonical_name': u'vyekkirala#gordon',
    u'contact_email': None,
    u'contact_info': None,
    u'default_directory': None,
    u'department': None,
    u'description': u'Mirrors xsede#gordon except that this uses test-oa4mp.iu.xsede.org for authentication/delegation.',
    u'disable_verify': False,
    u'display_name': None,
    u'expire_time': None,
    u'expires_in': 0,
    u'force_encryption': False,
    u'gcp_connected': None,
    u'gcp_paused': None,
    u'globus_connect_setup_key': None,
    u'host_endpoint': None,
    u'host_endpoint_display_name': None,
    u'host_endpoint_id': None,
    u'host_path': None,
    u'id': u'cf08f264-6d04-11e5-ba46-22000b92c6ec',
    u'in_use': False,
    u'info_link': None,
    u'is_globus_connect': False,
    u'is_go_storage': False,
    u'keywords': None,
    u'location': u'Automatic',
    u'max_concurrency': 4,
    u'max_parallelism': 8,
    u'my_effective_roles': [],
    u'myproxy_dn': None,
    u'myproxy_server': None,
    u'name': u'gordon',
    u'network_use': u'normal',
    u'oauth_server': u'test-oa4mp.iu.xsede.org',
    u'organization': None,
    u'preferred_concurrency': 2,
    u'preferred_parallelism': 4,
    u'public': True,
    u's3_owner_activated': False,
    u's3_url': None,
    u'shareable': True,
    u'sharing_target_endpoint': None,
    u'sharing_target_root_path': None,
    u'subscription_id': None,
    u'username': u'vyekkirala'},
   {u'_rank': 0.396413,
    u'acl_available': False,
    u'acl_editable': False,
    u'activated': False,
    u'canonical_name': u'xsede#gordon',
    u'contact_email': None,
    u'contact_info': None,
    u'default_directory': None,
    u'department': None,
    u'description': None,
    u'disable_verify': False,
    u'display_name': None,
    u'expire_time': u'2015-08-25T21:14:17+00:00',
    u'expires_in': 0,
    u'force_encryption': False,
    u'gcp_connected': None,
    u'gcp_paused': None,
    u'globus_connect_setup_key': None,
    u'host_endpoint': None,
    u'host_endpoint_display_name': None,
    u'host_endpoint_id': None,
    u'host_path': None,
    u'id': u'c5e7e362-6d04-11e5-ba46-22000b92c6ec',
    u'in_use': False,
    u'info_link': None,
    u'is_globus_connect': False,
    u'is_go_storage': False,
    u'keywords': None,
    u'location': u'Automatic',
    u'max_concurrency': 4,
    u'max_parallelism': 8,
    u'my_effective_roles': [],
    u'myproxy_dn': None,
    u'myproxy_server': u'myproxy.xsede.org',
    u'name': u'gordon',
    u'network_use': u'normal',
    u'oauth_server': u'oa4mp.xsede.org',
    u'organization': None,
    u'preferred_concurrency': 2,
    u'preferred_parallelism': 4,
    u'public': True,
    u's3_owner_activated': False,
    u's3_url': None,
    u'shareable': True,
    u'sharing_target_endpoint': None,
    u'sharing_target_root_path': None,
    u'subscription_id': u'1813a867-5f94-11e4-b64e-12313940394d',
    u'username': u'xsede'}],
}
  • Single endpoint. (Reference)

    GET /endpoint/ddb59aef-6d04-11e5-ba46-22000b92c6ec

    The value ddb59aef-6d04-11e5-ba46-22000b92c6ec is the id of "Globus Tutorial Endpoint 1", owned by user "go", with legacy canonical name "go#ep1". Note that using the legacy canoncical name will work in place of the id (GET /endpoint/go%23ep1), but this is deprecated and will be removed in the future. Use GET /endpoint_search to find endpoints and determine their id.

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "DATA": [
        {
          "DATA_TYPE": "server",
          "hostname": "ep1.transfer.globus.org",
          "id": 207976,
          "is_connected": true,
          "is_paused": false,
          "port": 2811,
          "scheme": "gsiftp",
          "subject": null,
          "uri": "gsiftp://ep1.transfer.globus.org:2811"
        }
      ],
      "acl_available": false,
      "acl_editable": false,
      "activated": false,
      "canonical_name": "go#ep1",
      "contact_email": null,
      "contact_info": null,
      "default_directory": null,
      "department": null,
      "description": null,
      "disable_verify": false,
      "display_name": "Globus Tutorial Endpoint 1",
      "expire_time": "2015-10-24T21:50:16+00:00",
      "expires_in": -1,
      "force_encryption": false,
      "gcp_connected": null,
      "gcp_paused": null,
      "globus_connect_setup_key": null,
      "host_endpoint": null,
      "host_endpoint_display_name": null,
      "host_endpoint_id": null,
      "host_path": null,
      "id": "ddb59aef-6d04-11e5-ba46-22000b92c6ec",
      "in_use": false,
      "info_link": null,
      "is_globus_connect": false,
      "is_go_storage": false,
      "keywords": null,
      "location": "Automatic",
      "max_concurrency": 4,
      "max_parallelism": 8,
      "my_effective_roles": [],
      "myproxy_dn": null,
      "myproxy_server": "myproxy.globusonline.org",
      "name": "ep1",
      "network_use": "normal",
      "oauth_server": null,
      "organization": null,
      "preferred_concurrency": 2,
      "preferred_parallelism": 4,
      "public": true,
      "s3_owner_activated": false,
      "s3_url": null,
      "shareable": true,
      "sharing_target_endpoint": null,
      "sharing_target_root_path": null,
      "subscription_id": "964be8f5-5f9b-11e4-b64e-12313940394d",
      "username": "go"
    }
  • Endpoint create. (Reference)

    POST /endpoint
    Content-Type: application/json
    
    {
      "display_name": "Big data storage at acme university",
      "oauth_server": "oauth.acme.edu",
      "DATA_TYPE": "endpoint",
      "description": "Example gridftp endpoint."
      "DATA": [
        {
          "DATA_TYPE": "server",
          "hostname": "gridftp.example.org",
          "scheme": "gsiftp",
          "port": 2811,
        }
      ],
    }

    Note the content-type header; this is required whenever POSTing or PUTing data to the API.

    201 Created
    X-Transfer-API-KOA-Version: 4.5
    Location: https://transfer.test.api.globusonline.org/v0.10/endpoint/testuser%23testep.json
    Content-Type: application/json
    
    {
      "code": "Created",
      "resource": "/endpoint",
      "DATA_TYPE": "endpoint_create_result",
      "id": "d9a5511e-687f-4e5a-9019-afe73b861199",
      "globus_connect_setup_key": null,
      "request_id": "6UKB1S7iV",
      "message": "Endpoint created successfully"
    }
  • Globus Connect Personal endpoint create. (Reference)

    POST /endpoint
    Content-Type: application/json
    
    {
      "DATA_TYPE": "endpoint",
      "description": "My work laptop running globus connect personal"
      "display_name": "Work Laptop",
      "public": false,
      "is_globus_connect": true
    }

    To complete installation of Globus Connect Personal, you must enter the setup key, which you get from the create response:

    201 Created
    Content-Type: application/json
    Location: https://transfer.api.globusonline.org/v0.10/endpoint/USERNAME%23ENDPOINT_NAME.json
    
    {
      "globus_connect_setup_key": "5c93772f-98f3-4173-bd22-5ea405177af8",
      "resource": "/endpoint",
      "DATA_TYPE": "endpoint_create_result",
      "id": "a98d9e2d-19b4-4335-a067-932157d2b339",
      "code": "Created",
      "request_id": "NwfXW3WNZ",
      "message": "Endpoint created successfully"
    }

    The globus_connect_setup_key will also be available in the endpoint representation until it is used to complete setup. It is deleted after first use.

  • Endpoint update. (Reference)

    PUT /endpoint/ID
    Content-Type: application/json
    
    {
      "DATA_TYPE": "endpoint",
      "display_name": "New name for my endpoint"
    }

    Note that the id is in the URL, not the representation itself.

    200 OK
    X-Transfer-API-KOA-Version: 4.5
    Content-Type: application/json
    
    {
      "message": "Endpoint updated successfully",
      "code": "Updated",
      "resource": "/endpoint/ENDPOINT_ID",
      "DATA_TYPE": "result",
      "request_id": "GCgXqTE9n"
    }

3.3.1. Public Endpoints

Globus users can share endpoints with one another by making the endpoint public. This can be done by setting the public property to true on an endpoint document when creating or updating the endpoint.

Globus also maintains several sets of commonly used endpoints under special usernames:

  • Globus Tutorial Endpoint 1, Globus Tutorial Endpoint 2 - These endpoints can be used by any Globus user without authenticating. They have limited disk quota, and should only be used for basic testing.

3.4. Endpoint Directory Listing

3.4.1. Endpoint Activation

Getting a directory listing from an endpoint requires activating the endpoint - providing the service with a credential, so the service can perform the operation on behalf of the user.

The first step in activation is determining what activation methods are supported by the endpoint, and what data is needed to perform the activation. This information is exposed in the activation_requirements resource:

GET /endpoint/ID/activation_requirements

The API currently supports two activation methods: myproxy and delegate_proxy. myproxy activation accepts a MyProxy server and login information, and the service uses this information to request a time limited credential for that user. If an endpoint has a default myproxy configured, that will be pre-filled in to the requirements. delegate_proxy activation is designed for clients that already have a copy of the user’s credential (or a proxy of their credential). The server provides a public key, and the client must create a delegated X.509 proxy credential using that public key, signed by the local credential.

All endpoints support delegate_proxy activation, but some endpoints may not allow myproxy activation.

To activate an endpoint, pick one of the supported activation methods, fill in or overwrite value properties on the requirements as needed, and POST the activation_requirements back:

POST /endpoint/ID/activate

For more details see the API reference for /endpoint/ID/activate.

Auto-Activation

The Globus tutorial endpoints ("Globus Tutorial Endpoint 1", "Globus Tutorial Endpoint 2") and all Globus Connect Personal endpoints do not require external credentials, and can be activated without specifying any myproxy credentials. This is done by POSTing an empty body to /endpoint/ID/autoactivate.

Endpoints with a default MyProxy or OAuth MyProxy server also support auto-activation, by using a cached credential. When you activate an endpoint from a given myproxy server, you can auto-activate other endpoints that have that myproxy server configured as the default. For example, all XSEDE endpoints are configured with the XSEDE OAuth MyProxy server as the default, so once you activate a single XSEDE endpoint, the other XSEDE endpoints can be auto-activated, without having to specify the myproxy credentials again. This also works if the user has logged in to www.globus.org using their XSEDE identity.

Autoactivation can also be done conditionally, by passing the if_expires_in query parameter. It takes an integer value in seconds, and only attempts to autoactivate the endpoint if it’s not activated, or if the current credential will expire within the specified number of seconds. This will work even on endpoints that don’t normally support autoactivation (see failure case below), so it’s useful to call this on all endpoints before attempting a more complex activation flow that may require the user to enter credentials, and without having to check the activated state on the endpoint. A reasonable value to use is 7200 seconds, or 2 hours. When submitting a transfer or delete task that may take a very long time, a much higher value could be used, to make sure the user provides a credential with a long lifetime.

If auto-activation fails (e.g. if no cached credential is present), activate returns an activation_requirement list as part of the activation_result. This allows clients to attempt auto-activation on all endpoints; if that fails, they can use the activation_requirement list to prompt the user for the required data and try again using manual activation, without having to do another round trip requesting the activation_requirements. The activation_result can be POSTed back to /endpoint/ID/activate after the required fields are filled in; activate accepts both activation_result and activation_requirements resources as input, and ignores all the fields except for the activation_requirement sub-documents.

OAuth and Activation

Some MyProxy servers provide an OAuth interface for fetching credentials. The simplest method for clients to make use of this feature is to direct users to open their web browser and activate via the globus.org website. If that is not an option, to use OAuth MyProxy more directly, a client would need to perform the OAuth process itself to get a credential, and then use delegate_proxy activation to delegate a credential to the transfer service. There is a oauth_server field in endpoint, activation_requirements, and activation_result documents that indicates the hostname of the oauth server.

Activation Options

The following query parameters are supported by /endpoint/ID/activate:

  • timeout - time in seconds to wait for a response from the remote myproxy server before giving up.

  • if_expires_in - only activate if the endpoint is not already activated or is activated but expires within the specified number of seconds.

Note that both use seconds as the unit; all time deltas in the API use seconds.

3.4.2. Directory Listing

Directory listing on an endpoint is exposed as a sub-resource of the endpoint:

GET /endpoint/ID/ls?path=/~/directory

If the endpoint connection succeeds and the path is a valid directory with appropriate permission for the user, a file_list is returned.

/~/ is an alias for the users' home directory on the server. path can be an empty string, in which case the "default" directory is used, currently /~/.

Note that only directory listing is supported - if path points to a file, an error will be returned. Paging, filtering, ordering, and field selection are supported. Unlike most paged resources, all records are returned by default. This is because the gsiftp protocol does not support partial listing, so the entire list is always fetched.

3.5. Creating Directories

To create a directory on an endpoint, submit a mkdir document to POST /operation/endpoint/ID/mkdir

{
  "path": "/~/newdir",
  "DATA_TYPE": "mkdir"
}

If the path field does not contain an absolute path, it’s assumed to be relative to the user’s home directory (~).

A standard error document is returned on failure; on sucess a "mkdir_result" is returned, with status 202 and code DirectoryCreated:

{
  "message": "The directory was created successfully",
  "code": "DirectoryCreated",
  "resource": "/operation/endpoint/427a0454-77dd-45d4-89d3-282c431c6bfe/mkdir",
  "DATA_TYPE": "mkdir_result",
  "request_id": "abc123"
}

Note that recursive transfers implicitly create directories as needed at the destination; the purpose of the mkdir resource is to provide explicit creation.

3.6. Transfer Submission

A transfer is a request to copy files and directories from a source endpoint to a destination endpoint. The request document is essentially a list of transfer items containing source / destination path pairs, with flags to indicate if the path is a directory to be copied recursively or a single file to be transfered. To fullfill the request, the service creates a task, which can be monitored using the task_id.

For recursive (directory) transfer items, the contents of the source directory are copied to the destination directory, including any subdirectories. Any intermediate/parent directories that don’t exist on the destination will be created.

For non-recursive (file) transfer items, the source file is copied to the file path specified as the destination. The destination path can’t be a directory, This is to avoid inconsistent behavior depending on whether or not the destination exists, so when run repeatedly (for example to keep two copies in sync) it performs the same operation each time.

Both endpoints need to be activated before the transfer is submitted. If an endpoint expires before the transfer is complete, the endpoints can be re-activated to allow it to continue, up until the deadline (which defaults to 24 hours after the request time).

When submitting a transfer, you must first get a submission_id:

GET /submission_id

The submission id should be saved in case the submission is interrupted before a result is received from the server. The transfer can then be resubmitted, and if the original request was successful it will not double submit, it will simply return a result indicating that it’s a duplicate id, with the id of the task created to fulfill the request.

The transfer itself is submitted via POST /transfer:

{
  "DATA_TYPE": "transfer",
  "submission_id": "VAwPR1dFRhAHQn93dmd3EkETBSs2ejJnVQRWIyp6YytFUl8O",
  "source_endpoint": "d561f96b-6161-4abd-96ad-2b14612f9fe6",
  "destination_endpoint": "e0d7e8a7-6347-40af-b5bb-df0c84731dd4",
  "label": "example transfer label",
  "sync_level": null,
  "DATA": [
    {
      "source_path": "/~/file1.txt",
      "destination_path": "/~/dir1/file1copy.txt",
      "recursive": false,
      "DATA_TYPE": "transfer_item"
    }
    {
      "source_path": "/~/some_directory/",
      "destination_path": "/~/some_directory_copy/",
      "recursive": true,
      "DATA_TYPE": "transfer_item",
    }
  ]
}

and returns a transfer_result:

{
  "submission_id": "UAlfRFdDQEsHQn8tJGd3EkETBStoemJnVQRWIyp6YytFUl8O",
  "code": "Accepted",
  "resource": "/transfer",
  "task_id": "5f63266a-f6ba-11e0-a861-f0def10a689e",
  "DATA_TYPE": "transfer_result",
  "request_id": "abc123",
  "message": "Transfer submission accepted.",
}

sync_level can be used to request that only modified files are transferred, using different mechanisms to determine modification. See the transfer document type for details on the different sync levels. If sync_level is not included or null, all files will be transferred.

3.7. Task Monitoring

To track the progress of a newly submitted task, use the task_id field of the returned result document.

GET /task/TASK_ID

This returns a task document.

A request to cancel the task can be submitted like this:

POST /task/TASK_ID/cancel

It is possible that the transfer will finish before the cancellation goes through; a result document type is returned with a message describing what happened.

3.8. Delete Submission

Remote files and directories can be deleted on an endpoint by submitting a delete document to POST /delete:

{
  "submission_id": "AA1bFgMUEBgHQn8ufWd3EkETBSgzdGZnAgYBd39zYn0RCANT",
  "endpoint": "ddb59af0-6d04-11e5-ba46-22000b92c6ec",
  "recursive": false,
  "DATA_TYPE": "delete",
  "label": "example delete label",
  "length": 2,
  "ignore_missing": false,
  "DATA": [
    {
      "path": "/~/bashrc_copy_example",
      "DATA_TYPE": "delete_item"
    }
  ]
}

The submission_id, label, and deadline fields behave just like the same fields in a transfer document, and the delete_result returned after submission is the same as a transfer_result.

If any of the paths point to a directory, recursive must be set to true and the entire directory contents will be deleted. Deleting a directory only if it is empty is not supported.

If ignore_missing is not set, the job will fail and stop deleting paths if one of the paths does not exist.

To avoid breaking backward compatibility in 0.10, delete tasks are not included by default in task_list. To include delete tasks, use filter=type:TRANSFER,DELETE.

4. Common Query Parameters

Most resources support field selection using the fields paramater. List resources support pagination using limit and offset, filtering on certain fields using a filter parameter, and sorting on certain fields using orderby.

4.1. Paging

List resources which use paging can be controlled with the offset and limit query parameters. The default offset is 0, while the default limit and maximum offset and limit vary among resources. Most list resources have a default limit of 10 and a maximum of 1000. Typical usage involves starting with offset 0, choosing a page size (limit=PAGE_SIZE), and incrementing offset by PAGE_SIZE to display successive pages.

For example, with a page size of 50:

# page 1
GET /task_list?offset=0&limit=50

# page 2
GET /task_list?offset=50&limit=50

# page 3
GET /task_list?offset=100&limit=50

4.2. Filtering

Only certain fields support filtering; see the reference documentation for a full list. There are several types of filters, including date range, a single value, or a list of values. See the field documentation for descriptions and examples.

This example for the task list returns ACTIVE and SUCCESSFUL tasks submitted before December 20 2010:

GET /task_list?filter=status:ACTIVE,SUCCESSFUL/request_time:,2010-12-20 00:00:00

The new convention for filters is to use separate parameters for each, of the form filter_NAME - see Endpoint Search for an example.

4.3. Sorting

The orderby parameter sets a sort field and direction. Only fields which support filtering are sortable. The value is a comma separated list of field names, with an optional direction specifier. For example:

GET /task_list?orderby=status,request_time desc

returns tasks first ordered by status, in ascending alphabetical order, then within tasks with the same status sorts by request_time, with newer tasks first (descending).

4.4. Limiting Result Fields

The fields query parameter can be used to limit which fields are included in the response, for example:

GET /task_list?fields=task_id,status

will return a task list with only task_id and status fields in each task. This can save bandwidth and parsing time if you know you only need certain fields.

Field selection can also be done on sub-documents, by prefixing the field name with the document type name. For example:

GET /endpoint_search?filter_scope=my-endpoints&fields=id,display_name

will include only the id and display_name of each endpoint.


© 2010- The University of Chicago Legal