Looping Batched Move

Tip

This example is a demonstration of looping in a flow. For most use cases, moving a directory in a single transfer task is a more appropriate solution.

This example demonstrates while-loop-style looping in a flow. It moves files from a source to a destination until the source is empty.

The contents of the given source path are transferred in batches of 100 items and are deleted from the source path as the transfers succeed.

Note

The sample flow does not validate the results of the transfer and delete tasks.

Even if a transfer task fails, the flow will attempt to delete the batch of items. This could lead to data loss.

Even if a delete task fails, the flow will continue to loop if there are still items to transfer in the source path. This could lead to an infinite loop.

Highlights

The flow starts with an ls operation on the source in the List state, and then uses a Choice state, CheckIfDone, to inspect the results. Choice states allow flows to dispatch and make loops possible.

Although the flow has some known failure modes, it can rely on the guarantee that when the Transfer state is done executing, the data transfer will be complete. That makes it relatively safe to delete data in the Delete state.

Source code

{
  "StartAt": "List",
  "States": {
    "List": {
      "Comment": "As an improvement, it may be worthwhile to first verify that $.source.path is a directory. This can be accomplished with two states -- an 'Action' state that calls 'stat', and a 'Choice' state that confirms the type is a 'dir'.",
      "Type": "Action",
      "ActionUrl": "https://transfer.actions.globus.org/ls",
      "Parameters": {
        "endpoint_id.$": "$.source.id",
        "path.$": "$.source.path",
        "limit": 100
      },
      "ResultPath": "$.ls_results",
      "Next": "CheckIfDone"
    },
    "CheckIfDone": {
      "Comment": "This is the gatekeeper for looping. This will continue to loop until all files and directories have been transferred and deleted.",
      "Type": "Choice",
      "Choices": [
        {
          "Variable": "$.ls_results.details.total",
          "NumericEquals": 0,
          "Next": "Done"
        }
      ],
      "Default": "PrepareTransferDATA"
    },
    "Done": {
      "Type": "Pass",
      "End": true
    },
    "PrepareTransferDATA": {
      "Type": "ExpressionEval",
      "ResultPath": "$.transformations",
      "Next": "Transfer",
      "Parameters": {
        "transfer_DATA.=": "[{'source_path': source.path.rstrip('/') + '/' + item.name, 'destination_path': destination.path.rstrip('/') + '/' + item.name} for item in ls_results.details.DATA]"
      }
    },
    "Transfer": {
      "Comment": "Some validation may need to be added to verify that the items were transferred. This can be accomplished using a Choice state that looks at values in the $.transfer_results object.",
      "Type": "Action",
      "ActionUrl": "https://transfer.actions.globus.org/transfer",
      "ResultPath": "$.transfer_results",
      "Next": "PrepareDeleteDATA",
      "Parameters": {
        "DATA.$": "$.transformations.transfer_DATA",
        "source_endpoint.$": "$.source.id",
        "destination_endpoint.$": "$.destination.id"
      }
    },
    "PrepareDeleteDATA": {
      "Type": "ExpressionEval",
      "ResultPath": "$.transformations",
      "Next": "Delete",
      "Parameters": {
        "delete_DATA.=": "[{'path': source.path.rstrip('/') + '/' + item.name} for item in ls_results.details.DATA]"
      }
    },
    "Delete": {
      "Comment": "Some validation may need to be added to verify that the items were deleted. This can be accomplished using a Choice state that looks at values in the $.delete_results object. The delete task may also benefit from some options like 'ignore_missing'.",
      "Type": "Action",
      "ActionUrl": "https://transfer.actions.globus.org/delete",
      "ResultPath": "$.delete_results",
      "Next": "List",
      "Parameters": {
        "DATA.$": "$.transformations.delete_DATA",
        "endpoint.$": "$.source.id"
      }
    }
  }
}

{
  "type": "object",
  "required": [
    "source",
    "destination"
  ],
  "properties": {
    "source": {
      "type": "object",
      "title": "Source",
      "format": "globus-collection",
      "required": [
        "id",
        "path"
      ],
      "properties": {
        "id": {
          "type": "string",
          "title": "Source Collection ID",
          "format": "uuid",
          "description": "The UUID for the collection which serves as the source of the Move"
        },
        "path": {
          "type": "string",
          "title": "Source Collection Path",
          "description": "The path on the source collection for the data"
        }
      },
      "propertyOrder": [
        "id",
        "path"
      ],
      "additionalProperties": false
    },
    "destination": {
      "description": "NOTE! The *contents* of the source directory will be transferred to the directory selected here!",
      "type": "object",
      "title": "Destination",
      "format": "globus-collection",
      "required": [
        "id",
        "path"
      ],
      "properties": {
        "id": {
          "type": "string",
          "title": "Destination Collection ID",
          "format": "uuid",
          "description": "The UUID for the collection which serves as the destination for the Move"
        },
        "path": {
          "type": "string",
          "title": "Destination Collection Path",
          "description": "The path on the destination collection where the data will be stored"
        }
      },
      "propertyOrder": [
        "id",
        "path"
      ],
      "additionalProperties": false
    }
  },
  "propertyOrder": [
    "source",
    "destination"
  ],
  "additionalProperties": false
}

{
  "source": {
    "id": "6c54cade-bde5-45c1-bdea-f4bd71dba2cc",
    "path": "/~/source-directory"
  },
  "destination": {
    "id": "31ce9ba0-176d-45a5-add3-f37d233ba47d",
    "path": "/~/destination-directory"
  }
}