Globus automation services provide secure and reliable task orchestration across a set of heterogeneous resources at scale. These services can be used to automate tasks as simple as replicating data across multiple storage systems, or as intricate as managing multiple conditional data analysis and results distribution tasks, with optional human intervention where needed for say review and confirmation. Globus automation services include the Flows API, which enables composition and deployment of task pipelines called flows, and action providers, which provide functionality for each step of the flow.
The Globus automation overview provides a detailed guide to the ecosystem that delivers these capabilities.
When a user starts a flow, it is called a flow run, or simply a run. Users may search for flows to run, provide input parameters, and start a run. Once a run has started, users can monitor and manage the run, including view the status, cancel, and grant permission to others to manage or monitor the run.
A command line interface is also available to start runs, and monitor them. Runs can also be started automatically using the CLI, including triggering the start of a flow based on file system events. Some examples of starting runs automatically are provided.
Globus provides a couple flows that you can run by using the Globus Web App:
Flows are a series of tasks that are executed in a specified order. The Globus Flows service provides managed and reliable execution of flows. Users may define their own flows and allow other users or groups to see, run, or manage these flows. Globus Auth and Groups services are used for authentication and authorization in Flows.
Each task, or step, in the flow calls an action provider, which can act on a resource in either Globus services or externally managed services. An action provider implements a single task, which users then reference in a flow as a step to create resilient automation pipelines. Several Globus capabilities, including data transfer and permission management, are implemented as Globus-provided action providers and are available for use. Action providers double as a powerful extension point, allowing users to define and run their own custom action providers to expand the set of tools that can be leveraged in flows.
A flow is written in two parts: A flow definition, which defines the steps in a flow and the logic between them, and an input schema, which defines the structure of the input parameters for a run of a flow. Both the flow definition and the input schema are written in either JSON or YAML. Once the flow definition and input schema are written, the flow is registered with Globus using the flow deploy command of the Automate CLI. After a flow is deployed, the CLI or Globus Web App can be used to manage the flow and assign roles to the flow to determine who can see, run, or manage the flow.
How to Create a Flow gives an overview of using the Automate CLI to create, deploy, and manage a flow. We provide example flow definitions to demonstrate some simple operations that can be used as initial templates for developing flows. Our flow authoring guide provides a detailed description of how to write a flow definition and input schema.
Globus provides and operates a number of action providers which may be invoked directly or used within Flows:
In addition to Globus-provided action providers, you can create custom action providers to incorporate other applications and capabilities into flows. For example, if you have developed a service to perform a domain-specific analysis, you can create an action provider that allows you and others to apply that analysis as an action in a Globus flow.
Globus Action Provider Tools provide a Python library for implementing custom action providers for use in Globus Flows.
We provide a Python SDK and a command line interface as part of our client tools for use with Globus Flows and the action provider API.
Globus Automate Client: QuickStart Install the Python SDK and CLI for interacting with flows and action providers.
Globus Automate Command Line Interface A text-based interface for use with Globus Flows.
Globus Automate Client The Globus Automate Client GitHub repository.
Globus Automate Python SDK Python clients for interacting with Globus Flows.
globus_automate_client.ActionClientis a client class with methods for interacting with action provider interface compliant services.
globus_automate_client.FlowsClientis a client class for deploying and interacting with Flows via the Flows API.
Globus Flows API Specification The Globus Flows API reference documentation.
Globus Action Provider API Specification The Globus Action Provider API reference documentation.