Globus Flows
Globus automation services provide secure and reliable task orchestration across a set of heterogeneous resources at scale. These services can be used to automate tasks as simple as replicating data across multiple storage systems, or as intricate as managing multiple conditional data analysis and results distribution tasks, with optional human intervention where needed for say review and confirmation. Globus automation services include the Flows API, which enables composition and creation of task pipelines called flows, and action providers, which provide functionality for each step of the flow.
The Globus automation overview provides a detailed guide to the ecosystem that delivers these capabilities.
1. Running Flows
When a user starts a flow, it is called a flow run, or simply a run. Users may search for flows to run, provide input parameters, and start a run. Once a run has started, users can monitor and manage the run, including view the status, cancel, and grant permission to others to manage or monitor the run.
How to Start a Run and How to Monitor a Run provide step-by-step instructions for using the Globus Web App to start and monitor runs.
The Globus CLI can also start and monitor runs. Runs can also be started automatically using the CLI, including triggering the start of a flow based on file system events. Some examples of starting runs automatically are provided.
Globus provides a couple flows that you can run by using the Globus Web App:
2. Creating Flows
Flows are a series of tasks that are executed in a specified order. The Globus Flows service provides managed and reliable execution of flows. Users may define their own flows and allow other users or groups to see, run, or manage these flows. Globus Auth and Groups services are used for authentication and authorization in Flows.
Each task, or step, in the flow calls an action provider, which can act on a resource in either Globus services or externally managed services. An action provider implements a single task, which users then reference in a flow as a step to create resilient automation pipelines. Several Globus capabilities, including data transfer and permission management, are implemented as Globus-provided action providers and are available for use. Action providers double as a powerful extension point, allowing users to define and run their own custom action providers to expand the set of tools that can be leveraged in flows.
A flow is written in two parts: A flow definition, which defines the steps in a flow and the logic between them, and an input schema, which defines the structure of the input parameters for a run of a flow.
Both the flow definition and the input schema are written in either JSON or YAML.
Once the flow definition and input schema are written, the flow is registered with Globus using the flows create
subcommand of the Globus CLI.
After a flow is created, the CLI or Globus Web App can be used to manage the flow and assign roles to the flow to determine who can see, run, or manage the flow.
How to Create a Flow gives an overview of using the Globus CLI to create and manage a flow. We provide example flow definitions to demonstrate some simple operations that can be used as initial templates for developing flows. Our flow authoring guide provides a detailed description of how to write a flow definition, and the input schema authoring guide provides information for how to write an input schema.
3. Globus-Provided Action Providers
Globus hosts a number of action providers which may be invoked directly or used within Flows:
4. Creating Action Providers
In addition to Globus-provided action providers, you can create custom action providers to incorporate other applications and capabilities into flows. For example, if you have developed a service to perform a domain-specific analysis, you can create an action provider that allows you and others to apply that analysis as an action in a Globus flow.
Globus Action Provider Tools provide a Python library for implementing custom action providers for use in Globus Flows.
5. Tutorial Resources
You can immediately start experimenting with creating and running a simple flow to move and share data using our Jupyter notebook, or watch an overview of automating instrument data handling or a tutorial on building and running your own flows.
6. CLI, SDK, and APIs
We provide a Python SDK and a command line interface as part of our client tools for use with Globus Flows and the action provider API.
-
Globus CLI: Flows Commands Install the Globus CLI to manage flows and runs.
-
Globus Python SDK The Globus Python SDK provides clients for interacting with your flows and runs.
-
Globus Python SDK Examples Examples illustrating how to use the Globus Python SDK to write applications that interact with Globus Flows.
-
Globus Automate Client Migration Guide If you are coming from the earlier Globus Automate Client application, we’ve provided a migration guide to help you transition to the Globus CLI and Globus Python SDK.
-
Globus Flows API Specification The Globus Flows API reference documentation.
-
Globus Action Provider API Specification The Globus Action Provider API reference documentation.
7. Mailing List
Release updates, feature announcements, and discussions for all Globus services take place on the Globus Discuss mailing list: discuss@globus.org
If you have any questions, please contact support@globus.org.