Globus Connect Server Streaming Guide

Table of Contents

1. Introduction
2. Key Highlights
3. High Level User Walk Through
4. Admin Guide
- 4.1. Deploying Globus Connect Server with Streaming Support
- 4.2. Creating a Stream Gateway and Stream Access Point
5. User Guide
6. Additional Reading
7. Support

1. Introduction

Globus streaming enables applications to stream data securely across wide area networks (WANs). Example use cases include streaming data from scientific instruments to high-performance computing (HPC) centers for real-time processing. The bidirectional nature of these streams enables feedback loops, supporting scenarios requiring near-real-time steering or control of remote instruments.

As a capability of Globus Connect Server (GCS), Globus data streaming leverages the same mechanisms used for establishing data transfer channels. An administrator deploys a stream gateway on the GCS endpoint and configures a stream access point. Stream access points support the same authentication and authorization mechanisms available for mapped collections used for data transfer, ensuring a consistent security posture across these constructs.

Users authenticate to stream access points and establish secure tunnels between them. The tunnel information is then used by applications to seamlessly stream data to each other via the tunnel. Globus provides a library that transparently handles tunnel communications. Applications simply read from and write to local ports while the library routes traffic through the secure tunnel without requiring application code modifications.

The Globus web application and CLI offer interfaces for discovering stream access points, creating tunnels, and monitoring and managing established tunnels.

2. Key Highlights

Provides authenticated, bidirectional data streaming across WAN without requiring pre-deployed keys (e.g., SSH keys)
The entire tunnel route is authenticated securing each leg of the connection. End-to-end encryption is left to the users' applications.
Leverages well-established mechanisms for secure wide area network connections used in Globus data transfer
Consistent security model, where GCS security configuration (authentication and authorization policies) is applied to streaming capabilities
Globus provided tooling for minimal to no code change in the applications that stream data

3. High Level User Walk Through

The user discovers the stream access points on resources they want to stream between, for example, stream access points on GCS deployments at an instrument facility and at an HPC center. They then authenticate to meet the policy on each of the stream access points, and submit a request to create a tunnel between the two access points.
Globus transfer service uses the control channel connection to the GCS deployments at both sites to establish a secure tunnel between the access points.
A tunnel identifier is returned to the user.
Using Globus tooling, the user configures their applications to use the tunnel to stream data.
Globus seamlessly routes application connections through the tunnel.

4. Admin Guide

4.1. Deploying Globus Connect Server with Streaming Support

4.1.1. Installation Instructions

Globus Streams support is included with GCS version 5.4.92 or later. Existing endpoints can be upgraded, or a new endpoint can be setup as described here.

4.2. Creating a Stream Gateway and Stream Access Point

Once the installation and deployment of Globus Connect Server is complete, a stream gateway can be created. The stream gateway controls access to a stream access point much like a storage gateway controls access to a mapped collection. When you create a stream gateway, a stream access point with the same name is automatically created.

4.2.1. Stream Gateway Configuration

A stream gateway is created with the globus-connect-server stream-gateway create command. The options include familiar storage-gateway authorization policies such as --domain, --identity-mapping, and --user-allow/--user-deny, as well as the streaming-specific policies --lan-name and --lan-secret-required.

Note

There are three underlying TCP connections in the route when a user forms a connection through a tunnel. Two of the connections are considered "LAN connections", because these are formed between end user applications and their local GCS servers. The leg between the GCS servers is the WAN portion. The WAN connection is always authenticated with TLS in the same manner that Globus file transfers are. The LAN portions can be authenticated with a challenge based protocol. Admins of a stream gateway can enforce this authentication on all of their user’s tunnels by including the --lan-secret-required option. For more information see the Globus Streams Authentication page.

We’ll create a basic stream gateway with the default streaming policies.

Example 1. Basic Stream Gateway

The display name of the stream gateway is used for the stream access point as well. We’ll use a descriptive name like Example.org Streaming Access Point 1, and allow access from all users with an example.org identity.

globus-connect-server stream-gateway create "Example.org Streaming Access Point 1" --domain example.org

The stream gateway and stream access point setup is complete.

If you have access to another streaming access point, continue with the User Guide to create a tunnel between that access point and the one you created. Or, you can repeat these steps to create another streaming endpoint for testing.

5. User Guide

To create a tunnel, you will need access on a stream access point on each side of the desired data stream.

5.1. Create a Secure Tunnel Between Two Stream Access Points

Start by visiting https://app.globus.org/streams. In the top right corner of the page, click the (+) Create Tunnel link. This will take you to https://app.globus.org/streams/create:

The first two fields define each end of the tunnel. The Initiator Access Point is the side of your tunnel that will be making the active connection. The Listening Access Point is the side of your tunnel that will be listening for connections. In each box you can search for your stream access point by name or UUID. The Label field is an optional user friendly name for the tunnel. The Lifetime field is the number of minutes that the tunnel will be available. When the lifetime expires, the tunnel will be automatically stopped. In the stopped state, Information about the tunnel will remain available until you delete the tunnel.

5.2. Manage Your Tunnels

Once created, your tunnel will be displayed in a list at https://app.globus.org/streams:

Here you can monitor the state of your tunnels, stop your tunnels, and delete your tunnels. You can also add the application listener’s contact string by pressing the Play button. This will activate the tunnel. Tunnel activation can also be done with the Globus Tunnel tools as described in the next section.

5.3. Update Your Streaming Application to Use a Tunnel

Globus provides tooling to help adapt your application for use with Globus data streaming. While this may not always be possible, our goal is to require zero changes to existing applications in order to stream data through a Globus tunnel.

Any streaming workflow that makes use of Globus tunnels will consist of two applications: a listener and an initiator. The listener is the side that passively waits for an incoming TCP connection to accept. The initiator is the side that actively forms TCP connections. While the data flow between the applications can be bidirectional, the initiator must connect to the listener; the listener cannot connect to the initiator.

In the examples here, we will use telnet as the initiator application and netcat (nc -l <port>) as our listener. Telnet will initiate a stream to netcat through a Globus tunnel.

5.3.1. Client Tools

globus-streams CLI

The systems where the applications run will require the globus-streams CLI. This is a tool that helps set up your application environment for use with Globus data streaming. It logs the user into the Globus and sets some environment variables. It uses the installed LD_PRELOAD library, which seamlessly intercepts your application’s socket calls and redirects them for use with Globus data streaming.

globus-streams can be installed from the package repository into a python virtual environment using the following steps. This step requires python on your system; adjust the python3 command to python if necessary.

python3 -mvenv ~/streams-cli && \
. ~/streams-cli/bin/activate && \
pip install --extra-index-url https://downloads.globus.org/globus-connect-server/stable/wheels/ globus-streams-cli

PRELOAD Socket Intercept Library

The LD_PRELOAD library is part of the globus-streams-libs package, which is installed automatically on Globus Connect Server nodes. If you need this package on a node without Globus Connect Server, it can be installed directly.

Install the updated streaming packages:

sudo dnf install globus-streams-libs

Install the updated streaming packages:

sudo apt install globus-streams-libs

5.3.2. Listener Application

In order to initialize the listener you will need the ID of the tunnel that you created.

In this example, the nc application will be listening on the IP address 10.0.2.164 on port 8888. The --listener-contact-string option tells Globus this will be the listener application and the contact string where it will be listening.

globus-streams environment initialize --listener-contact-string 10.0.2.164:8888 ${TUNNEL_ID}

Note

You may be prompted to log in twice while the initialize step logs in to both the Globus service and the listener’s Globus Connect Server endpoint. In order to avoid the possibility of multiple authentication flows, you may optionally provide the ID of the GCS endpoint that hosts the Listener Access Point.

Now that the environment is initialized, the listening application can be started. To start the listening application so that it uses the tunnel, run it using the globus-streams-launch.sh helper script in the following way:

/usr/share/globus/streams/globus-streams-launch.sh -p 8888 ${TUNNEL_ID} nc -l 8888

This will run your application in the Globus data streaming environment that has been configured for use with your tunnel. The -p 8888 switch tells Globus that your application intends to listen on port 8888 for any connection request that comes through the tunnel. The remaining arguments are the exact arguments that you use to run your application. In this case it is netcat listening on port 8888.

5.3.3. Initiator Application

The initiator application environment is initialized in a similar way as the listener application.

globus-streams environment initialize ${TUNNEL_ID}

Note

You may be prompted to log in twice while the initialize step logs in to both the Globus service and the initiator’s Globus Connect Server endpoint. In order to avoid the possibility of multiple authentication flows, you may optionally provide the ID of the GCS endpoint that hosts the Listener Access Point.

This time we do not require the listening address because this side is forming active connections out. It will retrieve the contact string at run time via the globus-streams CLI.

Note from the output Your contact string is: globus.0e8a675b-6b84-4220-89e4-a6a7a0d823fb:3664. This is very important to record. Whenever your application wants to form a connection through the tunnel, it must use globus.0e8a675b-6b84-4220-89e4-a6a7a0d823fb as the hostname and 3664 as the port.

We will use those values to connect using the telnet application. Start the telnet application in the following way, again using the globus-streams-launch.sh helper script:

/usr/share/globus/streams/globus-streams-launch.sh ${TUNNEL_ID} telnet globus.0e8a675b-6b84-4220-89e4-a6a7a0d823fb 3664

You should see a successful connection, and any output you type in the initiator application will be shown in the output of the listener application.

6. Additional Reading

Streaming Application Tools: A look under the hood at how our application tools can be used to integrate with your application.
Streaming Connection Authentication: A description of the authentication protocol used between your application and the Globus Connect Server.

7. Support

For questions on streaming, please contact support@globus.org.