Globus Streams Application Tools

In order to make it easy for applications to seamlessly integrate with Globus Streams, a client side package is provided. This package includes a system level library that we call the PRELOAD library. It dynamically loads into your application’s linux process space and intercepts some standard socket library calls. This allows for the library to do the needed authentication protocol and contact point lookups.

Additionally there is a CLI that works in conjunction with these client libraries. Installation of both of these tools is discussed here. In this document we will discuss how these tools work together to provide a seamless integration with your application.

1. Streams CLI

The globus-streams CLI is a simple command line utility that provides 2 major functions:

  1. Associating Tunnel IDs which application contact strings.

  2. Fetching LAN secrets from GCS for use with authentication.

1.1. Tunnel IDs and Contact Strings

When a user creates a tunnel they are given a tunnel ID as the point of reference for forming connections through a tunnel. Let’s take a look at how that tunnel ID can be used to find the underlying network endpoints required for the network overlay that makes up a Globus Stream.

A tunnel ID is simply a UUID that is the key to metadata which is stored inside of Globus Transfer services. For the connections that make up a Globus Stream to be formed, the TCP contact strings must be associated with the UUID, and users applications must be able to look up these contact strings based on the tunnel ID.

data streaming points 001

The above diagram shows the network overlay of a Globus Stream. The three different colored arrows represent the three required TCP connections. Each arrow connects to a listener. The green arrow (1) connects from the user’s application to a listener on its local GCS sever. The red arrow (3) is the connection that is formed from the listening side’s GCS server and to a TCP listener that the user’s application is running.

Note the blue connection: this is between the two GCS servers. We will not be discussing this part of the Stream here. Instead we will focus on the application facing connections which are the green and the red ones.

1.1.1. Listening Application

Let’s start by looking at the connection to the listening side of the application, (3)which is represented by the red connection. The application passively listens on a TCP port awaiting active connections. In order for the listening side GCS server to find the addressable hostname and port,it must be associated with the tunnel ID. The following command allows a user to do this:

globus-streams environment initialize --listener-contact-string 192.168.0.10:8888 $TUNNEL_ID

This command logs into Globus Transfer and tells it where the application associated with that tunnel ID is listening. In this way, when a Globus Stream connection request comes into the listening GCS server, it knows how to form the final connection in the overlay (the red arrow).

Note
This command is only run once. If the application needs to listen on a new location a new Globus Tunnel must be created.

1.1.2. Initiating Application

We saw that the listening end of the application must tell GCS how to contact it. The initiating application has the opposite problem, it needs to figure out how to contact GCS in order to form the first leg (green arrow) of the Globus Stream. Again the globus-streams CLI can be used for this:

globus-streams environment initialize $TUNNEL_ID

While this is a similar command to the above, what it does is a little more subtle so we will discuss it in more detail. When this command is run it contacts Globus Transfer and requests the ip:port of the green listener. If it is available that location is written to a secure file under $HOME/.globus/streams/<tunnel id>.conf.

However, it might not be available yet. Unlike the listening side, here the application is not telling Globus Transfer it is ready, it is instead asking Globus if it is listening and ready. The answer might be "not ready".

To further complicate things, the green contact string may change over time. As discussed elsewhere, a Globus Tunnel can "restart" if a network error causes it to fail. When a restart occurs the green contact string will change and the user’s application will need to fetch the new location.

To help with this there is the following command:

globus-streams environment contact-lookup $TUNNEL_ID

This command will first check to see if the file $HOME/.globus/streams/<tunnel id>.conf exists on the system. In that file is the green contact string, the time it was last fetched, and timestamp of when it should be looked up again. If the contact string has not expired, the program will simply print out the location. If it has expired it will fetch a new value, write it to the file, and print out that new value.

1.2. LAN Secret Fetching

Above we described how the globus-streams environment initialize command is used to associate the tunnel ID with the underlying listener addresses. The command does another important thing, it requests that a LAN secret be associated with the tunnel.

When run on either the listener or the initiator, a call to GCS is made that creates a random string, associates it with the tunnel in the GCS server, and returns it to the calling CLI. The CLI then writes that value to the $HOME/.globus/streams/<tunnel id>.conf file. This value is good for the entire lifetime of the tunnel. It is used only for that tunnel.

You can see that the $HOME/.globus/streams/<tunnel id>.conf has all the needed information for forming application tunnels, and that the globus-streams environment toolset can help manage that information. However, using these tools without further automation is not going to provide a seamless application integration. In the next sections we will discuss further automation around these tools.

2. PRELOAD Library

The Globus Streams PRELOAD library is a lightweight dynamically loadable library that can run inside the process space of your application to automatically detect connection attempts that should be redirected to Globus Streams locations.

data streaming preload

Here we see a diagram of how the preload library works. When a user runs their program the environment variable LD_PRELOAD is set to libglobus_streams_client.so.0. Now when their program runs the Globus library will be injected into the process space and it will intercept some calls to the system’s socket library. When those calls are intercepted, the library will verify if they are destined for Globus Stream or not. If not, the calls are simply passed directly through to the socket library. If the connection calls are associated with a Globus Stream then the PRELOAD library needs to determine the underlying contact points, and perform the connection hand shake.

Note
The calls to read and write are never intercepted. This is important because it shows that once the connection is established, no further processing, or other overhead, is introduced by the Globus system in the user’s process space. The only system calls that we need to intercept are the ones that relate to connection establishment.

2.1. Listening Side

On the listening side, the PRELOAD library needs to know what listener ports are intended to be routed through a Globus Stream connection, and what specific tunnel that port is associated with. To do that the following environment variable must be set prior to starting the application:

GLOBUS_STREAMS_INTERCEPT_PORT_<port number>=$TUNNEL_ID

When a call to bind() is intercepted, PRELOAD looks at the listening port being requested. If that port has a matching GLOBUS_STREAMS_INTERCEPT_PORT_<port> environment variable then it knows this is a Globus Streams port and the tunnel ID is the value of that environment variable. Once a bound listener is identified as a Globus Stream listener, the PRELOAD library will intercept any calls to accept() and perform the needed authentication protocol.

2.2. Initiator Side

The connecting side has a little bit more work to do. In addition to recognizing when a connection requires the handshake protocol needs, it also needs to find the real location to connect to. Let’s start by backing up to the CLI initialize command discussed above.

Note the example output of the initiator side’s initialize command:

$ globus-streams environment initialize 0a866857-141d-432f-a4b9-88dbbeb09cbb
Initializing the environment for tunnel: 0a866857-141d-432f-a4b9-88dbbeb09cbb
The environment is initialized for use with tunnel 0a866857-141d-432f-a4b9-88dbbeb09cbb
Your application key file base directory is /home/ubuntu/.globus/streams/
Your contact string is: globus.0a866857-141d-432f-a4b9-88dbbeb09cbb:3425

The last line shows the contact string to use when trying to form a connection through a tunnel. The hostname is globus.0a866857-141d-432f-a4b9-88dbbeb09cbb and the port is 3425. These are dummy values formatted so that the PRELOAD library can identify them as Globus Stream connections.

When a call to gethostbyname, or getaddrinfo, is intercepted by PRELOAD it checks to see if the hostname being looked up string starts with globus.. If so it knows that this connection is destined for a Globus Stream and thus it must look up the real endpoint contact string.

To do this it execs out a call globus-streams environment contact-lookup and associates the IP:PORT of the green listener with the lookup value that it returns to the application. Later, when a connect() call is intercepted by the PRELOAD library, if the host information matches and the port matches, then the PROLOAD library will perform the connection handshake protocol.