GCS Limit Collection HTTP Concurrency

Globus Connect Server utilizes the Apache HTTP web server to provide access to collections. By default, there may not be a limit to the number of concurrent HTTP requests other than the server-wide MaxRequestWorkers configuration directive. This document describes how to set a maximum number of concurrent HTTP requests for collections on a GCS endpoint by modifying the Apache HTTPD configuration.

There are several Apache modules and directives that can be used to limit the number of concurrent collection HTTP requests. In this guide, we will focus on using the mod_qos module, which can enforce several Quality of Service (QoS) rules, including concurrency limits. The mod_qos module is not included in the default Apache HTTP distribution, so it must be installed separately.

See https://mod-qos.sourceforge.net/ for more information about the mod_qos module and its capabilities. See the documentation for your specific Linux distribution for instructions on how to install mod_qos for your Apache HTTP installation.

Warning
In addition to hosting HTTP access to collections, GCS uses Apache HTTP to serve the GCS Manager API and GridFTP control channel access for Transfer tasks. Setting access concurrency limits too low could result in users being unable to access those services.

1. Limiting Concurrency with Apache mod_qos

Below are some example mod_qos directives that can be used to limit the number of requests for collection HTTP access.

Note
Collection HTTP virtual host configurations are name-based virtual hosts so only request-level mod_qos directives (QS_Loc*, QS_Deny*, etc) can be used in collection host configurations. Connection-level controls (QS_Srv*) must be used in the main Apache configuration file (e.g. /etc/httpd/conf/httpd.conf or /etc/apache2/apache2.conf) and will apply to all virtual hosts on the server.
QS_LocRequestLimit <location> <requests>

The QS_LocRequestLimit directive limits the number of concurrent requests for a specific location within the collection virtual host. <location> is the URL path to apply the limit to (e.g. / for the entire virtual host) and <requests> is the maximum number of concurrent requests allowed for that location. For example, to limit the number of concurrent requests to 10 for the entire collection virtual host, you could use the following directive:

# Limit concurrent connections for the virtual host
QS_LocRequestLimit / 10
QS_LocRequestPerSecLimit <location> <requests>

QS_LocRequestPerSecLimit can be used to limit the number of requests per second for a specific location, which can help mitigate short bursts of high concurrency. For example, to limit the number of requests to 15 per second for the entire collection virtual host, you could use the following directive:

# Limit connections per second for the virtual host
QS_LocRequestPerSecLimit / 15
QS_LocKBytesPerSecLimit <location> <kbytes>

The QS_LocKBytesPerSecLimit directive can be used to limit the bandwidth per connection for a specific location, which can help mitigate resource exhaustion caused by a small number of very large downloads. For example, to limit the download bandwidth to 500 KB/s for the entire collection virtual host, you could use the following directive:

# Limit download bandwidth per connection for the virtual host
QS_LocKBytesPerSecLimit / 2
# Warning: This may cause an increase in concurrent download connections

See https://mod-qos.sourceforge.net/ for more information about these and other mod_qos directives that can be used to provide QoS rules for collection HTTP access.

2. Configuration File Locations

As of GCS 5.4.93, administrators may Extend Collection Virtual Host Configurations to customize the Apache HTTP configuration for collections on a GCS endpoint. Using this feature, administrators can place directives in the appropriate configuration file to set a maximum concurrency for a collection.

Note
It is the administrator’s responsibility to place these vhost configurations on all nodes in the endpoint. GCS does not manage, synchronize, or replicate these manual configuration files between nodes in a multi-node deployment.

In this example, we will set a maximum concurrent request limit of 20 for the collection with the ID 08a2788f-0b5c-4064-a2f0-f4d2b1f77e91. To do this, we will create a configuration file at /etc/globus/httpd/08a2788f-0b5c-4064-a2f0-f4d2b1f77e91.conf with the following content:

QS_LocRequestLimit / 20

Restart the Apache HTTP service for the changes to take effect.

Ubuntu, Debian, SUSE
$ sudo systemctl restart apache2.service
Fedora, CentOS, RedHat
$ sudo systemctl restart httpd.service

3. Verifying Custom QoS Settings

Verifying that your custom QoS settings are working as expected can be challenging, as it requires generating a specific number of concurrent requests to the collection. One way to do this is to use a load testing tool such as Apache JMeter to simulate multiple concurrent users accessing the collection. You can configure the tool to send HTTP requests to the collection’s URL and monitor the responses to see if requests are being rejected once the concurrency limit is reached.

For requests requiring authenticated Collection HTTP access, see HTTPS Access to Collections for generating necessary access tokens.

See Verifying Changes in the Extend Collection Virtual Host Configurations guide for more details on how to verify correct functionality after applying custom Apache configurations.