Scroll Query

Method	POST
URL	/v1/index/<index_id>/scroll
Authentication required?	Only for non-public data
Required Roles	None
Request Body	a GScrollRequest document
Response Body	a GScrollResult document

Authentication & Authorization

Tokens for this call must have one of these scopes.

urn:globus:auth:scope:search.api.globus.org:all
urn:globus:auth:scope:search.api.globus.org:search

Examples

Scroll via curl

To run a scroll query, we send it via a POST to the API, e.g.

curl -XPOST \
    -H 'Content-Type: application/json' \
    'https://search.api.globus.org/v1/index/4de0e89e-a395-11e7-bc54-8c705ad34f60/scroll' \
    --data '
{
  "q": "a scroll request with filtering",
  "filters": [
    {
      "type": "range",
      "field_name": "path.to.date",
      "values": [
        {
          "from": "*",
          "to": "2014-11-07"
        }
      ]
    }
  ]
}'

Request Schemas

GScrollRequest

This is the main document type for encoding a scrolling query.

Field Name	Type	Description
q	String	User-supplied query, conforming to the query syntax. Required if there are no filters.
advanced	Boolean	Defaults to False When true, interpret q with the advanced query syntax
limit	Integer	Optional. Limit the results given to limit many items. Defaults to 10
bypass_visible_to	Boolean	Defaults to False Allowed for Index Admins only. When true, visible_to restrictions will be ignored for this search query.
filter_principal_sets	List of Strings	Optional. A list of `principal_set` names. The caller’s identity set will be matched against any `principal_sets` assigned to entry documents, and filtered to matches for any of these strings. If this parameter is provided, at least one match must be present.
filters	Array	Optional. An array of GFilter Documents. Filters to apply to the search
marker	String	Optional. An opaque token from a previous scroll result document, used to request the nex page of results.

Examples

{
  "q": "the quick brown fox jumps"
}

GFilter

A GFilter document is one of several document types which encode a filter. The type of filter is identified by the type field. See the table below for the various filter types.

Type	Schema
`match_all`	GFilterMatch
`match_any`	GFilterMatch
`range`	GFilterRange
`geo_bounding_box`	GFilterGeoBoundingBox
`geo_shape`	GFilterGeoBoundingBox
`exists`	GFilterExists
`like`	GFilterLike
`not`	GFilterNot
`and`	GFilterAnd
`or`	GFilterOr

Note

All filters support a post_filter field. Note that post_filter is only valid on filters when they are in the top level filters array of a request.

When filters are nested under and, or, or not filters, post_filter is no longer valid.

GFilterMatch

A matching filter for finding results which match some set of text terms.

"match_any" and "match_all" refer to the different possible behaviors of the filter values. As their names imply, if "match_any" is specified, the filter will match results for which any of filter values match, while "match_all" requires that all of the values match on every result.

Field Name	Type	Description
type	String	One of `{"match_any", "match_all"}`
field_name	String	The field to which the filter refers. Any dots (".") must be escaped with a preceding backslash ("\") character.
values	Array of Strings or Booleans	The values to evaluate against the field_name. If the field is a boolean field, this must be an array of booleans only. For string fields, it may be a mixture of strings or booleans.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True for `match_any` and False for `match_all`.

Note

"match_any" and "match_all" are the same when there’s only one value as far as filtering is concerned, but they may have different impact on the way that facets are interpreted.

Examples

{
  "type": "match_any",
  "field_name": "globus_metadata.resource_type",
  "values": [
    "Globus Endpoint"
  ]
}

{
  "type": "match_all",
  "field_name": "globus_metadata.keywords",
  "values": [
    "hpc",
    "internet2",
    "uchicago"
  ]
}

{
  "type": "match_any",
  "field_name": "globus_metadata.snorkels",
  "values": [
    "few",
    "many",
    true
  ]
}

Note

This filter is only valid if globus_metadata.snorkels is a string field because string fields can contain boolean values.

If it is a boolean field (which cannot contain string values), the query will fail with an error regarding the improper mapping of "few" and "many" onto a boolean field.

GFilterRange

A range filter for finding results which have numeric or date values within a specified range.

Field Name	Type	Description
type	String	Must have the value `"range"`
field_name	String	The field to which the filter refers. Any dots (".") must be escaped with a preceding backslash ("\") character.
values	Array of Objects	The values to evaluate against the `field_name`. Each object has the fields `from` and `to` OR each object has exactly one of `gte` or `gt` and exactly one of `lt` or `lte`.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

Note

values.from and values.to may be the special string "*" indicating that the range is unbounded on this end. An example is given below.

Examples

{
  "type": "range",
  "field_name": "path.to.date",
  "values": [
    {
      "from": "1970-01-01",
      "to": "2015-01-01"
    }
  ]
}

{
  "type": "range",
  "field_name": "cardinality_of_foobar",
  "values": [
    {
      "from": "10",
      "to": "50"
    }
  ]
}

This example filter has multiple clauses. The combination is implicitly joined with "or" semantics. This means that we allow values from 0 to 5, and greater than or equal to 10.

{
  "type": "range",
  "field_name": "cardinality_of_foobar",
  "values": [
    {
      "from": "0",
      "to": "5"
    },
    {
      "from": "10",
      "to": "*"
    }
  ]
}

{
  "type": "range",
  "field_name": "path.to.date",
  "values": [
    {
      "from": "1970-01-01",
      "to": "2015-01-01"
    },
    {
      "gte": "2015-01-01",
      "lte": "2016-01-01"
    },
    {
      "gt": "2016-01-15",
      "lt": "*"
    }
  ]
}

GFilterGeoBoundingBox

A bounding box filter for finding geo_shape and geo_point values which intersect with a specified bounding box.

Field Name	Type	Description
type	String	Must have the value `"geo_bounding_box"`
field_name	String	The field to which the filter refers. Any dots (".") must be escaped with a preceding backslash ("\") character.
top_left	Object	An object describing a coordinate pair. It must contain the keys `lat` and `lon`, each of which must have a numeric value.
bottom_right	Object	An object describing a coordinate pair. It must contain the keys `lat` and `lon`, each of which must have a numeric value.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

Note

top_left is required to be northwest of bottom_right.

Examples

{
  "type": "geo_bounding_box",
  "field_name": "country.center",
  "top_left": {
    "lat": 49.1,
    "lon": -124.9
  },
  "bottom_right": {
    "lat": 24.9,
    "lon": -67.1
  }
}

GFilterGeoShape

A geo filter for finding geo_shape and geo_point values which intersect with or are contained within a given shape.

Field Name	Type	Description
type	String	Must have the value `"geo_shape"`
field_name	String	The field to which the filter refers. Any dots (".") must be escaped with a preceding backslash ("\") character.
shape	Object	A GeoJSON formatted Geometry. See note below on supported geometries.
relation	String	The shape relationship to test. One of `{"intersects", "within"}`. Defaults to `"intersects"`.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

Supported Geometries

Only two-dimensional GeoJSON data are allowed in geo_shape filters. That means that coordinates should be encoded as JSON arrays of length 2.

Globus Search only supports filters using GeoJSON Polygons. Furthermore, Polygons are restricted to simple polygons, consisting of only one coordinate ring. This means that polygons with internal cut-outs are forbidden.

Examples

{
  "type": "geo_shape",
  "field_name": "city.boundary",
  "shape": {
    "type": "Polygon",
    "coordinates": [
      [
        [
          -5.8,
          51.5
        ],
        [
          10.0,
          51.5
        ],
        [
          10.0,
          41.0
        ],
        [
          -5.8,
          41.0
        ],
        [
          -5.8,
          51.5
        ]
      ]
    ]
  }
}

GFilterExists

An "existence" filter which checks if a field is present in a document with a non-null value. Note that a field being present but with a value of null is considered the same, under exists filters, as the field being absent from the document.

Field Name Type Description

Field Name	Type	Description
type	String	Must have the value `"exists"`
field_name	String	The field to which the filter refers. Any dots (".") must be escaped with a preceding backslash ("\") character.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

type

String

Must have the value "exists"

field_name

String

The field to which the filter refers. Any dots (".") must be escaped with a preceding backslash ("\") character.

post_filter

Boolean

Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results.

Defaults to True.

Examples

The following filter finds documents where the field foo exists:

{
  "type": "exists",
  "field_name": "foo"
}

GFilterLike

A "like" filter which checks if a field matches a "like-expression". Like expressions are matching strings containing the wildcard characters:

* matches any number of characters
? matches any one character

Field Name	Type	Description
type	String	Must have the value `"like"`.
field_name	String	The field to which the filter refers. It must be a text field.
value	String	The filter expression to apply as a match.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

Examples

The following filter finds documents where the field filename contains a string ending in .csv.

{
  "type": "like",
  "field_name": "filename",
  "value": "*.csv"
}

Note that this does not technically guarantee that the filename ends with .csv. For example, it is possible for the filter to match on a value like "filename": "foo.csv bar".

GFilterNot

A "not" filter for inverting any other valid filter.

Field Name Type Description

Field Name	Type	Description
type	String	Must have the value `"not"`
filter	Object	Any valid GFilter object.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

type

String

Must have the value "not"

filter

Object

Any valid GFilter object.

post_filter

Boolean

Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results.

Defaults to True.

Examples

The following filter finds documents where the field foo does not exist:

{
  "type": "not",
  "filter": {
    "type": "exists",
    "field_name": "foo"
  }
}

GFilterAnd

An "and" filter for joining any other valid filters. In order for an "and" filter to match on documents, all of the filters it contains must match.

Field Name Type Description

Field Name	Type	Description
type	String	Must have the value `"and"`
filters	Array of Object	An array of GFilter objects.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

type

String

Must have the value "and"

filters

Array of Object

An array of GFilter objects.

post_filter

Boolean

Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results.

Defaults to True.

Examples

The following filter finds documents where the field title exists and keywords contains hpc:

{
  "type": "and",
  "filter": [
    {
      "type": "exists",
      "field_name": "title"
    },
    {
      "type": "match_any",
      "field_name": "keywords",
      "values": [
        "hpc"
      ]
    }
  ]
}

GFilterOr

An "or" filter for joining any other valid filters. In order for an "or" filter to match on documents, at least one of the filters it contains must match.

Field Name Type Description

Field Name	Type	Description
type	String	Must have the value `"or"`
filters	Array of Object	An array of GFilter objects.
post_filter	Boolean	Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results. Defaults to True.

type

String

Must have the value "or"

filters

Array of Object

An array of GFilter objects.

post_filter

Boolean

Control whether or not this filter should be applied before or after facets are calculated. If True, the filter will not impact facet results, but will filter the query results.

Defaults to True.

Examples

The following filter finds documents where either the author.institution or the dataset.institution is uchicago.edu. One or both can be a match:

{
  "type": "or",
  "filter": [
    {
      "type": "match_any",
      "field_name": "author.institution",
      "values": [
        "uchicago.edu"
      ]
    },
    {
      "type": "match_any",
      "field_name": "dataset.institution",
      "values": [
        "uchicago.edu"
      ]
    }
  ]
}

Response Schemas

GScrollResult

This is the document type for all results from scrolling queries.

Field Name	Type	Description
gmeta	Array	An array of GMetaResult documents, the main body of the result
count	Integer	The number of results returned; i.e. the size of the gmeta array. May be 0
total	Integer	The total number of matches for the search. May be 0 if no matches are found
has_next_page	Boolean	True if there’s another page of results available, False otherwise
marker	String	An opaque marker used to request the next page of results

Examples

{
  "count": 1,
  "gmeta": [
    {
      "@datatype": "GMetaResult",
      "@version": "2019-08-27",
      "entries": [
        {
          "content": {
            "cuisine": [
              "mexican"
            ],
            "handle": "salsa-verde",
            "ingredients": [
              {
                "amount": {
                  "number": 10
                },
                "default": "tomatillo",
                "preparation": "simmer 20 minutes",
                "type": "fruit"
              },
              {
                "amount": {
                  "number": 2
                },
                "default": "serrano pepper",
                "preparation": "seeded",
                "substitutes": [
                  "jalapeno",
                  "thai bird chili"
                ],
                "type": "fruit"
              },
              {
                "amount": {
                  "number": 2,
                  "unit": "clove"
                },
                "default": "garlic",
                "type": "vegetable"
              },
              {
                "amount": {
                  "number": 0.5
                },
                "default": "yellow onion",
                "type": "vegetable"
              },
              {
                "amount": {
                  "number": 2,
                  "unit": "tsp"
                },
                "default": "salt",
                "type": "spice"
              },
              {
                "amount": {
                  "number": 2,
                  "unit": "tbsp"
                },
                "default": "coriander",
                "preparation": "ground",
                "substitutes": [
                  "cumin"
                ],
                "type": "spice"
              }
            ],
            "keywords": [
              "salsa",
              "tomatillo",
              "coriander",
              "serrano pepper"
            ],
            "origin": {
              "author": "Diana Kennedy",
              "title": "Regional Mexican Cooking",
              "type": "book"
            }
          },
          "entry_id": null
        }
      ],
      "subject": "https://en.wikipedia.org/wiki/Salsa_verde"
    }
  ],
  "total": 1,
  "has_next_page": true,
  "marker": "3d34900e3e4211ebb0a806b2af333354"
}

GMetaResult

These are components in a search result.

A GMetaResult is a structure similar to a GMetaEntry from the Ingest API, with the following significant differences:

visibility information is not exposed; i.e. visible_to is not included
metadata for any subject may be an aggregate of multiple documents with different visibility rules or sources. Thus, the result is always returned as an array in which each element represents data provided by a different source or with different visibility

GMetaResult

Field Name Type Description

Field Name	Type	Description
subject	String	the resource described by this metadata, often a URI
entries	Array	An array of objects containing the data pertaining to the subject. Each object has the fields `content`, `entry_id`, and `matched_principal_sets`. The `content` is an object with the entry data which was sent to Search, and the `entry_id` is its ID. If there are any assigned `principal_sets` for the entry which match the current caller, they will be returned as an array of strings in `matched_principal_sets`.

subject

String

the resource described by this metadata, often a URI

entries

Array

An array of objects containing the data pertaining to the subject.

Each object has the fields content, entry_id, and matched_principal_sets. The content is an object with the entry data which was sent to Search, and the entry_id is its ID. If there are any assigned principal_sets for the entry which match the current caller, they will be returned as an array of strings in matched_principal_sets.

{
  "entries": [
    {
      "content": {
        "alpha": {
          "beta": "gamma"
        }
      },
      "matched_principal_sets": [],
      "entry_id": null
    },
    {
      "content": {
        "alpha": {
          "beta": "delta"
        }
      },
      "matched_principal_sets": [],
      "entry_id": "with_delta"
    }
  ],
  "subject": "http://example.com"
}