Types, Type Detection, and Schemas
Globus Search uses dynamic schemas for your data in an index.
Whenever a new field is seen in a document, it is added to the schema for the index. However, that does not mean that Globus Search does not have any schemas for data at all. Rather, it maps input data to types, and then, once those types have been detected, it enforces them strictly.
1. Type Detection on Ingest
When a document is added to an index via the Ingest API, any fields which do not exist in the index have their types detected and added to the index.
For example, if "foo": 1 is added to the index, then "foo" is identified as
being an integer. Attempting to store the data "foo": "bar" in the index at a
later date — even in a separate document — will be rejected because "bar"
is not numeric data.
Type detection understands the following types and maps them as follows:
| Input Type | Example Values | Detected Type |
|---|---|---|
Boolean |
|
Boolean |
Float |
|
Float |
Integer |
|
Integer |
Date String |
|
Date |
String |
|
String |
1.1. Automatic Widening Conversions
In the above example, "foo": 1 causes a later entry of "foo": "bar" to be
rejected. However, if the data is indexed in the opposite order, the results
will be different.
First, "foo": "bar" would be detected as a string. However, "foo": 1 can
be put through a widening conversion to "foo": "1" and stored, and Search
will do this automatically.
Similarly, integers can be converted to floats and booleans to strings.
Search will not automatically perform any lossy conversions, such as truncating floats to integers.
2. Date Formatted Strings
Globus Search understands the following string formats as dates:
yyyy-MM-dd yyyy-MM yyyy/MM/dd yyyy-MM-dd HH:mm:ss yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ yyyy/MM/dd HH:mm:ss Z yyyy/MM/dd Z
So, for example, each of the following would parse as a valid date:
2020-07-01 1970-01-01 00:00:00 1986-08
3. Changing the Type of Indexed Data
Changing the type of a field requires a full reindex of all data in the index. It therefore requires intervention of the Globus Search team.
If you have indexed data in Globus Search and the type is incorrect, please
file a support ticket with support@globus.org.