Data Types in Catalogue

Overview

A catalogue field’s data type decides what the field can store, how incoming values are merged into the profile, and which operators the field offers in the activation modules such as Audiences and Journeys. Choose it when you add the field: mapping a source field to a catalogue field of a different type can cause those records, or the whole file, to fail ingestion. The Catalogue supports these data types:

String

Integer

Double

Timestamp

List (Array) of String, Integer or Double

Object

List (Array) of Objects

Scalar types

String

You can use the ‘String’ data type in the Zeotap Catalogue to represent text, numbers and alpha-numeric values like Firstname, Lastname as per your use case. Note that the operations available in the activation/downstream modules like Audiences and Journeys depend on the data type selected when adding the catalogue field. In the below image, you can see the illustration of an example of a Catalogue field added as a String, along with the associated operations available for that specific field in Audience.

Integer

You can use the ‘Integer’ data type in the Zeotap Catalogue to represent whole numbers like Age, Customer ID. Ensure that you map any incoming field of integer values to a Catalogue field of the integer type only. Otherwise, you may encounter ingestion failures for those records or for the file. Note that the operations available in the activation/downstream modules like Audiences and Journeys depend on the data type selected when adding the catalogue field. In the below image, you can see the illustration of an example of a Catalogue field added as an Integer, along with the associated operations available for that specific field in Audience.

Double

You can use the ‘Double’ data type in the Zeotap Catalogue to represent decimal/floating numbers such as cart_value, car_price. Ensure that you map any incoming field of type double to a Catalogue field of the double type only. Otherwise, you may encounter ingestion failures for those records or for the file. Note that the operations available in the activation/downstream modules like Audiences and Journeys depend on the data type selected when adding the catalogue field. In the below image, you can see the illustration of an example of a Catalogue field added as a Double, along with the associated operations available for that specific field in Audience.

NoteWhen you opt for European notation of decimal, which is using comma, you need to apply the necessary custom transformation before storing the value. Otherwise, it may lead to ingestion error. Ensure to find and replace custom transformer for such fields at the time of mapping the source. For more information about how to use the custom transformation enricher while mapping the source, refer here, and for a worked example see Transformation of currency data using a custom transformer.

Timestamp

You can use the Timestamp data type in the Zeotap Catalogue to denote a specific point in time, typically recording when an event occurred. In other words, it refers to a precise date and time combination, stored in a format that enables chronological sorting and calculations between different time instances. For Example, 2023-11-30 15:45:00 - This timestamp represents November 30, 2023, at 3:45 PM. Note that the operations available in the activation/downstream modules like Audiences and Journeys depend on the data type selected when adding the catalogue field. In the below image, you can see the illustration of an example of a Catalogue field added as a Timestamp, along with the associated operations available for that specific field in Audience.

By default, the system considers the ingested timestamp as the timestamp information in the record. However, if you wish to recognise a specific field as a Timestamp field, then ensure to map the incoming field to the “event timestamp” field in the Zeotap Catalogue. For example, if you are importing consent data and want the date of consent collection to be stored as the date for that consent in Zeotap, map the date attribute from your source to the event timestamp field. Note that for streaming sources, Zeotap automatically maps this field to the time of the record.

NoteFor information about the different formats that can be selected for Timestamp, refer here.

The Catalogue does not offer a Boolean data type. Model true/false values as a String (for example, true and false, or yes and no) and keep the incoming values consistent, so that audience rules can match them reliably.

Container types

List (Array) of String, Integer or Double

This represents a type of attribute in the Zeotap catalogue which can hold any incoming data that is of the type string, integer or double, or a List of string, integer or double. This incoming data is stored as a list of values against a user profile. For instance, you can map a list of incoming source data of cars or productcode to List type catalogue fields. That is, the values for car is mapped to List of strings and values for the productcode is mapped to List of integers and the result is as shown below. List of String cars:["Ford", "BMW", "Fiat"] List of Integer Productcode:[1102,1105,1456] Note that you can also map a single value string, integer and double and store it as a List in Zeotap. Let’s consider an example to understand this better. Imagine you have incoming data of Cars for the UserID: U1 as shown in the table below.

UserID	Cars
U1	Ford
U1	BMW
U1	Fiat

In this case, if you map the incoming cars data as a List of Strings within the Zeotap Catalogue, then it organises these individual car entries into a unified list format as shown below. cars:["Ford", "BMW", "Fiat"]

Note:Ensure that you consider the following points while selecting the List Data Type:

You can map both List and single-value source fields to the List type catalogue field. Therefore, when a single value source field comes into the system for this Category, then the new data is upserted (update+insert) on the existing data without overwriting it. For example, if the source field is a single value field like “Product Name” and it is mapped to ProdName field of the data type List in the Catalogue, then any new value that comes into the system for the product name will append to the existing one in the format, {p1, p2, p3…}, preserving the previous entries.
You can only map a source field of the data type List to a catalogue field of the data type List. For example, if the source field is of type List of String, then you can map it to only List type attributes in the Catalogue field drop-down of the source mapping screen.
When sharing source data containing List type information, ensure that you use the correct format and avoid null values. Otherwise, it might lead to mapping issues or data being dropped during the data ingestion process. If your source sends several values in one delimited string, split them with a custom transformer first — see Transformation of array using a custom transformer.
The Object, List and List of Object data types are only supported for the categories–User trait, Product, Event and Campaign. Raw personally identifiable information (PII) is not supported for these data types. Hence, refrain from mapping PII information to these types.

List fields are not eligible for Ingestion Filter Criteria, regardless of any data type override applied to them.Ingestion Filter Criteria offers only single-value (primitive) source fields. A source field that arrives as a List stays unavailable there even after you override it to String.

How values are merged and trimmed

For List of String/Integer/Double under the User trait or Product category, values from the older record are appended after values from the newer record (determined by the group _ts). If the configured limit is exceeded, values are trimmed from the end. Since primitive values do not carry individual timestamps, ordering reflects merge order, not event recency, and late-arriving values may be dropped even if they are logically newer. Example (limit = 3):

Existing list at _ts=200: ["B", "A"]
Late-arriving update at _ts=150: ["C", "D"]
Merged:                   ["B", "A", "C", "D"]
After trim:               ["B", "A", "C"]

Here, "D" is dropped even though it belongs to a newer event sequence than "A".

Object

Object represents a type of attribute in Zeotap Catalogue that can hold a JSON object that contains one or more key-value pairs, also called properties. In the following example, productItem is an object in a user profile with properties, ProductName, price and photo as shown below.

Code

productItem
{
  productName: "EXP Portable Hard Drive",
  price: "800.00",
  photo: "external-hard-drive.jpg"
}

The system can store any incoming data field of the type string, integer, double, timestamp, list or object into an object type data field in Zeotap Catalogue. Note that the properties of an object in Zeotap Catalogue can be of type string, double, integer and list of string/integer/double only. Object can hold only one set of properties. Hence, any incoming data will always override the entire object with the incoming set of properties. To understand this better, refer to the ‘scenario 1’ explained below.

Note:Ensure that you consider the following points while mapping the Object data fields:

We recommend you to share the object type data in the following format only:
- Enclose the data within curly braces {}. For example,
Code
"personal_details": { "first_name": "John", "age": 25, "birthdate": "1999-03-15" }
- Separate each key-value pair with a comma. Note that the order of key-value pairs is irrelevant.
- A key-value pair consists of a key and a value, separated by a colon ( : ).
The Object, List and List of Object data types are only supported for the categories–User trait, Product, Event and Campaign. Raw personally identifiable information (PII) is not supported for these data types. Hence, refrain from mapping PII information to these types.

Scenarios of using the ‘Object’ Data Type

Scenario 1: Let’s say, there is an existing productItem object in a user profile with the properties shown above. Whenever there is an incoming record for the productItem object, it replaces the object stored in the user profile as shown below.

Code

productItem
{
  productName: "Luxury Ultra thin Wrist Watch",
  price: "500.00",
  photo: "laptop.jpg"
}

Scenario 2: If you want to store the incoming object properties in separate attributes in the Zeotap catalogue such as ID, user trait, consent, event and more, then you can achieve that by mapping the object properties to the regular Zeotap Catalogue fields. For example:

Code

Campaign
{
  email: "p@gmail.com",
  hasclicked: "true",
  hasclickedtime: "2023-09-15T22:32:59.873Z"
}

email can be mapped to the Email raw attribute of type ID.
hasclicked can be mapped to the hasclicked attribute of type event.
hasclickedtime can be mapped to the timestamp attribute of type event.

To pull a single property out of an incoming list of objects into its own catalogue field, use a custom transformer — see Extraction of object data using a custom transformer. Scenario 3: When two profiles with Object attributes merge due to unification, the most recent Object data among the two profiles will be retained against the profile.

List (Array) of Objects

List of Object represents a type of attribute in the Zeotap catalogue that can hold a list of JSON objects that contains one or more key-value pairs, also called properties. It can be used to store an object or List of object from source data as a List of object type attribute in the Zeotap catalogue. A List of Objects attribute uses one of its properties as the Primary Key, alongside the properties themselves. The Primary Key is what lets the system keep a unique set of objects in the list when it performs an upsert. For example, imagine you are a TV broadcasting company that serves three customers: Viewer A, Viewer B and Viewer C. Each has unique preferences (properties) for a TV package (object) — subscription name, renewal date, engagement scores and package cost. These details are subject to change with time. Here, by configuring the incoming data as List of Objects and choosing the subscription name as the Primary Key, you establish a crucial link between records that allows the system to update and insert the data for the properties renewal date, engagement scores and package cost, whenever new data arrives for the respective fields. Based on the result obtained, you can plan your marketing strategies accordingly. Note that without this designated Primary Key, incoming data would merely accumulate without ensuring the accuracy of your customer records.

Note:Ensure that you consider the following points while selecting the List of Objects Data Type:

For List of Object under the User trait or Product category, FIFO logic is applied after reaching the limit for new value ingestion.
The Object, List and List of Object data types are only supported for the categories–User trait, Product, Event and Campaign. Raw personally identifiable information (PII) is not supported for these data types. Hence, refrain from mapping PII information to these types.

Sample JSON Structure for “List of Objects”

Code

{
  "wishlist": [
    {
      "SKU": "ABC123",
      "Product": "T-Shirt",
      "Price": 19.99,
      "Size": "M",
      "Color": "Red"
    },
    {
      "SKU": "XYZ456",
      "Product": "Jeans",
      "Price": 49.99,
      "Size": "32",
      "Color": "Blue"
    },
    {
      "SKU": "LMN789",
      "Product": "Sneakers",
      "Price": 89.99,
      "Size": "10",
      "Color": "White"
    }
  ]
}

Data-type limits

The following limits apply by default. Reach out to your Zeotap POC for any specific requirements about the system limit.

Data type	Limits
List of String/Integer/Double	Up to 200 values per List attribute, and up to 15 List attributes per organisation.
Object	Up to 50 properties per object. Object and List of Objects attributes share one allowance: 10 in total per organisation.
List of Objects	Up to 200 objects per List of Objects attribute and up to 50 properties per object. Object and List of Objects attributes share one allowance: 10 in total per organisation.

Scalar fields (String, Integer, Double and Timestamp) have no limit of their own — how many values a profile keeps for them follows the field’s category, described in Categories for adding Catalogue fields. Two further constraints apply to the container types:

They are available only for the User trait, Product, Event and Campaign categories, and must not carry Raw PII.
An object’s properties can be String, Double, Integer, or a List of String/Integer/Double — a property cannot itself be an object.

Overriding data types

When the incoming source field and the catalogue field disagree, you do not have to change the catalogue field. While mapping the source you can override the source data type so the incoming values are processed as the catalogue field’s type instead. Records that conflict with the chosen type may be discarded or fail, so review the warning shown in the mapping screen before you save. For the full procedure, see Override the Source Data Type. Note that an override does not make a List-type source field eligible for Ingestion Filter Criteria.

Data Types in Catalogue

Overview

String

Integer

Double

Timestamp

List (Array) of String, Integer or Double

Object

List (Array) of Objects

Scalar types

String

Integer

Double

Timestamp

Container types

List (Array) of String, Integer or Double

How values are merged and trimmed

Object

Scenarios of using the ‘Object’ Data Type

List (Array) of Objects

Sample JSON Structure for “List of Objects”

Data-type limits

Overriding data types

Understand the Basic Fields

Override the Source Data Type

Ingestion Filter Criteria

Use Cases for Custom Transformers

Add a Catalogue Field

Categories for adding Catalogue fields

​Overview

String

Integer

Double

Timestamp

List (Array) of String, Integer or Double

Object

List (Array) of Objects

​Scalar types

​String

​Integer

​Double

​Timestamp

​Container types

​List (Array) of String, Integer or Double

​How values are merged and trimmed

​Object

​Scenarios of using the ‘Object’ Data Type

​List (Array) of Objects

​Sample JSON Structure for “List of Objects”

​Data-type limits

​Overriding data types

​Related Topics

Understand the Basic Fields

Override the Source Data Type

Ingestion Filter Criteria

Use Cases for Custom Transformers

Add a Catalogue Field

Categories for adding Catalogue fields

Overview

Scalar types

String

Integer

Double

Timestamp

Container types

List (Array) of String, Integer or Double

How values are merged and trimmed

Object

Scenarios of using the ‘Object’ Data Type

List (Array) of Objects

Sample JSON Structure for “List of Objects”

Data-type limits

Overriding data types

Related Topics