Creating the Table Using Compressed Event Analytics as the Provider

The main purpose of the Compressed Event Analytics Relational Provider is to split the aggregated events published to DAS into single events and load them to Apache Spark. In order to use the Compressed Event Analytics Relational Provider, the Data Access Layer (DAL) table should have the following two columns.

A meta field named meta_compressed indicating whether the data in the other column is compressed or not.
The flowData column that contains the aggregated events. Data in this column may be compressed (i.e. zipped, and followed by the Base64 encoding).

The aggregated data in the flowData column should be structured as shown below.

{
	'events': [{
		'componentType': 'Proxy Service',
		'componentId': 'TestProxy',
		'children': [1],
		'entryPoint': 'TestProxy'
		. . .
	}, {
		'componentType': 'Sequence',
		'componentId': 'PROXY_INSEQ',
		'children': [2],
		'entryPoint': 'TestProxy'
		. . .
	}],
	'messageFlowId': 123456,
	'payloads': [{
		'payload': '?',
		'events': [{
			'eventIndex': 16,
			'attribute': 'beforePayload'
		. . .
		}]
	}, {
		'payload': '?',
		'events': [{
			'eventIndex': 15,
			'attribute': 'beforePayload'
			. . .
		}]
	}]
}

Use the following query syntax to create a table in the Spark environment (i.e. if it does not already exist) using data from Compressed Events Analytics.

CREATE TEMPORARY TABLE <temp_table_name> 
USING CompressedEventAnalytics 
OPTIONS (tableName <table_name>,  schema "componentType STRING, componentId STRING, children STRING, entryPoint STRING -sp, messageFlowId FACET

Compressed Event Analytics relation provider options

The options that can be used with the Compressed Event Analytics Relation Provider are described in the table below.

Specify the options in key value pairs separated by commas, and enter the values within quotation marks.

Option Description Example

tableName/streamName

The name of the table in the DAL.

tableName "plug_usage"

or

streamName "plug.usage"

schema

The schema of the temporary table. It is required to specify this.

Schema names are column name and column type value pairs with indexing options. These fields should be comma separated.

Schema indexing options are as follows.

-i denotes an indexed column. All indexed columns should be of the numeric type.
-sp denotes an indexed column with a score parameter.

The following fields are special fields in an analytics table:-

_timestamp denotes the timestamp of the record when it is persisted
_tenantId denotes the tenant id of the record

schema "componentType STRING, componentId STRING, children STRING, entryPoint STRING -sp, messageFlowId FACET -i"

mergeSchema This is a boolean flag used for schema merging. If this option is set to true, the given schema is merged with the corresponding table schema in the DAL (if a schema exists). The default value is "false". mergeSchema "false"