This section describes the additional configurations that can be done to data collected by DAS before they are processed.
Setting a custom timestamp value for incoming events
When an event is received by WSO2 DAS, the timestamp of that event is set as a parameter value. This value often needs to be overridden in many situations including the following examples.
- Incremental data processing
- Timestamp based Analytics
In order to override the timestamp value assigned at the time an event is received, the client that publishes events to WSO2 DAS should set an attribute named
_timestamp of the
Long type as a payload attribute in each event.
Querying for the timestamp in a Spark script
You can query for events by the timestamp in an Analytics script. In order to do this, you need to add a
_timestamp long option in the table schema.
e.g., If there is a table named
TestTable1 in WSO2 DAS with the name
STRING, b INT schema, the timestamps of the events in the table can be queried as follows.
Additionally, you can use the
_timestamp option to insert a custom value for the timestamp parameter of an event.
e.g., If you need to insert a set of values from the
table1 table to another table named
table2, you can write a query as follows.
The following query inserts the results of the
SELECT a1, _timestamp FROM table1 query into the
You can override the behavior of the
_timestamp field to use the current timestamp of the event, by setting -1 as shown below.