This documentation is for WSO2 Enterprise Integrator version 6.0.0 . View documentation for the latest release.

All docs This doc

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Data purging is an option to remove historical data in ESB Analytics. This is important since you cannot delete tables or table data in ESB Analytics. By purging data, you can achieve high performance in data analysis without removing analyzed summary data. You can purge only stream data that are generated by the ESB profile that are contained in the following tables:

...

ESB Analytics stores data and performs various analysis operations on them according to defned analytic queries. Thereby, as the volume of the data stored grows over time, the analysis and summarization jobs will also consume more time. Hence, purging data allows you to increase the performance of ESB Analytics by reducing the time taken to execute the analytics scripts as well as optimising the disk usage.

Warning

Decide on a retention period based on the number of days you need to retain data for analysis and purge data that are older than that in the below tables.

  • ORG_WSO2_ESB_ANALYTICS_STREAM_EVENT

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECOND

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECONDALL

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTE

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTEALL

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERHOUR

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMONTH

  • ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERDAY

  • ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERSECOND

  • ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMINUTE

  • ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERHOUR

  • ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERDAY

  • ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMONTH

...

Table of Contents
maxLevel3
minLevel3

Using the

...

per-table method

 Follow the steps below to schedule data purging via the Management Console, follow the steps below:per-table method.

  1. Log in to the Management Console of the Analytics profile.
  2. In the Main tab, click Data Explorer to open the Data Explorer page.
  3. Select the required table in the Table Name field, and click Schedule Data Purging as shown below. 

    Note

    The Schedule Data Purging option is displayed only for users of whom the assigned role has the Delete permission under Record enabled.

    Image Modified
    This opens the Schedule Data Purging dialog box.

  4. In the Schedule Data Purging dialog box, set the time and days within which you want to purge data as shown below, and click Save.

    Note

    Do not purge data that is less than 2 days old as it can result in a data loss.

    The fields of the above screen are described below.

    FieldDescription
    Enable Data PurgingWhether you want to enable data purging or not.
    Schedule Time (Either cron string or HH:MM)Enter the time on which you want to purge data via cron expression or by defining the time in the following format: HH:MM. For example, the following cron expression will configure the data purging job to run at  12:00 PM (noon) every day 0 0 12 * * ?  For more information on cron expressions, go to Oracle Documentation.
    Purge Record Older Than (Days)Define the value as to keep data of only the last 'n' no of days back in the selected table. For example, if you give 1 as the value, the system will purge all data stored before yesterday.
  5. Repeat these steps for all the required tables to schedule them to be cleared regularly.

Removing scheduled data purging operations

Follow the steps below to remove a data purging operation that you already scheduled.

  1. Log in to the Management Console as an admin user, if you are not already logged in.
  2. Click Main, and then click Data Explorer.
  3. Select the required table in the Table Name field, and then click Schedule Data Purging to open the Schedule Data Purging dialog box.
  4. Clear the Enable Data Purging check box as shown below.
    disable data purgingImage Added
  5. Click Save, and close the dialog box.

Using the global method

Info

This action affects all tenants.

  1. Open the <EI_HOME>/wso2/analytics/conf/analytics/analytics-config.xml file.
  2. Change the content under the <analytics-data-purging> property as shown below:

    Note

    Change the value of the <data-retention-days> property as per your requirement. However, do not purge data that is less than 2 days old as it can result in a data loss.

    Code Block
    languagexml
    <analytics-data-purging>
        <purging-enable>true</purging-enable>
        <cron-expression>0 0 12 * * ?</cron-expression>
        <purge-include-table-patterns>
          <table>
             <table>ORG_WSO2_ESB_ANALYTICS_STREAM_EVENT</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECOND</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERSECONDALL</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTE</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMINUTEALL</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERHOUR</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERMONTH</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_STATPERDAY</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERSECOND</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMINUTE</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERHOUR</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERDAY</table>
        	 <table>ORG_WSO2_ESB_ANALYTICS_STREAM_MEDIATORSTATPERMONTH</table>
          </table>
          </purge-include-table-patterns>
        <data-retention-days>2</data-retention-days>
      </analytics-data-purging>

    The properties of the above configuration file are shown below.

    PropertyDescription
    <purging-enable>
    Change the value to true if you want to enable data purging.
    <cron-expression>
    The cron expression to define how you want to schedule the data purging operation. For example, the following cron expression will configure the archive job to run at  12:00 PM (noon) every day 0 0 12 * * ?  For more information on cron expressions, go to Oracle Documentation.
    <purge-include-table-patterns>
    Specify the tables of which you want to purge data. By default, it is configured to perform data purging on all tables as follows:

    <table>.*</table> However, you can specify the required tables by defining a regular expression or a table name within the <table> property. Define one tag per each regular expression if you want to specify multiple tables.

    <data-retention-days>

    Define the value as to keep data of only the last 'n' no of days back in the selected table.   For example, the default value 365 will purge all data stored before 1 year.

    Info

    You can purge all records by setting a minus value (e.g. -1) for <data-retention-days>.

  3. Save your changes.