Analytics for Microgateway

This documentation is for the API Microgateway component that is coupled with WSO2 API Manager 2.5.0.
View the documentation for the latest release - WSO2 API Microgateway 3.1.0.

The following sections explain how to configure the WSO2 API-M Analytics server to work with the Microgateway.

Architecture

The current architecture depicting the connection between the API Microgateway and the Analytics server is shown below.

How Analytics works with the Microgateway

The Microgateway receives the requests from the API. To process the requests, there are two main tasks that run in the Microgateway.

When an API is invoked through the Microgateway, the subsequent events related to the invocations are populated and written in a file. To avoid the files attaining a large size, the following tasks are executed at particular time intervals.

File rotating task

This task creates large files by rotating them at particular time intervals. The size of the files depends on the TPS (Transactions Per Second) of the environment in which the microgateway is running. Once the file is rotated, it is compressed into a zipped file. The rotatingPeriod can be configured as shown in the following section under step 2.

File uploading task

This task uploads the zipped file created by the previous task, to the Analytics server/web application. This is persisted in the WSO2AM_STATS_DB database in the Analytics server node. A task running in the Analytics server processes the persisted data and sends it to the event stream. The time interval in which the zipped files are uploaded can be configured by setting the uploadingTimeSpanInMillis parameter.

Once the compressed file is uploaded to the Microgateway web application endpoint in the Analytics server, the Analytics server persists it in the WSO2AM_STATS_DB. After successfully completing the upload and persistence is achieved, the corresponding files are deleted from the Microgateway.

Configuring Analytics for the Microgateway

The following sections describe how to configure the WSO2 API-M Analytics Server for Microgateway.

Step 1 - Configuring the API-M Analytics Server

Before you begin...

Configure API Manager Analytics using the Standard Setup. For instructions, see Configuring APIM Analytics.

Create the AM_USAGE_UPLOADED_FILES table in the WSO2AM_STATS_DB database. A sample MySQL script to create the table is given below.

CREATE TABLE IF NOT EXISTS AM_USAGE_UPLOADED_FILES (
TENANT_DOMAIN varchar(255) NOT NULL,
FILE_NAME varchar(255) NOT NULL,
FILE_TIMESTAMP TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FILE_PROCESSED tinyint(1) DEFAULT 0,
FILE_CONTENT MEDIUMBLOB DEFAULT NULL,
PRIMARY KEY (TENANT_DOMAIN, FILE_NAME, FILE_TIMESTAMP)
);

Make sure that you correctly configured the WSO2AM_STATS_DB between WSO2 API Manager and WSO2 API-M Analytics.

Download the Microgateway Analytics REST API war file from here. To deploy the file, copy it to the <API-M_ANALYTICS_HOME>/repository/deployment/server/webapps folder.
Download the File Event Receiver for the Microgateway from here. To deploy the file, copy it to the <API-M_ANALYTICS_HOME>/repository/components/dropins folder.
Download the CApp file containing the file event receivers from here. To deploy the file, copy it to the <API-M_ANALYTICS_HOME>/repository/deployment/server/carbonapps folder.

Java system properties

You can define the following Java system properties passed to the server, to configure the custom file event receiver.

Property	Default value	Description
`-Dusage.publishing.frequency=20000`	`300000` (5 minutes)	The frequency in which the database is read by the event receiver. This parameter is defined in milliseconds.
`-Dusage.cleanup.frequency=36000`	`1800000` (30 minutes)	The frequency in which the cleanup task is run, to remove the processed files. This parameter is defined in milliseconds. Even though the cleanup task is run, the files are not removed from the database. To remove the files, define the `-Dfile.retention.days=5` property as given below.
`-Dfile.retention.days=5`	N/A	The allowed duration for files to remain in the database. This parameter is defined in days. If this property is not defined, the files remain in the database indefinitely.
`-Dusage.publishing.thread.count=3`	3	The number of worker threads used to read data from the database.

Example

-Dusage.cleanup.frequency=48000 -Dfile.retention.days=3

Step 2 - Configuring the Microgateway for Analytics

To do the configurations for Microgateway analytics, open the <MICROGW_HOME>/conf/micro-gw.conf file. The sample below shows the analytics-related configurations.

enable=false
uploadingTimeSpanInMillis=600000
uploadingEndpoint="https://localhost:9444/micro-gateway-analytics/v0.10/usage/upload-file"
rotatingPeriod=60000
task.uploadFiles=true
username="admin"
password="admin"

The configurations are described in the table below.

Property	Description
enable	Set this to true to enable Microgateway analytics. When enabled, the Microgateway starts persisting files with the analytics data in the `<MICROGW_HOME>/api-usage-data` folder.
uploadingTimeSpanInMillis	The time interval in which the uploading task is run.
uploadingEndpoint	The endpoint URL of the web application, to which the file has to be uploaded. This web app is deployed in the Analytics server, to retrieve files containing analytics data.
rotatingPeriod	The time interval, after which the file is rotated and compressed. This depends on the TPS (Transactions Per Second) capacity of the environment. Best Practice To avoid creating large files, we recommend setting a low `rotatingPeriod` if your environment has a higher TPS.
task.uploadFiles	Set this to true to enable the file upload task. If this property is disabled, the analytics files are not uploaded to the analytics server, although the files are persisted in the Microgateway system. Best Practice In a distributed setup, the analytics data is uploaded to a shared location from multiple Microgateway nodes. To avoid multiple nodes competing to upload the same file, we recommend enabling the `task.uploadFiles` property in only one node. Make sure that you disable this in the rest of the Microgateway nodes. You can also opt to have your own design as a workaround.
username	Username used in the Analytics server.
password	Password for the corresponding username.