This documentation is for WSO2 Enterprise Integrator version 6.4.0 . View documentation for the latest release in the 6.x.x family and the latest release in the 7.x.x family.

All docs This doc
||
Skip to end of metadata
Go to start of metadata

This section describes the parameters you need to configure to tune the performance of the Analytics profile when it is affected by high load, network traffic etc. You need to tune these parameters based on the deployment environment.

Tuning carbon.xml parameters

The following parameter is configured in the <EI_HOME>/conf/carbon.xml file.

ParameterDescriptionDefault ValueTuning Recommendation
AnalyticsServerPublishingIntervalThe number of milliseconds that should elapse after a batch of statistical data is processed to be published in the Analytics Dashboard before sending another batch.2000

The default value of 2000 milliseconds (i.e. 5 seconds) is recommended.

When WSO2 EI is handling a high load of requests, this value can be reduced to increase the frequency with which the resulting statistics published in the Analytics Dashboard. This helps to avoid storing too much data in WSO2 EI causing an overconsumption of memory.

When the load of requests handled by WSO2 EI is comparatively low, this time interval can be increased to reduce the system overhead incurred by frequent processing.

Tuning data-agent parameters

The parameters in the <EI-ANALYTICS_HOME>/conf/data-bridge/data-agent-config.xml file can be tuned as follows.

ParameterDescriptionDefault ValueTuning Recommendation
QueueSizeThe number of messages that can be stored in WSO2 EI at a given time before they are sent to be published in the Analytics Dashboard.32768

This value should be increased when the Analytics profile is busy due to a request overload or if there is high network traffic. This prevents the generation of the queue full, dropping message error.

When the Analytics profile is not very busy and when the network traffic is relatively low, the queue size can be reduced to avoid an overconsumption of memory.

The number specified for this parameter should be a power of 2.


BatchSizeThe WSO2 EI statistical data sent to the Analytics profile to be published in the Analytics Dashboard are grouped into batches. This parameter specifies the number of requests to be included in a batch.200This value should be tuned in proportion to the volume of requests sent from WSO2 EI to the Analytics profile. This value should be reduced if you want to reduce the system overhead of the Analytics profile. This value should be increased if WSO2 EI is generating a high amount of statistics and if the QueueSize cannot be further increased without causing an overconsumption of memory.
CorePoolSizeThe number of threads allocated to publish WSO2 EI statistical data to the Analytics Dashboard via Thrift at the time WSO2 EI is started. This value increases when the throughput of statistics generated increases. However, the number of threads will not exceed the number specified for the MaxPoolSize parameter.1The number of available CPU cores should be taken into account when specifying this value. Increasing the core pool size may improve the throughput of statistical data published in the Analytics Dashboard, but latency will also be increased due to context switching.
MaxPoolSizeThe maximum number of threads that should be allocated at any given time to publish WSO2 EI statistical data in the Analytics Dashboard.1The number of available CPU cores should be taken into account when specifying this value. Increasing the maximum core pool size may improve the throughput of statistical data published in the Analytics Dashboard, since more threads can be spawned to handle an increased number of events. However, latency will also increase since a higher number of threads would cause context switching to take place more frequently.
MaxTransportPoolSizeThe maximum number of transport threads that should be allocated at any given time to publish WSO2 EI statistical data to the Analytics Server.250This value must be increased when there is an increase in the throughput of events handled by WSO2 EI Analytics.

The value of the tcpMaxWorkerThreads parameter in the <EI-ANALYTICS_HOME>/repository/conf/data-bridge/data-bridge-config.xml must change based on the value specified for this parameter and the number of data publishers publishing statistics. e.g., When the value for this parameter is 250 and the number of data publishers is 7, the value for the tcpMaxWorkerThreads parameter must be 1750 (i.e., 7 * 250). This is because you need to ensure that there are enough receiver threads to handle the number of messages published by the data publishers.

SecureMaxTransportPoolSizeThe maximum number of secure transport threads that should be allocated at any given time to publish WSO2 EI statistical data to the Analytics Server.250

This value must be increased when there is an increase in the throughput of events handled by WSO2 EI Analytics.

The value of the sslMaxWorkerThreads parameter in the <EI-ANALYTICS_HOME>/repository/conf/data-bridge/data-bridge-config.xml must change based on the value specified for this parameter and the number of data publishers publishing statistics. e.g., When the value for this parameter is 250 and the number of data publishers is 7, the value for the sslMaxWorkerThreads parameter must be 1750 (i.e., 7 * 250). This is because you need to ensure that there are enough receiver threads to handle the number of messages published by the data publishers.


 Click here to view a list of errors that can be avoided by tuning the above parameters
  • TID: [-1234] [] [2016-05-05 19:12:02,393]  WARN {org.wso2.carbon.databridge.agent.DataPublisher} -  Event queue is full, unable to process the event for endpoint group [ ( Receiver URL : tcp://127.0.0.1:7611, Authentication URL : ssl://127.0.0.1:7711) ], dropping the event. {org.wso2.carbon.databridge.agent.DataPublisher}
  • TID: [-1] [] [2016-05-05 19:11:18,642] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpoint} -  Unable to send events to the endpoint.  {org.wso2.carbon.databridge.agent.endpoint.DataEndpoint}
    org.wso2.carbon.databridge.agent.exception.DataEndpointException: Cannot send Events
  • TID: [-1] [] [2016-05-05 19:11:18,642] ERROR {org.wso2.carbon.databridge.agent.endpoint.DataEndpoint} -  Unable to send events to the endpoint.  {org.wso2.carbon.databridge.agent.endpoint.DataEndpoint}
    org.wso2.carbon.databridge.agent.exception.DataEndpointException: Cannot send Events
  • 2018-01-18 12:00:19,362  WARN {org.wso2.carbon.databridge.agent.DataPublisher} -  Event queue is full, unable to process the event for endpoint group [ ( Receiver URL : tcp://10.36.239.67:7611, Authentication URL : ssl://10.36.239.67:7711),( Receiver URL : tcp://10.36.239.70:7611, Authentication URL : ssl://10.36.239.70:7711) ], 54995 events dropped so far.
  • No labels