The following diagram indicates the minimum deployment pattern used for high availability.
DAS supports a deployment scenario that has focus on high availability (HA) along with HA processing. To enable HA processing, you should have two DAS servers in a cluster.
For this deployment, both the DAS nodes should be configured to receive all events. To achieve this, clients can either send all the requests to both the nodes or each request to any one of the two nodes (i.e., using load balancing or failover mechanisms). If clients send all the requests to both nodes, the user has to specify that events are duplicated in the cluster (i.e., the same event comes to all the members of the cluster). Alternatively, if a client sends a request to one node, internally it will send that particular request to the other node as well. This way, even if the clients send requests to only one node, both DAS nodes will receive all the requests.
In this scenario, one DAS node works in active mode and the other works in passive mode. However, both nodes process the whole data.
If the active node fails, the other node becomes active as receives all the requests.
When the failed node is up again, through syncing, it will fetch all the internal states of the current active node.
The newly arrived node will then become the passive node, and start processing all the incoming messages to keep its state allied with the active node, such that it can become active if the current active node fails.
Warning: Some of the requests may be lost during the time the passive node switches to the active mode.
Before you configure a minimum high availability DAS cluster, the following needs to be carried out.
- Download and install WSO2 DAS from here.
Make sure that you have allocated the required memory for DAS nodes, and installed the required supporting applications as mentioned in WSO2 DAS Documentation - Installation Prerequisites.
In WSO2 DAS clustered deployments, Spark is run in a seperate JVM. It is recommended to allocate 4GB of memory for Carbon JVM and 2GB for Spark.
- Follow the steps below to set up MySQL.
Download and install MySQL Server.
Download the MySQL JDBC driver.
Unzip the downloaded MySQL driver zipped archive, and copy the MySQL JDBC driver JAR (
mysql-connector-java-x.x.xx-bin.jar) into the
<DAS_HOME>/repository/components/libdirectory of all the nodes in the cluster.
- Enter the following command in a terminal/command window, where
usernameis the username you want to use to access the databases.
mysql -u username -p
- When prompted, specify the password that will be used to access the databases with the username you specified.
Create two databases named
About using MySQL in different operating systems
For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.
mysql> create database <DATABASE_NAME> character set latin1;
For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.
mysql> create database <DATABASE_NAME>;
Execute the following script for the two databases you created in the previous step.
mysql> source <DAS_HOME>/dbscripts/mysql.sql;
Click here to view the commands for performing steps 6 and 7
Create the following databases in MySQL.
It is recommended to create the databases with the same names given above because they are the default JNDI names that are included in the
<DAS_HOME>/repository/conf/analytics/analytics-conf.xmlfile as shown in the extract below. If you change the name, the
analytics-conf.xmlfile should be updated with the changed name.
When configuring the minimum high availability cluster following setups should be done for both nodes.
- Do the following database related configurations.
Follow the sub steps below to configure the <
DAS_HOME>/repository/conf/datasources/master-datasources.xmlfile as required
Enable the all the nodes to access the users database by configuring a datasource to be used by user manager as shown below.
Enable the nodes to access the registry database by configuring the
WSO2REG_DBdata source as follows.
For detailed information about registry sharing strategies, see the library article Sharing Registry Space across Multiple Product Instances.
<DAS_HOME>/repository/conf/datasources/analytics-datasources.xmlfile as shown below.
For more information, see Datasources in DAS documentation.
To share the user store among the nodes, open the
<DAS_HOME>/repository/conf/user-mgt.xmlfile and modify the
dataSourceproperty of the
<configuration>element as follows.
The datasource name specified in this configuration should be the same as the datasource used by user manager that you configured in sub step a, i.
<DAS_HOME>/repository/conf/registry.xmlfile, add or modify the
dataSourceattribute of the
<dbConfig name="govregistry">element as follows.
Do not replace the following configuration when adding in the mounting configurations. The registry mounting configurations mentioned in the above steps should be added in addition to the following.
- Update the
<DAS_HOME>/repository/conf/axis2/axis2.xmlfile as follows to enable Hazlecast clustering for both nodes.
Click here to view the complete clustering section of the axis2.xml file. with the changes mentioned above.
clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"to true as shown below to enable Hazlecast clustering.
Enable wka mode on both nodes as shown below. For more information on wka mode see About Membership Schemes.
Add both the DAS nodes as well known members in the cluster under the
memberstag in each node as shown in the example below.
For each node, enter the respective server IP address as the value for the
localMemberHostproperty as shown below.
<DAS_HOME>/repository/conf/event-processor.xmlfile as follows to cluster DAS in receiver.
Click here to view the complete event-processor.xml file with the changes mentioned above.
HAmode by setting the following property.
Distributedmodes by setting the following properties.
For each node, enter the respective server IP address of the event under the
HA modeConfig section as shown in the example below. Here, you need to specify the server IP address for event synchronizing nodes, manager nodes as well as presenter nodes.
When you enable the HA mode for WSO2 DAS, state persistence is enabled by default. If there is no real time use case that requires any state information after starting the cluster, you should disable event persistence by setting the
<DAS_HOME>/repository/conf/event-processor.xmlfile as shown below.
When state persistence is enabled for WSO2 DAS, the internal state of DAS is persisted in files. These files are not automatically deleted. Therefore, if you want to save space in your DAS pack, you need to delete them manually.
These files are created in the
<DAS_HOME>/cep_persistence/<tenant-id>directory. This directory has a separate sub-directory for each execution plan. Each execution plan can have multiple files. The format of each file name is
1493101044948_MyExecutionPlan). If you want to clear files for a specific execution plan, you need to leave the two files with the latest timestamps and delete the rest.
When you enable the HA mode, events are synchronized between the active node and the passive node by default. This can cause a high system overhead and affect performance when the throughput per second is very high. Therefore, it is recommeded to configure both nodes to receive the same events instead of allowing event synchronization to take place.
To configure both nodes to receive the same events, set the
event.duplicated.in.cluster=trueproperty for each event receiver configured for the cluster. When this property is set for a receiver, each event received by it is sent to both the nodes in the cluster, and therefore, WSO2 DAS does not carry out event synchronization for that receiver.
Make sure you avoid specifying
localhost /127.0.0.1 /0.0.0.0as the hostname in above configuration. This is because the the hostname of a node is used by the other node in the cluster to communicate with it.
The following node types are configured for the HA deployment mode in the
eventSync: Both the active and the passive nodes in this setup are event synchronizing nodes as explained in the introduction. Therefore, each node should have the host and the port on which it is operating specified under the
management: In this setup, both the nodes carry out the same tasks, and therefore, both nodes are considered manager nodes. Therefore, each node should have the host and the port on which it is operating specified under the
presentation: You can optionally specify only one of the two nodes in this setup as the presenter node. The dashboards in which processed information is displayed are configured only in the presenter node. Each node should have the host and the port on which the assigned presenter node is operating specified under the
<presentation>element. The host and the port as well as the other configurations under the
<presentation>element are effective only when the
presenter enable="falseproperty is set under the
<!-- HA Mode Config -->section.
<DAS_HOME>/repository/conf/analytics/spark/spark-defaults.conffile as follows to use the Spark cluster embedded within DAS.
- Keep the
local. This instructs Spark to create a Spark cluster using the Hazelcast cluster.
2as the value for the
carbon.spark.master.countconfiguration. This specifies that there should be two masters in the Spark cluster. One master serves as an active master and the other serves as a stand-by master.
<DAS_HOME>is in the same path in both nodes, specify the absolute link to the
carbon.das.symbolic.linkparameter. If the
<DAS_HOME>indicates different paths, create a symbolic link pointing to the
<DAS_HOME>, and specify the path to that link for the
carbon.das.symbolic.linkparameter. When you use the symbolic link, note that the name of the link should be the same in both nodes. For more information, see Spark Configurations in DAS documentation.
In a multi node DAS cluster that runs in a RedHat Linux environment, you also need to update the
<DAS_HOME>/bin/wso2server.shfile with the following entry so that the
<DAS_HOME>is exported. This is because the symbolic link may not be resolved correctly in this operating system.
Export CARBON_HOME=<symbolic link>
The following example shows the
<DAS_HOME>/repository/conf/analytics/spark/spark-defaults.conffile with changes mentioned above.
- Keep the
In order to share the C-Apps deployed among the nodes, configure the SVN-based deployment synchroniser. For detailed instructions, see Configuring SVN-Based Deployment Synchronizer.
DAS Minimum High availability Deployment set up does not use a manager and a worker. For the purpose of configuring the deployment synchroniser, you can add the configurations relevant to the manager for the node of your choice, and add the configurations relating to the worker for the other node.
If you do not configure the deployment synchroniser, you are required to deploy any C-App you use in the DAS Minimum High Availability Deployment set up to both the nodes.
Starting the cluster
Once you complete the configurations mentioned above, start the two DAS nodes. If the cluster is successfully configured, the following CLI logs are generated.
The following is displayed in the CLIs of both nodes, and it indicates that the registry mounting is successfully done.
A CLI log similar to the following is displayed for the first node you start to indicate that it has successfully started.
Once you start the second node, a CLI log similar to the following will be displayed for the first node to indicate that another node has joined the cluster.
A CLI log similar to the following is displayed for the second node once it joins the cluster.
Testing the HA deployment
The HA deployment you configured can be tested as follows.
- Access the Spark UIs of the active master and the stand-by master using <
node ip>:8081in each node.
- Information relating to the active master is displayed as shown in the example below.
- Information relating to the stand-by master is displayed as shown in the example below.
- Information relating to the active master is displayed as shown in the example below.
- Click the links under Running Applications in the Spark UI of the active master to check the Spark application UIs of those applications. A working application is displayed as shown in the following example.
- Click the Environment tab of a Spark application UI to check whether all the configuration parameters are correctly set. You can also check whether the class path variables in this tab can be accessed manually.
- Check the Spark UIs of workers to check whether they have running executors. If a worker UI does not have running executors or if it is continuously creating executors, it indicates an issue in the Spark cluster configuration. The following example shows a worker UI with a running executor.
- Check the symbolic parameter, and check if you could manually access it via a
cd <directory>command in the CLI.
- Log into the DAS Management Console and navigate to Main => Manage => Batch Analytics => Console to open the Interactive Analytics Console. Run a query in this console.