The following sections provide information and instructions on how to cluster the ESB profile of WSO2 Enterprise Integrator (WSO2 EI) with a third-party load balancer.
The deployment pattern
This deployment scenario uses a two-node ESB cluster. That is, two ESB nodes are configured to serve requests with high availability and scalability. As depicted by the following diagram, the product nodes in the cluster are fronted by an external third-party load balancer, which routes requests to the two nodes on a round-robin basis.
Note that the two ESB nodes are configured as well-known members. It is always recommended to have all nodes of the cluster as well-known members.
When configuring your WSO2 products for clustering, it is necessary to use a specific IP address and not localhost or host names in your configurations.
Configuring the load balancer
The load balancer automatically distributes incoming traffic across multiple WSO2 product instances. It enables you to achieve greater levels of fault tolerance in your cluster and provides the required balancing of load needed to distribute traffic.
Note the following facts when configuring the load balancer:
These configurations are not required if your clustering setup does not have a load balancer.
Load balancer ports are HTTP 80 and HTTPS 443 in the deployment pattern shown in the above diagram. If your system uses any other ports, be sure to replace 80 and 443 values with the corresponding ports when you follow the configuration steps in this section.
- The load balancer directs requests to the server on a round robin basis. For example, the load balancer will direct requests to node 1 (xxx.xxx.xxx.xx1) of the ESB cluster as follows:
HTTP requests will be directed to node 1 using the
http://xxx.xxx.xxx.xx1/<service>URL via HTTP 80 port.
HTTPS requests will be directed to node 1 using the
https://xxx.xxx.xxx.xx1/<service>URL via HTTPS 443 port.
The management console of node 1 will be accessed using the
URL via HTTPS 443 port.
It is recommended to use NGINX Plus as your load balancer of choice.
Install NGINX Plus or Nginx community version configured in a server within your cluster network.
Create a VHost file (ei
.http.conf) in the
/conf.d directory and add the following configurations. This configures NGINX Plus to direct the HTTP requests to the two ESB nodes (xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2) via the HTTP 80 port using the
Create a VHost file (ei.https.conf) in the /etc/nginx/conf.d directory and add the following configurations. This configures NGINX Plus to direct the HTTPS requests to the two ESB nodes (xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2) via the HTTPS 443 port using the
Configure Nginx to access the management console as
https://ui.ei.wso2.com/carbonvia HTTPS 443 port. To do this, create a VHost file (ui
.ei.https.conf) in the
/etc/nginx/conf.d/directory and add the following configurations into it.
Follow the instructions below to create SSL certificates for both ESB nodes.
Enter the host name (ei
.wso2.com) as the common name when creating keys.
Execute the following command to create the Server Key:
Execute the following command to request to sign the certificate:
Execute the following commands to remove the passwords:
Execute the following command to sign your SSL Certificate:
Execute the following command to add the certificate to the
Give the default password
Execute the following command to restart the NGINX Plus server:
Execute the following command if you do not need to restart the server when you are simply making a modification to the VHost file:
Creating the databases
All profiles of WSO2 EI uses a database to store information such as user management details and registry data. All nodes in the cluster must use one central database for config and governance registry mounts. You can create the following databases and associated datasources.
|JDBC user store and authorization manager|
|Shared database for config and governance registry mounts in the product's nodes|
|Local registry space in Node 1|
|Local registry space in Node 2|
It is recommended to use an industry-standard RDBMS such as Oracle, PostgreSQL, MySQL, MS SQL, etc. for most enterprise testing and production environments. However, you can also use the embedded H2 database only for the
Following the steps below to create the databases necessary.
These instructions assume you are installing MySQL as your relational database management system (RDBMS), but you can install another supported RDBMS as needed.
Download and install MySQL Server.
Download the MySQL JDBC driver.
Throughout this guide,
<EI_HOME>refers to the extracted directory of the WSO2 EI product distribution.
- Unzip the downloaded MySQL driver, and copy the MySQL JDBC driver JAR (
mysql-connector-java-x.x.xx-bin.jar) into the
<EI_HOME>/lib/directory of both ESB nodes.
Add the following line to the
/etc/hostsfile to define the hostname for configuring permissions for the new database:
Do this step only if your database is not on your local machine and on a separate server.
- Execute the following command in a terminal/command window, where username is the username you want to use to access the databases: mysql
-u username -p
Specify the password to access the databases with the username you specified when prompted.
Create the databases using the following commands:
About using MySQL in different operating systems
For users of Microsoft Windows, when creating the database in MySQL, it is important to specify the character set as latin1. Failure to do this may result in an error (error code: 1709) when starting your cluster. This error occurs in certain versions of MySQL (5.6.x) and is related to the UTF-8 encoding. MySQL originally used the latin1 character set by default, which stored characters in a 2-byte sequence. However, in recent versions, MySQL defaults to UTF-8 to be friendlier to international users. Hence, you must use latin1 as the character set as indicated below in the database creation commands to avoid this problem. Note that this may result in issues with non-latin characters (like Hebrew, Japanese, etc.). The following is how your database creation command should look.
mysql> create database <DATABASE_NAME> character set latin1;
For users of other operating systems, the standard database creation commands will suffice. For these operating systems, the following is how your database creation command should look.
mysql> create database <DATABASE_NAME>;
Configuring the ESB profile node
Do the following configurations for all nodes of your cluster.
Mounting the registry
Add the following configurations to the
REGISTRY_DB database) mount on both ESB nodes.
Note the following when adding these configurations:
- The existing
wso2registrymust not be removed.
- The datasource you specify in the
<dbConfig name="sharedregistry">tag must match the JNDI Config name you specify in the
The registry mount path denotes the type of registry. For example, ”
/_system/config” refers to configuration Registry, and "
/_system/governance" refers to the governance registry.
The <dbconfig> entry enables you to identify the datasource you configured in the
.xml file. The unique name "sharedregistry" refers to that datasource entry.
<remoteInstance>section refers to an external registry mount. Specify the read-only/read-write nature of this instance, caching configurations and the registry root location in this section.
Also, specify the cache ID in the
<remoteInstance>section. This enables caching to function properly in the clustered environment.
Cache ID is the same as the JDBC connection URL of the registry database. This value is the Cache ID of the remote instance. It should be in the format of
[email protected]$database_url, where
$database_usernameis the username of the remote instance database and
$database_urlis the remote instance database URL. This cacheID denotes the enabled cache. In this case, the database it should connect to is
REGISTRY_DB, which is the database shared across all the nodes. You can find that in the mounting configurations of the same datasource that is being used.
Define a unique name in the
<id>tag for each remote instance. This is then referred to from mount configurations. In the above example, the unique ID for the remote instance is "
Specify the actual mount path and target mount path in each of the mounting configurations. The target path can be any meaningful name. In this instance, it is "
Connecting to the databases
Update the datasources by following the steps given below.
.xml file, and configure the datasources to point to the relevant databases for each ESB node.
- Replace the username, password, and database URL of your MySQL environment accordingly.
- If you have not enabled SSL, append the
useSSL=falseproperty to the value of the
For ESB node 1, configure the datasources to point to the
WSO2_USER_DBdatabases as follows:
For node 2, configure the datasources to point to the
WSO2_USER_DBdatabases as shown below. Change the username, password, and database URL as needed for your environment.
Add the following configuration in the
<EI_HOME>/conf/user-mgt.xml file to configure the user stores. Update the
dataSourceproperty in all nodes in the
<EI_HOME>/conf/user-mgt.xmlfile as shown below to configure the datasource:
Configuring cluster settings
WSO2 products use Hazelcast as its default clustering engine. Given below are the steps for connecting a product node to the cluster.
- Open the
<EI_HOME>/conf/axis2/axis2.xmlfile for each of the two ESB nodes, and appply the following cluster configurations:
Enable clustering for each node as follows:
Set the membership scheme to "wka" to enable the well-known address registration method as shown below. Each node sends cluster initiation messages to the WKA members.
Specify the name of the cluster to which the node joins:
Specify the host to communicate cluster messages. For example, if the IP addresses of the two ESB nodes are xxx.xxx.xxx.xx1 and xxx.xxx.xxx.xx2, they should be specified in the configuration as shown below.
Specify the port to communicate cluster messages as follows:
This port number is not affected by the port offset value specified in the
<EI_HOME>/conf/carbon.xml file. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if there are two servers running on the same machine, ensure that a unique port is set for each server.
Specify the well-known members in the cluster as shown below. For example, when you configure one ESB node, you need to specify the other nodes in the cluster as well-known members as shown below. The port value for the WKA node must be the same value as it's localMemberPort (in this case it is 4100).
You can also use IP address ranges for the hostname (e.g., 192.168.1.2-10). However, you can define a range only for the last portion of the IP address. Smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members. The best practice is to add all the members (including itself) in all the nodes to avoid any conflicts in configurations.
Uncomment and edit the
Uncomment and edit the
<EI_HOME>/conf/carbon.xml file as follows to configure the hostname:
Configuring Hazelcast properties
You can configure the hazelcast properties for the product nodes by following the steps given below.
hazelcast.propertiesfile with the following property configurations, and copy the file to the
The above configurations are explained below.
- Hazelcast shutdown hook: This configuration disables the shutdown hook in hazelcast, which ensures that the hazelcast instance shuts down gracefully whenever the product node shuts down. If the hazelcast shutdown hook is enabled (which is the default behavior of a product), you will see errors such as "Hazelcast instance is not active!" at the time of shutting down the product node: This is because the hazelcast instance shuts down too early when the shutdown hook is enabled.
- Hazelcast logging type: This configuration sets the hazelcast logging type to log4j, which allows hazelcast logs to be written to the
If you have enabled log4j for hazelcast logging as shown above, be sure to enter the configuration shown below in the
log4j.propertiesfile (stored in the
<EI_HOME>/conf/directory). This can be used to configure the log level for hazelcast logging. For a clustered production environment, it is recommended to use INFO as the log level as shown below.
Specify the port offset value in the
This step is optional and only required if all server instances are running on the same machine. This is not recommended for production environments. Change all ports used in your configurations based on the offset value if you are setting a port offset.Click here for more information on configuring the port offset.
When you run multiple products/clusters or multiple instances of the same product on the same server or virtual machines (VMs), change their default ports with an offset value to avoid port conflicts. An offset defines the number by which all ports in the runtime (e.g., HTTP(S) ports) are increased. For example, if the default HTTP port is 9763 and the offset is 1, the effective HTTP port will change to 9764. For each additional product instance, set the port offset to a unique value. The offset of the default ports is zero.
The port value will automatically increase as shown in the Port Value column in the following table, allowing all five WSO2 product instances or servers to run on the same machine.
WSO2 product instance
WSO2 server 1
WSO2 server 2
WSO2 server 3
WSO2 server 4
WSO2 server 5
- Add the host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster to map the hostnames to the IP addresses. For an example, you can map the IP address of the database server. In this example, MySQL is used as the database server, so
<MYSQL-DB-SERVER-IP>is the actual IP address of the database server and the host entry is as follows:
- Edit the
<EI_HOME>/conf/tomcat/catalina-server.xmlfile as follows:
Click here for more information on this configuration.
org.apache.coyote.http11.Http11NioProtocolclass with the port defined as
org.apache.coyote.http11.Http11NioProtocolclass with the port defined as
Connector protocoltag sets the protocol to handle incoming traffic. The default value is
HTTP/1.1, which uses an auto-switching mechanism to select either a blocking Java-based connector or an APR/native connector. If the
LD_LIBRARY_PATH(on most UNIX systems) environment variables contain the Tomcat native library, the APR/native connector will be used. If the native library cannot be found, the blocking Java-based connector will be used. Note that the APR/native connector has different settings from the Java connectors for HTTPS.
The non-blocking Java connector used is an explicit protocol that does not rely on the auto-switching mechanism described above. The following is the value used:
portnumber is the value that this
Connectorwill use to create a server socket and await incoming connections. Your operating system will allow only one server application to listen to a particular port number on a particular IP address. If the special value of 0 (zero) is used, Tomcat will select a free port at random to use for this connector. This is typically only useful in embedded and testing applications.
Deploying artifacts across the nodes
Use the following deployment synchronization recommendations based on the rate of change of artifacts that happen in your cluster:
- For a high rate of changes (i.e., if changes happen very frequently):
- Network File Share (NFS)
- For a medium rate of change
- Remote Synchronization (Rsync)
- For a low rate of changes (i.e., if changes happen once a week):
- use the configuration management system to handle artifacts
- other deployment options (e.g., Puppet, Chef etc.)
However, using a shared file system is the first preference that WSO2 recommends to synchronize the artifacts among the nodes because you can use it to publish artifacts to any of the nodes without any issues. You need to use Rsync only if you are unable to have a shared file system. This is because Rsync does not support two-way synchronization: That is, Rsync allows write permissions only to one node, and thereby creates the vulnerability of a single point of failure.
Using Network File Share (NFS)
You can use a common shared file system such as Network File System (NFS) or any other shared file system as the content synchronization mechanism. You need to mount the
<EI_HOME>/repository/deployment/server directory of the two nodes to the shared file system to share all the artifacts between both nodes.
Using Remote Synchronization (Rsync)
If you are unable to maintain a shared file system, you can synchronize content using Rsync. The Rsync tool, which is a file copying tool is another common approach for synchronizing artifacts across all cluster nodes. Therefore, you can first deploy artifacts in one node of the cluster and then use Rsync to copy those artifacts to other nodes as described below.
Create a file called
nodes-list.txt,which lists all the nodes in the deployment. The following is a sample of the file for two nodes.
Different nodes are separated into individual lines.
Create a file to synchronize the
<PRODUCT_HOME>/repository/deployment/server/directory between the nodes.
You must create your own SSH key and define it as the
pem_file. Alternatively, you can use an existing SSH key. Specify the
ei_server_dirdepending on the location in your local machine. Change the
logs.txtfile path and the lock location based on where they are located in your machine.
Configure rsync in the
<EI_HOME>/repository/tenant/directory to share the tenant artifacts across the cluster.
Execute the following command in your CLI to create a Cron job that executes the above file every minute for deployment synchronization.
Testing the cluster
Follow the steps below to test the cluster.
Deploy artifacts to each product deployment location.
Use a deployment synchronization mechanism to synchronize the artifacts in the
<EI_HOME>/repository/deployment/directory. Always deploy artifacts first to the ESB server profile node with the registry configured as read/write. Next, deploy the artifacts to the other nodes.
Restart the configured load balancer.
- Start both ESB nodes. See Running the Product for instructions.
Check for ‘member joined’ log messages in all consoles.
Additional information on logs and new nodes
When you terminate one node, all nodes identify that the node has left the cluster. The same applies when a new node joins the cluster. If you want to add another new node, copy existing node without any changes if you are running it on a new server (such as xxx.xxx.xxx.184). If you intend to use the new node on a server where another WSO2 product is running, use a copy of node and change the port offset accordingly in the
<EI_HOME>/conf/carbon.xmlfile. You also have to change
<EI_HOME>/conf/axis2/axis2.xml file if that product has clustering enabled. Also, map all hostnames to the relevant IP addresses when creating a new node. The log messages indicate if the new node joins the cluster.
- Access the management console through the LB using the following URL:
- Test load distribution via the following URLs:
Add a sample proxy service with the log mediator in the inSequence so that it will display logs in the terminals, and then observe the cluster messages sent.
Send a request to the endpoint through the load balancer to verify that the proxy service is activated only on the active node(s) while the nodes remain passive. This is to test that the load balancer manages the active and passive states of the nodes, activating nodes as needed and leaving the rest in passive mode. For example, you would send the request to the following URL: http
Tuning performance of the cluster
Follow the steps below to tune performance of the cluster:
The below example parameter values might not be the optimal values for the specific hardware configurations in your environment. Therefore, it is recommended to carry out load tests on your environment to tune the load balancer and other configurations accordingly.
- Change the following default memory allocation settings for the server node and the JVM tuning parameters in the server startup scripts (i.e., the
<EI_HOME>/bin/integrator.batfile) according to the expected server load:
-Xms256m -Xmx1024m -XX:MaxPermSize=256m
Modify important system files, which affect all programs running on the server. It is recommended to familiarize yourself with these files using Unix/Linux documentation before editing them.