This documentation is for WSO2 Enterprise Service Bus version 5.0.0. For the latest ESB, view the latest WSO2 Enterprise Integrator documentation.

All docs This doc

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This section describes how to set up a WSO2 ESB worker/manager separated cluster and how to configure this cluster with a third-party load balancer. The following sections give you information and instructions on how to set up your cluster.

Note

For information on configuring a minimum high availability deployment cluster of WSO2 ESB Analytics, go to Minimum High Availability Deployment in the WSO2 Clustering Guide.

Table of Contents
maxLevel3
minLevel3

...

Info
titleThings to keep in mind

The configuration steps in this document are written assuming that default 80 and 443 ports are used and exposed by the 3rd party load balancer for this ESB cluster. If any other ports are used instead of the default ones, please replace 80 and 443 values with the corresponding ports in the relevant places.

So with the above in mind, please note the following:

  • Load balancer ports are HTTP 80 and HTTPS 443 as indicated in the deployment pattern above.
  • Direct the HTTP requests to the worker nodes using http://xxx.xxx.xxx.xx3/<service> via HTTP 80 port.
  • Direct the HTTPS requests to the worker nodes using https://xxx.xxx.xxx.xx3/<service> via HTTPS 443 port.
  • Access the management console as  https://xxx.xxx.xxx.xx2/carbon  via HTTPS 443 port
  • In a WSO2 ESB cluster, the worker nodes address service requests on the PassThrough Transport ports (8280 and 8243) and can access the Management Console using the HTTPS 9443 port.

...

  1. Download and unzip the WSO2 ESB binary distribution. Consider the extracted directory as <PRODUCT_HOME>.
  2. Set up the cluster configurations. Edit the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file as follows.
    1. Enable clustering for this node: 
      <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
    2. Set the membership scheme to wka to enable the well-known address registration method (this node sends cluster initiation messages to the WKA members that we define later). 
      <parameter name="membershipScheme">wka</parameter>
    3. Specify the name of the cluster this node will join.
      <parameter name="domain">wso2.esb.domain</parameter>
    4. Specify the IP address (NOT localhost or the host name) of the host used to communicate cluster messages. 
      <parameter name="localMemberHost">xxx.xxx.xxx.xx2</parameter>
    5. Specify the port used to communicate cluster messages. This port number is not affected by the port offset value specified in  <PRODUCT_HOME>/repository/conf/carbon.xml. If this port number is already assigned to another server, the clustering framework automatically increments this port number. However, if two servers are running on the same machine, you must ensure that a unique port is set for each server.
      <parameter name="localMemberPort">4100</parameter>

    6. Specify the well-known members. 
      In the following example, the well-known member is a worker node. Here, the hostName value is the IP address of the host. The port value for the WKA worker node must be the same value as it's localMemberPort (in this case 4200).

      Code Block
      languagexml
      <members>
      	<member>
      		<hostName>xxx.xxx.xxx.xx3</hostName>
      		<port>4200</port> 
      	</member>
      </members>
      Note

      Although this example only indicates one well-known member, it is recommended to add at least two well-known members here. This is done to ensure that there is high availability for the cluster.

      You can also use IP address ranges for the hostName. For example, 192.168.1.2-10. This should ensure that the cluster eventually recovers after failures. One shortcoming of doing this is that you can define a range only for the last portion of the IP address. You should also keep in mind that the smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.

    7. Change the following clustering properties. 

      Code Block
      languagexml
      <parameter name="properties">
                  <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
                  <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
      </parameter>
  3. Configure the HostName. To do this, edit the <PRODUCT_HOME>/repository/conf/carbon.xml file as follows.

    Code Block
    languagexml
    <HostName>esb.wso2.com</HostName>
    <MgtHostName>mgt.esb.wso2.com</MgtHostName>
  4. Enable SVN-based deployment synchronization with the AutoCommit property marked as true. To do this, edit the <PRODUCT_HOME>/repository/conf/carbon.xml file as follows. See Configuring Deployment Synchronizer for more information on this.

    Code Block
    languagexml
    <DeploymentSynchronizer>
    	<Enabled>true</Enabled>
    	<AutoCommit>true</AutoCommit>
    	<AutoCheckout>true</AutoCheckout>
    	<RepositoryType>svn</RepositoryType>
    	<SvnUrl>https://svn.wso2.org/repos/esb</SvnUrl>
    	<SvnUser>svnuser</SvnUser>
    	<SvnPassword>xxxxxx</SvnPassword>
    	<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
    </DeploymentSynchronizer>
  5. In the <PRODUCT_HOME>/repository/conf/carbon.xml file, you can also specify the port offset value. This is ONLY applicable if you have multiple WSO2 products hosted on the same server.

    Expand
    titleClick here for more information on configuring the port offset.

    When you run multiple products/clusters or multiple instances of the same product on the same server or virtual machines (VMs), you must change their default ports with an offset value to avoid port conflicts. An offset defines the number by which all ports in the runtime (e.g., HTTP(S) ports) are increased. For example, if the default HTTP port is 9763 and the offset is 1, the effective HTTP port will change to 9764. For each additional product instance, you set the port offset to a unique value. The offset of the default ports is considered to be 0.

    The port value will automatically be increased as shown in the Port Value column in the following table, allowing all five WSO2 product instances or servers to run on the same machine.

    WSO2 product instance

    Port Offset

    Port Value

    WSO2 server 1

    0

    9443

    WSO2 server 2

    1

    9444

    WSO2 server 3

    2

    9445

    WSO2 server 4

    3

    9446

    WSO2 server 5

    4

    9447

    Warning

    Important: This step is optional and only required if all server instances are running on the same machine. This is not a recommended approach for production environments. Note that you need to change all ports used in your configurations based on the offset value if you are to do this.

    Code Block
    languagexml
    <Ports>
    	...
    	<Offset>0</Offset>
    	...
    </Ports>
  6. Map the host names to the IP. Add the below host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You can map the IP address of the database server. In this example, MySQL is used as the database server, so <MYSQL-DB-SERVER-IP> is the actual IP address of the database server.

    Code Block
    <IP-of-MYSQL-DB-SERVER>	carbondb.mysql-wso2.com
  7. Allow access the management console only through the load balancer. Configure the HTTP/HTTPS proxy ports to communicate through the load balancer by editing the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file as follows.

    Code Block
    languagexml
    <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
    	port="9763"
    	proxyPort="80"
    	...
    	/>
    <Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
    	port="9443"
    	proxyPort="443"
    	...
    	/>
    Expand
    titleClick here for more information on this configuration.

    The Connector protocol tag sets the protocol to handle incoming traffic. The default value is HTTP/1.1, which uses an auto-switching mechanism to select either a blocking Java-based connector or an APR/native connector. If the PATH (Windows) or LD_LIBRARY_PATH (on most UNIX systems) environment variables contain the Tomcat native library, the APR/native connector will be used. If the native library cannot be found, the blocking Java-based connector will be used. Note that the APR/native connector has different settings from the Java connectors for HTTPS.

    The non-blocking Java connector used is an explicit protocol that does not rely on the auto-switching mechanism described above. The following is the value used:
    org.apache.coyote.http11.Http11NioProtocol

    The TCP port number is the value that this Connector will use to create a server socket and await incoming connections. Your operating system will allow only one server application to listen to a particular port number on a particular IP address. If the special value of 0 (zero) is used, Tomcat will select a free port at random to use for this connector. This is typically only useful in embedded and testing applications.

     

...

  1. Download and unzip the WSO2 ESB binary distribution. Consider the extracted directory as <PRODUCT_HOME>.
  2. Set up the cluster configurations. Edit the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file as follows.
    1. Enable clustering for this node.
      <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
    2. Set the membership scheme to wka to enable the well-known address registration method (this node will send cluster initiation messages to WKA members that we will define later).
      <parameter name="membershipScheme">wka</parameter>
    3. Specify the name of the cluster this node will join.
      <parameter name="domain">wso2.esb.domain</parameter>
    4. Specify the IP address (NOT localhost or the host name) of the host used to communicate cluster messages.
      <parameter name="localMemberHost">xxx.xxx.xxx.xx3</parameter>
    5. Specify the port used to communicate cluster messages. If this node is on the same server as the manager node, or another worker node, set this to a unique value, such as 4000 and 4001 for worker nodes 1 and 2. This port number will not be affected by the port offset in carbon.xml. If this port number is already assigned to another server, the clustering framework will automatically increment this port number.
      <parameter name="localMemberPort">4200</parameter>

    6. Define the sub-domain as worker by adding the following property inside the  <parameter name="properties">  element: 
      <property name="subDomain" value="worker"/>

    7. Specify the well-known member by providing the hostName and localMemberPort values. Here, the hostName value is the IP address of the host.
      In the following example, the well-known member is the manager node. Defining the manager node is useful since it is required for the Deployment Synchronizer to function in an efficient manner. The deployment synchronizer uses this configuration to identify the manager and synchronize deployment artifacts across the nodes of a cluster.

      Code Block
      languagexml
      <members>
      	<member>
      		<hostName>xxx.xxx.xxx.xx2</hostName>
      		<port>4100</port> 
      	</member>
      </members>
      Note

      Although this example only indicates one well-known member, it is recommended to add at least two well-known members here. This is done to ensure that there is high availability for the cluster.

      You can also use IP address ranges for the hostName. For example,  192.168.1.2-10. This should ensure that the cluster eventually recovers after failures. One shortcoming of doing this is that you can define a range only for the last portion of the IP address. You should also keep in mind that the smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.

    8. Uncomment and edit the WSDLEPRPrefix element under org.apache.synapse.transport.passthru.PassThroughHttpListener and
      org.apache.synapse.transport.passthru.PassThroughHttpSSLListener in the transportReceiver.

      Code Block
      languagexml
      <parameter name="WSDLEPRPrefix" locked="false">http://esb.wso2.com:80</parameter>
      
      <parameter name="WSDLEPRPrefix" locked="false">https://esb.wso2.com:443</parameter>
  3. Configure the <PRODUCT_HOME>/repository/conf/carbon.xml file to match your clustering configurations with your product.

    1. Configure the HostName. To do this, edit the <PRODUCT_HOME>/repository/conf/carbon.xml file as follows.

      Code Block
      languagexml
      <HostName>esb.wso2.com</HostName>
    2. Enable SVN-based deployment synchronization with the AutoCommit property marked as false. To do this, edit the <PRODUCT_HOME>/repository/conf/carbon.xml file as follows.

      Code Block
      languagexml
      <DeploymentSynchronizer>
      	<Enabled>true</Enabled>
      	<AutoCommit>false</AutoCommit>
      	<AutoCheckout>true</AutoCheckout>
      	<RepositoryType>svn</RepositoryType>
      	<SvnUrl>https://svn.wso2.org/repos/esb</SvnUrl>
      	<SvnUser>svnuser</SvnUser>
      	<SvnPassword>xxxxxx</SvnPassword>
      	<SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
      </DeploymentSynchronizer>
    3. In the <PRODUCT_HOME>/repository/conf/carbon.xml file, you can also specify the port offset value. This is ONLY applicable if you have multiple WSO2 products hosted on the same server.

      Expand
      titleClick here for more information on configuring the port offset.

      When you run multiple products/clusters or multiple instances of the same product on the same server or virtual machines (VMs), you must change their default ports with an offset value to avoid port conflicts. An offset defines the number by which all ports in the runtime (e.g., HTTP/S ports) are increased. For example, if the default HTTP port is 9763 and the offset is 1, the effective HTTP port will change to 9764. For each additional product instance, you set the port offset to a unique value. The offset of the default ports is considered to be 0.

      The port value will automatically be increased as shown in the Port Value column in the following table, allowing all five WSO2 product instances or servers to run on the same machine.

      WSO2 product instance

      Port Offset

      Port Value

      WSO2 server 1

      0

      9443

      WSO2 server 2

      1

      9444

      WSO2 server 3

      2

      9445

      WSO2 server 4

      3

      9446

      WSO2 server 5

      4

      9447

      Warning

      Important: This step is optional and only required if all server instances are running in the same machine. This is not a recommended approach for production environments. Note that you need to change all ports used in your configurations based on the offset value if you are to do this.

      Code Block
      languagexml
      <Ports>
      	...
      	<Offset>0</Offset>
      	...
      </Ports>
  4. Map the host names to the IP. Add the following host entries to your DNS, or “/etc/hosts” file (in Linux) in all the nodes of the cluster. You can map the IP address of the database server. In this example, MySQL is used as the database server, so <MYSQL-DB-SERVER-IP> is the actual IP address of the database server.

    Code Block
    <IP-of-MYSQL-DB-SERVER>	carbondb.mysql-wso2.com
  5. Create the second worker node by getting a copy of the WSO2 product you just configured as a worker node and change the following in the <PRODUCT_HOME>/repository/conf/axis2/axis2.xml file. This copy of the WSO2 product can be moved to a server of its own.
    <parameter name="localMemberPort">4300</parameter>

...

Info

When there is no worker manager separation in your clustering configuration, to ensure that tasks are scheduled in all ESB nodes, you need to edit the clusteringPattern parameter in <ESB_HOME>/repository/conf/axis2/axis2.xml file as follows: 


<parameter name="clusteringPattern">nonWorkerManager</parameter>

...