This documentation is for older versions of WSO2 products and may not be relevant now. Please see your respective product documentation for clustering details and configurations.
||
Skip to end of metadata
Go to start of metadata

When WSO2 Carbon servers are deployed in a clustered mode on Kubernetes, the Kubernetes Membership Scheme enables automatically discovering these servers. 

How it works

Once a WSO2 Carbon server starts, the Kubernetes Membership Scheme queries the Kubernets API to retrieve the IP addresses of the pods/containers of the cluster. Next, the configurations of the Hazelcast cluster instance is updated with the newly found pod IP addresses. As a result, the Hazelcast instance gets connected with all the other members in the cluster. 

When a new member is added to the cluster, all the other members automatically get connected to the new member.

The Kubernetes Membership Scheme supports finding the pod IP addresses using the Kubernetes API as well as the DNS.


Follow the steps below to configure the Kubernetes Membership Scheme to use the Kubernets API:

  1. Update the axis2.xml file, in the <CARBON_HOME>/repository/conf/axis2 directory with the following configurations. 

    ParameterDescription
    membershipSchemeThis is the membership scheme that will be used to manage membership of nodes in a cluster, e.g., kubernetes.
    domain

    This is the clustering domain/group.

    There will not be any interference between nodes in different domains. Messages received from members outside the domain will generally be ignored. For special messages, such as cluster management messages or membership messages from members outside the group will be allowed.

    localMemberHost

    This is the hostname or the IP address of the member. Set it to the pod's local IP address. Do not set it to localhost or 127.0.0.1.

    When a container/pod gets started, the init.sh dynamically replaces the local member IP address in WSO2 Docker images. For customizations, please change the init.sh.

    localMemberPortThis is the TCP port that is used by this member and through which other members will contact this member.
    propertiesWhen a member joins a group, these properties are bound to this member so that other members in the group can detect these properties. 
    membershipSchemeClassName
    KUBERNETES_NAMESPACEThis is the Kubernetes Namespace in which the pods are deployed,
    KUBERNETES_SERVICESThese are the Kubernetes Services that belong in the cluster.
    KUBERNETES_MASTER_SKIP_SSL_VERIFICATIONThis defines whether the SSL certificate verification of the Kubernetes API should be carried out or not.
    USE_DNSThis configures the membership schme to DNS (default) or Kuberntes API for pod IP resolution. Set this to false .

    Example:

    <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
                    enable="true">
        
            <parameter name="AvoidInitiation">true</parameter>
    
            <parameter name="membershipScheme">kubernetes</parameter>
            <parameter name="domain">pub.store.am.wso2.domain</parameter>
    
            <parameter name="localMemberHost">172.17.0.2</parameter>
            <parameter name="localMemberPort">4000</parameter>
    
            <!--
            Properties specific to this member
            -->
            <parameter name="properties">
                <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
                <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
                <property name="subDomain" value="worker"/>
            </parameter>
    
            <parameter name="membershipSchemeClassName">org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme</parameter>
            <parameter name="KUBERNETES_NAMESPACE">wso2-demo</parameter>
            <parameter name="KUBERNETES_SERVICES">store,publisher</parameter>
            <parameter name="KUBERNETES_MASTER_SKIP_SSL_VERIFICATION">true</parameter>
            <parameter name="USE_DNS">false</parameter>
    
            <groupManagement enable="false">
                <applicationDomain name="wso2.apim.domain"
                                   description="APIM group"
                                   agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
                                   subDomain="worker"
                                   port="2222"/>
            </groupManagement>
        </clustering>

    The following clustering parameters are required to communicate with the Kubernetes API.

    • KUBERNETES_API_SERVER : This is the Kubernetes API endpoint,e.g., http://172.17.8.101:8080 . Alternatively, an https endpoint can be set via KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT_HTTPS.
    • KUBERNETES_SERVICE_HOST : This is the Kubernetes API host name or IP address, e.g., kuberneteshostname 
    • KUBERNETES_SERVICE_PORT_HTTPS : This is the Kubernetes API https listening port. This must be an Integer value.
    • KUBERNETES_API_SERVER_TOKEN : This is the Kubernetes Master token for authentication (optional), e.g., yourkubernetestoken. Alternatively basic authentication can be set through KUBERNETES_API_SERVER_USERNAME and KUBERNETES_API_SERVER_PASSWORD.
    • KUBERNETES_API_SERVER_USERNAME : This is the Kubernetes Master username (optional), e.g., admin.
    • KUBERNETES_API_SERVER_PASSWORD : This is the Kubernetes Master password (optional).
    • KUBERNETES_NAMESPACE : This is the Kubernetes Namespace in which the pods are deployed.
    • KUBERNETES_SERVICES : These are the Kubernetes Services that are in the cluster.
    • KUBERNETES_MASTER_SKIP_SSL_VERIFICATION : This defines whether the SSL certificate verification of the Kubernetes API should be carried out or not.
    • USE_DNS : This configures the membership schme to DNS (default) or Kuberntes API for pod IP resolution. For Kubernetes, set this to false . For DNS set this to true..

Follow the steps below to use DNS lookups to reolve pod IP addresses:

  1. Download and copy the dnsjava-2.1.8.jar dependency library for DNS lookups  to the <CARBON_HOME>/repository/components/lib directory.

  2. Update the axis2.xml file, in the <CARBON_HOME>/repository/conf/axis2 directory with the following configurations. 

    ParameterDescription
    membershipSchemeThis is the membership scheme that will be used to manage membership of nodes in a cluster, e.g., kubernetes.
    domain

    This is the clustering domain/group.

    There will not be any interference between nodes in different domains. Messages received from members outside the domain will generally be ignored. For special messages, such as cluster management messages or membership messages from members outside the group will be allowed.

    localMemberHost

    This is the hostname or the IP address of the member. Set it to the pod's local IP address. Do not set it to localhost or 127.0.0.1 .

    When a container/pod gets started, the init.sh dynamically replaces the local member IP address in WSO2 Docker images. For customizations, please change the init.sh.

    localMemberPortThis is the TCP port that is used by this member and through which other members will contact this member.
    propertiesWhen a member joins a group, these properties are bound to this member so that other members in the group can detect these properties. 
    membershipSchemeClassName
    KUBERNETES_NAMESPACEThis is the Kubernetes Namespace in which the pods are deployed,
    KUBERNETES_SERVICESThese are the Kubernetes Services that belong in the cluster.
    KUBERNETES_MASTER_SKIP_SSL_VERIFICATIONThis defines whether the SSL certificate verification of the Kubernetes API should be carried out or not.
    USE_DNSThis configures the membership schme to DNS (default) or Kuberntes API for pod IP resolution. Set this to true .

    Example:

    <clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent"
                    enable="true">
        
            <parameter name="AvoidInitiation">true</parameter>
    
            <parameter name="membershipScheme">kubernetes</parameter>
            <parameter name="domain">pub.store.am.wso2.domain</parameter>
    
            <parameter name="localMemberHost">172.17.0.2</parameter>
            <parameter name="localMemberPort">4000</parameter>
    
            <!--
            Properties specific to this member
            -->
            <parameter name="properties">
                <property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
                <property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
                <property name="subDomain" value="worker"/>
            </parameter>
    
            <parameter name="membershipSchemeClassName">org.wso2.carbon.membership.scheme.kubernetes.KubernetesMembershipScheme</parameter>
            <parameter name="KUBERNETES_NAMESPACE">wso2-demo</parameter>
            <parameter name="KUBERNETES_SERVICES">store,publisher</parameter>
            <parameter name="KUBERNETES_MASTER_SKIP_SSL_VERIFICATION">true</parameter>
            <parameter name="USE_DNS">true</parameter>
    
            <groupManagement enable="false">
                <applicationDomain name="wso2.apim.domain"
                                   description="APIM group"
                                   agent="org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent"
                                   subDomain="worker"
                                   port="2222"/>
            </groupManagement>
        </clustering>

    The following clustering parameters are required to perform DNS lookups.

    • KUBERNETES_SERVICES : These are the Kubernetes Services that are in the cluster.
    • KUBERNETES_NAMESPACE : This is the Kubernetes Namespace in which the pods are deployed.

    The services that are used for the DNS lookup should be headless with no cluster IP. For more information, see DNS in Kubernetes.

  • No labels