WSO2 Data Analytics Server is succeeded by WSO2 Stream Processor. To view the latest documentation for WSO2 SP, see WSO2 Stream Processor Documentation.

All docs This doc
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

This section explains how to upgrade to DAS 3.1.0 from DAS 3.0.1. For more information on release versions, see the Release Matrix.

You cannot roll back the upgrade process. However, it is possible to restore a backup of the previous database so that you can restart the upgrade progress.

Preparing to upgrade

The following prerequisites should be completed before upgrading.

  • Make a backup of the DAS 3.0.1 database and copy the <DAS_HOME_3.0.1> directory in order to backup the product configurations.
  • Download any CAR files that are deployed in the DAS server and save them. Then delete them from DAS 3.0.1. This removes the scheduled tasks relating to the deployed artifacts and allows migration to be carried out smoothly. For more information, see Packaging Artifacts as a C-App Archive.
  • Download WSO2 DAS 3.1.0 from http://wso2.com/products/data-analytics-server/.
  • Make the required changes in the datasource files in the <DAS_HOME>/repository/conf/datasources  directory to make sure that they are configured as required. For more information, see Datasources.

Migrating Analytics tables

The table schema internally used by DAS 3.1.0 is different to that of DAS 3.0.x. Therefore, this meta data transformation should be done before using the Analytics tables in DAS 3.0.x in DAS 3.1.0 using the following procedure. The stored data itself is backward compatible. Therefore, it does not go through all the data. Instead, a constant time metadata conversion in tables is carried out.

  1. Navigate to the <DAS_HOME>/bin directory.
  2. Issue one of the following commands as appropriate.
    • On Windows:   analytics-backup.bat -migrateTableSchemaV30To31
    • On Linux/Solaris/Mac OS:   ./analytics-backup.sh -migrateTableSchemaV30To31

Migrating the configurations

Once you have completed the database upgrade, You can proceed with the configuration changes as explained below. The configurations can be directly migrated from DAS 3.0.1 to DAS 3.1.0.

 Configurations should not be copied directly between servers.

To connect DAS 3.1.0 to the upgraded database, configure the following files:

  1. Go to the <DAS_HOME>/repository/conf directory and update the datasource references in the user-mgt.xml and registry.xml files to match the updated configurations in the master-datasources.xml file that you made in the above step.
    For detailed instructions to configure the  user-mgt.xml file, see User Management.
    For detailed instructions to configure the  registry.xml file, see Registry.
  2. Compare the configuration files in the < DAS_HOME>/repository/conf/data-bridge directory for DAS 3.0.1, and 3.1.0.
  3. Compare the configuration files in the <DAS_HOME>/repository/conf/analytics directory for DAS 3.0.1, and 3.1.0. Change the default values in DAS 3.1.0 to the same values you have specified in DAS 3.0.1.
  4. Change the default values in <DAS_3.1.0_HOME>/repository/conf/event-processor.xml file to the same values you have specified in the <DAS_3.0.1_HOME>/repository/conf/event-processor.xml file.

    If you enable the HA mode for WSO2 DAS by setting the <mode name="HA" enable="true"> property in the event-processor.xml file, state persistence is enabled by default. If there is no real time use case that requires any state information after starting the cluster, you should disable event persistence by setting the persistence attribute to false in the same file as shown below.

    <persistence enable="false">
        <persistenceIntervalInMinutes>15</persistenceIntervalInMinutes>
        <persisterSchedulerPoolSize>10</persisterSchedulerPoolSize>
        <persister class="org.wso2.carbon.event.processor.core.internal.persistence.FileSystemPersistenceStore">
            <property key="persistenceLocation">cep_persistence</property>
        </persister>
    </persistence>
  5. Check for any other configurations that were done for DAS 3.0.1 based on your solution and update the configuration files in DAS 3.1.0 accordingly. For example, configurations related to external user stores, caching, mounting, transports etc.

Migrating index information

Each node stores its index data in the <DAS_HOME>/repository/data directory. There are two possible approaches to migrate this data. Click on the relevant tab based on the approach you want to follow.

This method involves running the analytics data backup/restore tool with the -reindexEvents setting to index data. Here, the indexing is done using the existing data in the database to which you already connected WSO2 DAS 3.1.0 in the Migrating the configurations section. For more information, see Backing Up or Restoring Analytics Data.

Follow the steps below to migrate the index information from DAS 3.0.1 to DAS 3.1.0.

  1. Copy the contents of the <DAS_3.0.1>/repository/data directory and replace the contents of the <DAS_3.1.0>/repository/data directory with it.
  2. Take copies of the following files in the <DAS_3.0.1_HOME> and use them to replace the same files located in the   <DAS_3.1.0_HOME>
    • <DAS_HOME>/repository/conf/analytics/my-node-id.dat
    • <DAS_301>/repository/conf/analytics/local-shard-allocation-config.conf

You should not mix the configuration files between nodes. The configuration files and index data of DAS 3.0.1 - node 1 should be copied to DAS 3.1.0 - node 1, and the configuration files and index data of DAS 3.0.1 - node 2 should be copied to DAS 3.1.0 - node 2 etc.

 

Migrating artifacts

All artifacts can be migrated from DAS 3.0.1 to DAS 3.1.0 via Composite Application Archive (CAR) files. For detailed instructions package artifacts as C-App archives, see Packaging Artifacts as C-App Archive .

You should manually deploy Composite Application Archive (CAR) files that you have in DAS 3.0.1 to DAS 3.1.0.

To migrate deployment artifacts:

  • Copy the <DAS_HOME>/repository/deployment/server directory from DAS 3.0.1 to DAS 3.1.0.
  • If multi-tenancy is used, copy the <DAS_HOME>/repository/tenants directory from DAS 3.0.1 to DAS 3.1.0.

Additional steps are required to migrate Spark scripts and dashboard elements and explained in the following sub sections.

Updating Spark scripts

The CarbonJDBC provider was rewritten for DAS 3.1.0. Therefore, the following additional input parameters are required to be added to the Spark Scripts migrated to DAS 3.1.0 for which Carbon JDBC is used as the provider.

  • schema (Required): The schema to be used within Spark for the specifies table.
  • primaryKeys (optional): Any database level unique keys that are either already specified or needed for the table.

The following example shows the difference in the query format between DAS 3.0.1 and DAS 3.1.0.

DAS 3.0.1

CREATE TEMPORARY TABLE <temp_table> using CarbonJDBC options (dataSource "<datasource name>" , tableName "<table name>" );

DAS 3.1.0

CREATE TEMPORARY TABLE <temp_table> using CarbonJDBC options (dataSource "<datasource name>" , tableName "<table name>" , schema "<schema>" [, primaryKeys "<primaryKeys>" ]);

For more information about improvements made to the CarbonJDBC provider, see Creating the Table Using Carbon JDBC as the Provider.

Updating execution plans

WSO2 DAS 3.1.0 uses Siddhi 3.1.2 whereas DAS 3.0.1 uses Siddhi 3.0.5.

All Siddhi versions older than Siddhi 3.1.2 add a length window of size 1 for the initial stream of join queries. From Siddhi 3.1.2 onwards, a length window of size 0 is added. Therefore, once you migrate execution plans from DAS 3.0.1 to DAS 3.1.0, you need to manually add a length window of size 1 for the join queries as shown below (i.e. if you were relying on a length window of size 1 to be added by Siddhi in CEP 4.1.0).

e.g., The following query is included in an execution plan in DAS 3.0.1.

DAS 3.0.1
from TempStream[temp > 30.0]#window.time(1 min) as T join RegulatorStream[isOn == false] as R
  on T.roomNo == R.roomNo
select T.roomNo, R.deviceID, 'start' as action
insert into RegulatorActionStream;

 

When this execution plan is migrated to DAS 3.1.0, #window.length(1) property should be manually added to it as shown below.

DAS 3.1.0
from TempStream[temp > 30.0]#window.time(1 min) as T  join RegulatorStream[isOn == false]#window.length(1) as R
  on T.roomNo == R.roomNo
select T.roomNo, R.deviceID, 'start' as action
insert into RegulatorActionStream;

Migrating dashboard elements

The Analytics Dashboard functionality in DAS 3.1.0 is powered by WSO2 Dashboard Server 2.1.0 whereas the same functionality in DAS 3.0.1 and earlier versions is powered by WSO2 Gadget Server 1.4.0. This section explains how to migrate the gadgets and dashboards from DAS 3.0.1 to 4.2.0.

There are two methods that can be used to migrate gadgets and dashboards between the two DAS versions.

  • Deploying the gadgets and dashboards in a CAR file
  • Migrating the dashboards and gadgets pointing your product home directory.

In both methods, the following two files are used to execute the migration.

File NamePathDescription
ds-migration-1.0.x-2.0.x.bat or ds-migration-1.0.x-2.0.x.sh<DAS_HOME>/migration/dashboards/migration-1.0.x-2.0.xThis triggers the execution of the migration
migration.xml<DAS_HOME>/migration/dashboards/migration-1.0.x-2.0.xThis contains the migration configuration that specifies the mode and user inputs.

 

Click on the relevant tab depending on the method you want to follow.

Follow the procedure below to migrate gadgets and dashboards using CAR files.

  1. Open the <DAS_HOME>/migration/dashboard/migration1.0x-2.0.x/migration.xml file, and comment out the Portal File Migration section as shown in the example below.

            <!-- Portal File Migration-->
            <!-- Uncomment the below part to enable Portal migration -->
    
            <!-- DSMmigration-->
    
                <!-- Mode - Specifiy the migration type CAR or Portal -->
                <!-- Mode>Portal</Mode-->
    
                <!-- SourceDir - Specify the older PRODUCT_HOME direcotory.-->
                <!-- <SourceDir></SourceDir> -->
    
                <!-- DestinationDir - Specify the destination directory for migrated Store and Dashboards -->
                <!-- <DestinationDir></DestinationDir>-->
    
                <!-- TrustStoreLocation - Specify the trustedstore location. EX:- PRODUCT_HOME/repository/resources/security/wso2carbon.jks -->
                <!-- <TrustStoreLocation>YOUR_PRODUCT_HOME/repository/resources/security/wso2carbon.jks</TrustStoreLocation> -->
    
                <!-- TrustStorePassword - Specify the trustedstore password. EX:- wso2carbon -->
                <!-- <TrustStorePassword>wso2carbon</TrustStorePassword> -->
    
                <!-- SourceURL - Specify the source url - Older version of your product . EX:-https://hostname:port -->
                <!-- <SourceURL>https://localhost:9443</SourceURL> -->
    
                <!-- SourceUsername - Specify the carbon.super user's username EX:-admin -->
                <!-- <SourceUsername>admin</SourceUsername> -->
    
                <!-- SourcePassword - Specify the carbon.super user's password EX:-admin -->
                <!-- <SourcePassword>admin</SourcePassword> -->
    
                <!-- DestinationURL - Specify the destination url - Newer version of your product. EX:-https://hostname:port -->
                <!-- <DestinationURL>https://localhost:9444</DestinationURL> -->
    
                <!-- DestinationUsername - Specify the carbon.super user's username EX:-admin -->
                <!-- <DestinationUsername>admin</DestinationUsername> -->
    
                <!-- DestinationPassword - Specify the carbon.super user's password EX:-admin -->
                <!-- <DestinationPassword>admin</DestinationPassword> -->
    
                <!-- TenantDomains - Specify the tenant domain - This will migrate all the dashboards within these tenants. If you do not put anything here, It will update all the tenant dashboards -->
                <!-- <TenantDomains></TenantDomains>-->
            <!-- </DSMigration> -->
  2. Run the following command to run the migration script.
    For Windows: <DAS-3.0.1_HOME>/migration/dashboards/migration-1.0.x-2.0.x/ds-migration-1.0.x-2.0.x.bat --run
    For Linux: <DAS-3.0.1_HOME>/migration/dashboards/migration-1.0.x-2.0.x/ds-migration-1.0.x-2.0.x.sh
  3. Pack the required gadgets and dashboards in WSO2 DAS 3.0.1 to a CAR file and deploy it in WSO2 DAS 3.1.0. For more information, see Packaging Artifacts as C-App Archive.
  4. Perform the following steps for the real time gadgets (i.e. gadgets that use real time event streams as their information source).

    <DESTINATION_DIRECTORY> is the directory you specified for the DestinationDir parameter in the Portal File Migration section of the <DAS_HOME>/migration/dashboard/migration1.0x-2.0.x/migration.xml file.

     

    1. Navigate to the home directory of the gadget. (e.g., <DESTINATION_DIRECTORY>/store/carbon.super/fs/gadget/my_gadget).
    2. Replace the <DESTINATION_DIRECTORY>/store/carbon.super/fs/gadget/my_gadget/js/main.js file with the <DAS_HOME>/migration/dashboards/migration-1.0.x-2.0.x/resources/main.js file.
    3. Similarly, replace the <DESTINATION_DIRECTORY>/store/carbon.super/fs/gadget/my_gadget/js/outputAdapterUiLibrary.js file with <DAS_HOME>/migration/dashboards/migration-1.0.x-2.0.x/resources/outputAdapterUiLibrary.js file.

In this approach the product home directory is defined as the source file. It migrates gadgets, layouts and widgets into the newer DAS version and copies them into the destination location specified by you.

Follow the procedure below to migrate gadgets and dashboards using the Portal.

  1. Open the <DAS_HOME>/migration/dashboard/migration1.0x-2.0.x/migration.xml file, and comment out the CAR File Migration section as shown in the example below.

    <!-- CAR File Migration-->
    <!-- Uncomment the below part to enable CAR file migration -->
    <DSMigration>
        <!-- Mode - Specifiy the migration type CAR or Portal -->
        <Mode>CAR</Mode>
        <!-- SourceDir - Specify the older PRODUCT_HOME direcotory.-->
        <SourceDir>CAR_FILE_SOURCE_DIRECTORY</SourceDir>
        <!-- DestinationDir - Specify the destination directory for migrated Store and Dashboards -->
        <DestinationDir>CAR_FILE_DESTINATION_DIRECTORY</DestinationDir>
    </DSMigration>
  2. Configure the parameters in the Portal File Migration section as explained in the table below.

    ParameterDescriptionExample
    ModeThe migration mode. This should be CAR (if you are migrating the dashboard elements in the CAR file), or Portal (if you are migrating the dashboard elements in via the portal). 
    SourceDirThe source of the dashboard elements to be migrated. The <DAS_3.0.1_HOME> should be specified as the source directory. 
    DestinationDirThe destination directory for the migrated Store. 
    TrustStoreLocationThis parameter specifies the trusted store location.<DAS_HOME>/repository/resources/security/wso2carbon.jks
    TrustStorePasswordThe password of the trust store.ashboardwso2carbon
    SourceURLThe source URL. This should be the URL of the DAS 3.0.1 instance from which you are migrating the dashboard elements.https://hostname:port
    SourceUsernameSpecify the username of the carbon super user.admin
    SourcePasswordSpecify the password of the carbon super user.admin
    DestinationURLThe destination URL. This should be the URL of the DAS 3.1.0 to which you are migrating the dashboard elements.https://hostname:port
    DestinationUsernameSpecify the username of the carbon super user.admin
    DestinationPasswordSpecify the password of the carbon super user.admin
    TenantDomainsThis parameter specifies the domains of all the tenants of whom the dashboards should be updated. If no value is specified for this parameter, the dashboards of all the tenants are updated. 

    The parameters that are required to have values depend on which dashboard elements you need to migrate as explained in the table below.

    OptionDescriptionRequired ParametersConfiguration
    1Convert gadgets, widgets, and layouts in DAS 3.0.1 into the newer version (i.e. based on WSO2 Dashboard Server 2.x.x) and copy them into the destination directory. This allows you to merge the migrated store into the newer version of the DAS dashboards store.
    • Mode
    • SourceDir
    • DestinationDir
    <DSMmigration>
        <Type>Portal</Type>
        <SourceDir>YOUR_PRODUCT_HOME</SourceDir>
        <DestinationDir>YOUR_DESTINATION_DIRECTORY</DestinationDir>
    <TrustStoreLocation></TrustStoreLocation>
        <TrustStorePassword></TrustStorePassword>
        <SourceURL></SourceURL>
        <SourceUsername></SourceUsername>
        <SourcePassword></SourcePassword>
        <DestinationURL></DestinationURL>
        <DestinationUsername></DestinationUsername>
        <DestinationPassword></DestinationPassword>
        <TenantDomains></TenantDomains>
    </DSMigration>
    2Convert gadgets, widgets, and layouts in DAS 3.0.1 into the newer version (i.e. based on WSO2 Dashboard Server 2.x.x) and copy them into the destination directory. In addition, get all the dashboards to reside in your source server (specified via the sourceURL parameter) and migrate it to the newer version. Once migration is done, it is copied into the destination directory defined in the migration.xml file. 
    • Mode
    • SourceDir
    • DestinationDir
    • TrustStoreLocation
    • TrustStorePassword
    • SourceURL
    • SourceUsername
    • SourcePassword
     
    <DSMmigration>
        <Type>Portal</Type>
        <SourceDir>YOUR_PRODUCT_HOME</SourceDir>
        <DestinationDir>YOUR_DESTINATION_DIRECTORY</DestinationDir>
    <TrustStoreLocation>YOUR_PRODUCT_HOME/repository/resources/security/wso2carbon.jks</TrustStoreLocation>
        <TrustStorePassword>wso2carbon</TrustStorePassword>
        <SourceURL>https://localhost:9443</SourceURL>
        <SourceUsername>admin</SourceUsername>
        <SourcePassword>admin</SourcePassword>
        <DestinationURL></DestinationURL>
        <DestinationUsername></DestinationUsername>
        <DestinationPassword></DestinationPassword>
        <TenantDomains></TenantDomains>
    </DSMigration>
    3

    Convert gadgets, widgets, and layouts in DAS 3.0.1 into the newer version (i.e. based on WSO2 Dashboard Server 2.x.x) and copy them into the destination directory. Update the dashboards in the server specified by the DestinationURL.

    This requires you to maintain the same trust store location and the trust password in both DAS versions.

    • Mode
    • SourceDir
    • DestinationDir
    • TrustStoreLocation
    • TrustStorePassword
    • SourceURL
    • SourceUsername
    • SourcePassword
    • DestinationURL
    • DestinationUsername
    • DestinationPassword
    <DSMmigration>
        <Type>Portal</Type>
        <SourceDir>YOUR_PRODUCT_HOME</SourceDir>
        <DestinationDir>YOUR_DESTINATION_DIRECTORY</DestinationDir>
    <TrustStoreLocation>YOUR_PRODUCT_HOME/repository/resources/security/wso2carbon.jks</TrustStoreLocation>
        <TrustStorePassword>wso2carbon</TrustStorePassword>
        <SourceURL>https://localhost:9443</SourceURL>
        <SourceUsername>admin</SourceUsername>
        <SourcePassword>admin</SourcePassword>
        <DestinationURL>https://localhost:9444</DestinationURL>
        <DestinationUsername>admin</DestinationUsername>
        <DestinationPassword>admin</DestinationPassword>
        <TenantDomains></TenantDomains>
    </DSMigration>
  3. Run the following command to run the migration script.
    For Windows: <DAS-3.0.1_HOME>/migration/dashboards/migration-1.0.x-2.0.x/ds-migration-1.0.x-2.0.x.bat --run
    For Linux: <DAS-3.0.1_HOME>/migration/dashboards/migration-1.0.x-2.0.x/ds-migration-1.0.x-2.0.x.sh
  4. Perform the following steps for each gadget.

    <DESTINATION_DIRECTORY> is the directory you specified for the DestinationDir parameter in the Portal File Migration section of the <DAS_HOME>/migration/dashboard/migration1.0x-2.0.x/migration.xml file.

     

    1. Navigate to the home directory of the gadget. (e.g., <DESTINATION_DIRECTORY>/store/carbon.super/fs/gadget/my_gadget).
    2. Replace the <DESTINATION_DIRECTORY>/store/carbon.super/fs/gadget/my_gadget/js/main.js file with the <DAS_HOME>/migration/dashboards/migration-1.0.x-2.0.x/resources/main.js file.
    3. Similarly, replace the <DESTINATION_DIRECTORY>/store/carbon.super/fs/gadget/my_gadget/js/outputAdapterUiLibrary.js file with <DAS_HOME>/migration/dashboards/migration-1.0.x-2.0.x/resources/outputAdapterUiLibrary.js file.
Validating path changes

DAS 3.1.0 has a different directory structure to DAS 3.0.1 that affects the path to the gadget store as shown below.

DAS 3.0.1: <DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/gadget
DAS 3.1.0: <DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/<StoreType [fs/es]>/gadget

Therefore, if a gadget has relative paths included in its index.xml file (i.e. in the <DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/<StoreType [fs/es] >/gadget/<GADGET_NAME> path), you should check whether they are updated with the change in the directory structure. This check should be carried out after running the migration script.

e.g., A path is given as follows in the index.xml file in DAS 3.0.1.

<script language="javascript" type="text/javascript" src="../../../../js/igviz.js"></script>

The same link should be given as follows in DAS 3.1.0 due to the change in the directory structure (i.e. an additional ../ should be added for the new directory added to the structure).

<script language="javascript" type="text/javascript" src="../../../../../js/igviz.js"></script>

Updating layout configurations

The dashboard layout configurations are defined in the index.xml file. This file differs as follows based on the DAS version.

DAS Version3.0.13.1.0
Location<DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/layout/<LAYOUT_NAME><DAS_HOME>/repository/deployment/server/jaggeryapps/portal/store/<Tenant_Name>/<StoreType [fs/es]>/layout/<LAYOUT_NAME>
Indexing1 based indexing0 based indexing
Format
{
    "blocks": [
        { "id": "a", "row": 1, "col": 1, "size_x": 4, "size_y": 3 },
        { "id": "b", "row": 1, "col": 5, "size_x": 4, "size_y": 3 },
        { "id": "c", "row": 1, "col": 9, "size_x": 4, "size_y": 3 },
        { "id": "d", "row": 4, "col": 1, "size_x": 4, "size_y": 3 },
        { "id": "e", "row": 4, "col": 5, "size_x": 4, "size_y": 3 },
        { "id": "f", "row": 4, "col": 9, "size_x": 4, "size_y": 3 }
    ]
}
{
    "blocks": [
        { "id": "a", "x": 0, "y": 0, "width": 4, "height": 3 },
        { "id": "b", "x": 4, "y": 0, "width": 4, "height": 3 },
        { "id": "c", "x": 8, "y": 0, "width": 4, "height": 3 },
        { "id": "d", "x": 0, "y": 3, "width": 4, "height": 3 },
        { "id": "e", "x": 4, "y": 3, "width": 4, "height": 3 },
        { "id": "f", "x": 8, "y": 3, "width": 4, "height": 3 }
    ]
}

 

The layouts migrated from DAS 3.0.1 to DAS 3.1.0 will have the format in which layouts are saved in DAS 3.0.1. To make them compatible with DAS 3.1.0

  1. Replace "size_x" and "size_y" attributes with "width" and "height" respectively.
  2. Replace "row" and "col" attributes with "x" and "y" respectively, and reduce their values by 1 to match the 0 based indexing.

e.g., { "id": "a", "row": 1, "col": 1, "size_x": 4, "size_y": 3 } should be replaced with { "id": "a", "x": 0, "y": 0, "width": 4, "height": 3 }.

Testing the upgrade

Verify that all the required scenarios are working as expected. This confirms that the upgrade is successful.

  • No labels