This documentation is for WSO2 Data Analytics Server 3.0.1. View documentation for the latest release.

All docs This doc

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

PropertyDefault ValueDescription
carbon.spark.master local

The Spark master has three possible states as follows:

  • local: This starts Spark in the local mode. e.g, carbon.spark.master local or carbon.spark.master local[2]
  • client : This mode results in WSO2 DAS acting as a client for an external Spark cluster. e.g., carbon.spark.master spark://<host name>:<port>. For more details on setting up WSO2 DAS and Apache Spark in this mode, see Connecting a DAS Instance to an Existing External Apache Spark Cluster.
  • cluster: This mode results in WSO2 DAS creating its own Spark cluster using Carbon Clustering. When Spark runs in this mode, it is required to specify a value for the carbon.spark.master.count property. e.g., carbon.spark.master local AND carbon.spark.master.count  <number of redundant masters>
carbon.spark.master.count 1

The maximum number of masters allowed at a given time when DAS creates its own Spark cluster.

Info

This property is applicable only when the Spark master runs in the cluster mode.

carbon.das.symbolic.link
Anchor
symbolic link
symbolic link
This links to your DAS home by default.

The symbolic link for the jar files in the Spark class path.

In a clustered DAS deployment, the directory path for the Spark Class path is different for each node depending on the location of the <DAS_HOME>. The symbolic link redirects the Spark Driver Application to the relevant directory for each node when it creates the Spark class path. The symbolic link should be located in the same path for each <DAS_HOME>.

Info
  • The symbolic link is not specified by default. When it is not specified, the jar files are added in the DAS home.
  • In a multi node DAS cluster that runs in a RedHat Linux environment, you also need to update the <DAS_HOME>/bin/wso2server.sh file with the following entry so that the <DAS_HOME> is exported. This is because the symbolic link may not be resolved correctly in this operating system.

    Export CARBON_HOME=<symbolic link>

Default Spark related configurations

...

It is recommended to run only one executor per DAS worker. If you observe any memory or Spark executor time issues for this executor, you can increase the amount of memory and the number of CPU cores allocated to it.

PropertyDefault ValueDescription
spark.executor.cores1The number of cores allocated to the Spark executors that are running in the DAS node. All the availble CPU cores of the worker are allocated to the executor(s) by default.
spark.executor.memory1gThe amount of CPU memory allocated to the spark executor(s).

Optional Spark related configurations

...