The latest version for DAS is WSO2 Data Analytics Server 3.1.0. View documentation for the latest release.
WSO2 Data Analytics Server is succeeded by WSO2 Stream Processor. To view the latest documentation for WSO2 SP, see WSO2 Stream Processor Documentation.

All docs This doc
Skip to end of metadata
Go to start of metadata

The following sections explain how the performance of Spark can be monitored via JMX.

Setting up JMX to monitor Spark

To enable Spark monitoring via JMX, follow the steps below:

If your DAS deployment is a cluster, the following configurations need to be added in all the nodes of the cluster.

  1. Stop the DAS server(s). For detailed instructions, see Running the Product.
  2. Create a file named metrics.properties, include the following configuration in it and save in in the <DAS_HOME>/repository/conf/analytics/spark directory.

    *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
  3. In the <DAS_HOME>/repository/conf/analytics/spark/spark-defaults.conf file, add a reference to the metrics.properties  file you created in the previous step as shown below.

    spark.metrics.conf /<Product_Home>/repository/conf/analytics/spark/metrics.properties
  4. Add the following configuration under the #Network in the <DAS_HOME>/repository/conf/analytics/spark/spark-defaults.conf file.

    spark.driver.extraJavaOptions -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8090 -Dcom.sun.management.jmxremote.rmi.port=8091 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=localhost
  5. Start the DAS server(s).

Viewing information in the J Console

Once you have done the required configurations to enable Spark monitoring via JMX as instructed in Setting up JMX to monitor Spark, you can view information relating to Spark performance as described in the WSO2 Product Administration Guide - JMX-Based Monitoring.

If your DAS deployment is a cluster, you can view information for the following additional process:

  • org.apache.spark.executor.CoarseGrainedExecutorBackend: This displays information related to the Spark worker JVM.
  • No labels