The latest version for DAS is WSO2 Data Analytics Server 3.1.0. View documentation for the latest release.
WSO2 Data Analytics Server is succeeded by WSO2 Stream Processor. To view the latest documentation for WSO2 SP, see WSO2 Stream Processor Documentation.

All docs This doc
||
Skip to end of metadata
Go to start of metadata

Introduction

Every time your Web server receives a request, it makes an entry to one or more log files. These log files are useful for a variety of purposes, from statistical analysis of your visitors to forensic analysis of an attack on your server. The HTTPD logs can provide information on everything that happens on your server from the initial request, through the URL mapping process, to the final resolution of the connection (including any errors that may have occurred in the process). Thereby, HTTPD logs provide you feedback about the activities and performance of the server as well as any problem that may be occurring, to effectively manage your Web server.

This HTTPD logs sample is intended to show the capability of WSO2 DAS which can analyze the raw HTTPD logs and produce useful results. It demonstrates how you can use logs to analyze the Web traffic that comes to your server from different regions. It calculates the region from the IP address in logs and visualizes it through the different mechanisms of presenting data in WSO2 DAS.

Prerequisites

Set up the following prerequisites before you start.

  1. Set up the general prerequisites required for WSO2 DAS.
  2. Copy the <DAS_HOME>/samples/udfs/org.wso2.das.samples.geoip-3.0.0-SNAPSHOT.jar file (which is the UDF library that is required for this sample), to <DAS_HOME>/repository/components/dropins/ directory.

  3. Add the following configurations within the <custom-udf-classes> element of the <DAS_HOME>/repository/conf/analytics/spark/spark-udf-config.xml file.

    <udf-configuration>
        <custom-udf-classes> 
    			......... 
    			<class-name>org.wso2.das.samples.geoip.IPCountryNameUDF</class-name>
    			<class-name>org.wso2.das.samples.geoip.IPCountryCodeUDF</class-name>
    			.........
    	</custom-udf-classes>
    </udf-configuration>

Building the sample

Follow the steps below to build the sample.

Uploading the Carbon Application

Follow the steps below to upload the Carbon Application (cApp) file of this sample. For more information, see Carbon Application Deployment for DAS.

  1. Log in to the DAS management console using the following URL: https://<DAS_HOST>:<DAS_PORT>/carbon/
  2. Click Main, and then click Add in the Carbon Applications menu.
  3. Click Choose File, and upload the <DAS_HOME>/capps/Httpd_Log_Analytics.car file as shown below.
    add a new cApp
  4.  Click Main, then click Carbon Applications, and then click List view, to see the uploaded Carbon application as shown below. 
    list of all available cApps

Executing the sample

Follow the steps below to execute the sample.

Running the data publisher

Navigate to <DAS_HOME>/samples/httpd-logs/ directory in a new CLI tab, and execute the following command to run the data publisher: ant

This reads the <DAS_HOME>/samples/httpd-logs/resources/access.log file, and sends each log line as an event to the event stream which is deployed through the above CApp.

Viewing the output

You may use the Data Explorer or the Analytics Dashboard of the WSO2 DAS Management Console to browse published sample events.

Using the Data Explorer

Follow the steps below to use the Data Explorer to view the output. 

  1. Log in to the DAS management console if you are not already logged in.
  2. Click Main, and then click Data Explorer in the Interactive Analytics menu.
  3. Select ORG_WSO2_SAMPLE_HTTPD_LOGS for the Table Name as shown below.
    select table in the Data Explorer
  4. Click Search. You view the published HTTPD logs as shown below.
    viewing published results in the Data Explorer

Using the Analytics Dashboard

Follow the steps below to use the Analytics Dashboard to view the output. 

  1. Log in to the DAS management console if you are not already logged in.
  2. Click Main, and then click Analytics Dashboard in the Dashboard menu.
  3. Log in to the Analytics Dashboard using admin/admin credentials.
  4. You view the HTTPD Log Analysis Dashboard which is already deployed through the CApp as shown below.
  5. Click View option of the Dashboard. It opens the HTTPD Log Analysis Dashboard with three gadgets in a new tab of your Web browser as shown below.
    full country based request summaryhigh traffic request countries
    low traffic request countries
  • No labels