This documentation is for older WSO2 products. View documentation for the latest release.
Page Comparison - Troubleshooting in Production Environments (v.4 vs v.5) - Clustering Guide 4.2.0 - WSO2 Documentation

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Find the thread ID (the one that belongs to the corresponding PID) that takes up the highest CPU usage by examining the thread-usage.txt file.

    Code Block
    %CPU CPU  NI S     TIME   PID   TID
    .......... 
      0.0   -   0 S 00:00:00  1519  1602
      0.0   -   0 S 00:00:00  1519  1603
     24.8   -   0 R 00:06:19  1519  1604
      2.4   -   0 S 00:00:37  1519  1605
      0.0   -   0 S 00:00:00  1519  1606
    ..........

    In this example, the thread ID that takes up the highest CPU usage is 1604.

  2. Convert the decimal value (in this case 1604) to hexadecimal. You can use an online converter to do this. The hexadecimal value for 1604 is 644.
  3. Search the thread-dump.txt file for the hexadecimal obtained in order to identify the thread that spins. In this case, the hexadecimal value to search for is 644. The thread-dump.txt file should have that value as a thread ID of one thread.
  4. That thread usually has a stack trace, and that's the lead you need to find the issue. In this example, the stack trace of the thread that spins is as follows.

    Code Block
    "HTTPS-Sender I/O dispatcher-1" prio=10 tid=0x00007fb54c010000 nid=0x644 runnable [0x00007fb534e20000]
       java.lang.Thread.State: RUNNABLE
            at org.apache.http.impl.nio.reactor.IOSessionImpl.getEventMask(IOSessionImpl.java:139)
            - locked <0x00000006cd91fef8> (a org.apache.http.impl.nio.reactor.IOSessionImpl)
            at org.apache.http.nio.reactor.ssl.SSLIOSession.updateEventMask(SSLIOSession.java:300)
            at org.apache.http.nio.reactor.ssl.SSLIOSession.inboundTransport(SSLIOSession.java:402)
            - locked <0x00000006cd471df8> (a org.apache.http.nio.reactor.ssl.SSLIOSession)
            at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
            at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160)
            at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342)
            at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320)
            at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
            at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
            at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604)
            at java.lang.Thread.run(Thread.java:722)

Capturing the state of the system

Carbondump is a tool used to collect all the necessary data from a running WSO2 product instance at the time of an error. The carbondump generates a ZIP archive with the collected data that helps to analyze the system and determine the problem that caused the error. Therefore, it is recommended that you run this tool as soon as an error occurs in the Carbon instance.

When using the tool, you have to provide the process ID (pid) of the product instance and the <PRODUCT_HOME> location, which is where your unzipped Carbon distribution files reside. The command takes the following format:

Code Block
sh carbondump.sh [-carbonHome path] [-pid of the carbon instance]

For example,

Code Block
In Linux: sh carbondump.sh -carbonHome /home/user/wso2carbon-3.0.0/ -pid 5151

In Windows: carbondump.bat -carbonHome c:\wso2carbon-3.0.0\ -pid 5151

The tool captures the following information about the system:

  • Operating system information** OS (kernel) version
    • Installed modules lists and their information
    • List of running tasks in the system
  • Memory information of the Java process** Java heap memory dump
    • Histogram of the heap
    • Objects waiting for finalization
    • Java heap summary. GC algo used, etc.
    • Statistics on permgen space of Java heap
  • Information about the running Carbon instance** Product name and version
    • Carbon framework version (This includes the patched version)
    • <PRODUCT_HOME>, <JAVA_HOME>
    • configuration files
    • log files
    • H2 database files
  • Thread dump
  • Checksum values of all the files found in the $CARBON_HOME