The following sections provide information on how to troubleshoot various problems that may arise for deployment in production environments.
Analyzing a stack trace
When your Java process starts to spin your CPU, you must immediately analyze the issue using the following two commands and obtain the invaluable information required to tackle the issue. This is done based on the process ID (pid).
jstack <pid> > thread-dump.txt
ps -C java -L -o pcpu,cpu,nice,state,cputime,pid,tid > thread-usage.txt
Tip: OS X users can alternatively use the command
ps M <PID>
instead.
These commands provide you with the thread-dump.txt file and the thread-usage.txt file. After obtaining these two files, do the following.
Find the thread ID (the one that belongs to the corresponding PID) that takes up the highest CPU usage by examining the thread-usage.txt file.
%CPU CPU NI S TIME PID TID .......... 0.0 - 0 S 00:00:00 1519 1602 0.0 - 0 S 00:00:00 1519 1603 24.8 - 0 R 00:06:19 1519 1604 2.4 - 0 S 00:00:37 1519 1605 0.0 - 0 S 00:00:00 1519 1606 ..........
In this example, the thread ID that takes up the highest CPU usage is 1604.
- Convert the decimal value (in this case 1604) to hexadecimal. You can use an online converter to do this. The hexadecimal value for 1604 is 644.
- Search the thread-dump.txt file for the hexadecimal obtained in order to identify the thread that spins. In this case, the hexadecimal value to search for is 644. The thread-dump.txt file should have that value as a thread ID of one thread.
That thread usually has a stack trace, and that's the lead you need to find the issue. In this example, the stack trace of the thread that spins is as follows.
"HTTPS-Sender I/O dispatcher-1" prio=10 tid=0x00007fb54c010000 nid=0x644 runnable [0x00007fb534e20000] java.lang.Thread.State: RUNNABLE at org.apache.http.impl.nio.reactor.IOSessionImpl.getEventMask(IOSessionImpl.java:139) - locked <0x00000006cd91fef8> (a org.apache.http.impl.nio.reactor.IOSessionImpl) at org.apache.http.nio.reactor.ssl.SSLIOSession.updateEventMask(SSLIOSession.java:300) at org.apache.http.nio.reactor.ssl.SSLIOSession.inboundTransport(SSLIOSession.java:402) - locked <0x00000006cd471df8> (a org.apache.http.nio.reactor.ssl.SSLIOSession) at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121) at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:160) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:342) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:320) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:604) at java.lang.Thread.run(Thread.java:722)