This section describes how to use Java tools to troubleshoot RTView applications.
SL recommends JDK v1.6 on systems running RTView server applications. JDK v1.6 includes tools such as “JConsole”, “Jmap Utility”, and JPS (which are not included with the JRE - see “Linux”). These tools are not required but make it easier to collect more complete information for troubleshooting. Documentation for the tools are located at:
http://www.oracle.com/technetwork/java/javase/tools6-unix-139447.html.
This section assumes you are using JDK v1.6.
Several of the steps in this section require an RTView application process ID (PID). To obtain a PID, see “Obtaining the RTView PID”.
There are several Java tools you can use to check CPU and memory usage: “JConsole” and the “Jmap Utility”.
Use the JConsole tool, included with JDK, to monitor CPU and memory usage of a Java application.
To use the JConsole:
Use the -jmxport option to specify the port number on which RTView accepts a JConsole connection. For example:
run_displayserver -jmxport:6789 ...
JConsole is then connected to the RTView application as:
jconsole localhost:6789
Note: -jmxport must specify a unique port number for each RTView application on a single host. If no jmxport was specified when the RTView application was started, connect JConsole by specifying the RTView application PID: jconsole PID
The JConsole opens.
View the following in the JConsole:
|
Memory Tab |
View application heap usage trends. Usage might go up and down between garbage collection intervals but, overall, the heap size should be stable. An exception to this is an RTView server that runs the RTVIew cache data source. In that case, heap usage grows until each cache history table reaches its maximum row counts or timeSpan. See “Caches” for information. |
|
MBeans Tab |
See the RTView node which contains data generated by RTView applications. See the RTViewHistorian node, which is the base class for the Historian, the Data Server, and the Display Server. Also see the java.lang node, which contains standard Java application JMX metrics. § VersionInfoDetails - Use this MBean for reporting problems to SL Technical Support. This table contains detailed information about the version of each jar used in an RTView application. The table contains the following columns: ApplicationName - The name of the application. For example, RTView Data Server. ApplicationConfiguration - The configuration string for the application. This string contains the main application version that corresponds to the version information that is printed to the console at startup. JarName - The name of the jar. JarConfiguration- The configuration string for the jar. JarVersionNumber - The version number for the jar. JarVersionDate - The version date for the jar. JarReleaseType - The release type for the jar. JarMicroVersion - The micro version for the jar. |
Note: JConsole v1.6+ has a Detect Deadlock button on the Threads tab. Click this if the server has become unresponsive and a thread deadlock is suspected.
To use the JConsole:
Use the Jmap utility, included with JDK, to measure heap memory usage as follows:
jmap -histo:live PID > heap.txt
where PID is the RTView application process ID.
This produces a memory usage histogram. If a memory leak is suspected, run Jmap several times, several minutes apart (or hours apart, depending on the rate of growth), with the output sent to different files (specified by the file option) for comparison.
To pinpoint memory leaks, produce a complete memory dump of the application, as follows:
jmap -dump:live,format=b,file=heap.dump PID
The JDK Jhat utility is used to analyze the dump files. The Jhat analysis can be performed at SL.
Note: The dump file can be very large. Again, if a memory leak is suspected, Jmap should be run several times several minutes (or hours, depending on the rate of growth) apart with the output sent to different files (specified by the file option) for comparison.
Check Application and CPU Bottlenecks
Use thread dumps to identify application deadlocks and CPU bottlenecks.
A thread dump of a Java application shows a stack trace of the methods being executed by each thread. Take multiple thread dumps of the application: take one when the application is healthy and behaving as expected, and three or four additional thread dumps, about 1 minute apart, when the application is not performing as expected.
You can produce thread dumps on “Linux”, “Windows”, and “JDK 1.6”.
To produce a thread dump:
Use kill -3 PID where PID is the application process ID.
The thread dump is sent to the application standard output stream. Or, if that stream has been redirected to a log file, the thread dump is sent to that file.
If the application was started with a custom script (and not the standard run_*.bat script), type Ctrl-C in the application console window.
If the application was started with the standard run_*.bat script the thread dump only shows the threads for the RTView launcher application (which is not useful).
Use jstack PID > threads.dump. This sends the thread dump to the specified file.
Each thread dump begins with the following:
2011-01-29 14:19:28
Full thread dump Java HotSpot(TM) Client VM (...version info ...)
Windows
Open the Windows Task Manager. If several Java applications are running on the same host, examine the command-line arguments for each in order to distinguish between them. Use the check box in the Windows Task Manager View / Select_Columns menu.
Note: This feature is not available on Windows XP.
Note: The Display Server, Data Server, and Historian command lines look very similar. In RTView 5.8+, there is a PROCESS_NAME argument that is unique for each of those applications.
Use top or ps. If the JDK is installed, jps -v can be used on either operating system. The JPS tool only lists Java applications. If several Java applications are running on the same host, examine the command line arguments for each in order to distinguish between them. Consult the man pages for top and ps, if those utilities do not show the command line by default.
Note: The Display Server, Data Server, and Historian command lines look very similar. In RTView 5.8+, there is a PROCESS_NAME argument that is unique for each of those applications.