Oracle Coherence Monitor User Guide

Oracle Coherence Monitor

User Guide

JMX Connection Choices
The RTView OC Monitor application collects capacity and performance metrics from an operational Coherence Cluster using standard JMX protocols. These metrics are made available to developers and support personnel for analysis and alerting using RTView desktop applications, Web browser clients, or passively via event-triggered alerts.

There are several modes by which the OC Monitor may connect to a Coherence cluster using JMX. With RTView, users have a choice as to which mode to use, either of which may be relevant or appropriate depending on the monitoring requirement. This is especially important in a situation where users are called on to monitor and manage multiple disparate clusters.

Connection to Cluster Using JMX Remote Port or RMI URL
In this mode, the OC Monitor makes a connection to a remote JMX port or RMI URL exposed by a node in the cluster that has been configured as a Coherence “management” node on startup. This node must also have defined its JMX remote port or RMI URL using standard JMX configuration properties and may include a requirement for secure user authentication.

Once connected, the OC Monitor begins querying all (or a subset) of the MBeans from the Coherence management node at a regular interval.

NOTE: The management node may exist on the same machine as the OC Monitor; the “remote” designation simply means that the JMX connection is made to MBeans instanced in a separate process from the OC Monitor.

The information required for the OC Monitor to connect in this manner is minimal, only the host and port, or RMI URL. Typically, this makes it quick and easy to begin monitoring a cluster, a particular advantage in development environments where clusters come and go on a regular basis. There is no need to configure, then start and stop an agent in order to monitor the cluster.

Another advantage of remote JMX collection is that you do not have to install anything in the cluster or in a production environment – often the cluster itself is running behind a firewall and the monitor does not have easy access to the data. As long as a management node in the cluster exposes JMX MBeans, the connection process can be completely hands off.

A third advantage to this mode is that the OC Monitor makes no Coherence API calls, meaning that there is a next-to-zero chance of corrupting or crashing the cluster through improper configuration. The rate at which the JMX data are queried can be easily tuned so as to put a minimal monitoring load on the management node in the cluster and on the cluster itself.

Additionally, by having a Coherence management node in the cluster, it can act as backup in case the monitoring system itself goes down.

One disadvantage of the remote JMX connection is that its performance can degrade as the number of monitoring MBeans grows with the complexity of the cluster. A simple measure of cluster complexity is the product of number of nodes (N) times the number of caches supported by the cluster (C). Practical experience has shown that a cluster consisting of 150 nodes and 10 caches (N * C = 1500) can be adequately monitored using the remote JMX connection. Clusters larger than this can benefit from the direct connection mode described in the next section.

Clusters larger than this can benefit from the JMX Tables mode or from the Direct Connection mode. The JMX Tables approach has higher performance than the raw JMX approach, but requires custom MBeans to be deployed in the Coherence cluster. The Direct Connection approach has higher performance than JXM Tables but has tradeoffs in the form of access to all of the important cluster configuration parameters, and having the OC Monitor join the cluster as a management node.

Optimizing Data Retrieval Using JMX Tables
An option is available to speed up retrieval of Coherence MBean information (over JMX) by providing the aggregated MBean data in tabular form by using custom MBeans. By using custom MBeans the data is aggregated within the cluster and transmitted in the form of tabular data, rather than as individual attributes. This reduces the time taken to query the data.

This option is useful when monitoring large clusters (clusters with a large number of nodes, caches and/or services) using JMX, where the volume of data retrieved can affect the time taken to retrieve all the data, and thus limit the sampling rate for monitoring data.

Enabling this requires (unlike default JMX monitoring) that the custom MBeans (contained in a jar) are deployed and registered on all nodes in the cluster, and the monitoring is configured to query the custom MBeans.

The Oracle Coherence Documentation describes registering custom MBeans in a declarative manner in detail: https://docs.oracle.com/cd/E18686_01/coh.37/e18682/custom_mbeans.htm#COHMG4712.

To use this option:

Configure the monitored Coherence cluster to use JMX Tables custom MBeans.
- Add the ocjmxtables.jar to the classpath of the cluster members.
- Set -Dtangosol.coherence.mbeans=/sl-custom-mbeans.xml for the cluster members JVM's.
Configure your OC Monitoring system to use JMX Tables .
- Configure your monitoring system to use JMX as normal.
- Edit the rtview.properties file to use the maincollector.sl.rtview.cmd_line=-ocjmxtables property for the OCM monitoring system.

Requirements:

The Custom MBeans must be found at run time. You must place the library that contains the MBeans in the classpath of the Coherence nodes/members, including the JMX management-enabled member.
The custom MBeans (contained in a jar) must be deployed and registered on all nodes in the cluster, and the monitoring configured to query the custom MBeans.
The Custom MBeans must be specified using a MBean Configuration Override File.
The Custom MBeans (CacheTable, ServiceTable, StorageManagerTable) are contained in the jar ocjmxtables.jar, located in the rtvapm/ocmon/lib directory of the OC Monitor installation. This jar file must added to the classpath of the Coherence members to be monitored. This may require that the jar be copied to a location that is visible to all the Coherence members. This may vary based on your deployment. It may prove convenient to copy it to where the Coherence jars are deployed, so they can use the same classpath root.
The tangosol.coherence.mbeans system property specifies an MBean configuration override file to be used instead of the default custom-mbeans.xml override file. The MBean configuration file to use is sl-custom-mbeans.xml, contained at the root of the ocjmxtables.jar. Thus when the ocjmxtables.jar is added to the Coherence members classpath, it can be specified by setting the tangosol.coherence.mbeans system property for the Coherence cluster members to reference it thus: -Dtangosol.coherence.mbeans=/sl-custom-mbeans.xml
The above should be applied to all Coherence cluster members so that the tangosol.coherence.mbeans system property is set to /sl-custom-mbeans.xml.

If you have configured your Coherence cluster correctly, you should be able to connect to the cluster using JConsole, and see in addition to the previous Cache, Service, and StorageManager MBeans the new custom CacheTable, ServiceTable, and StorageManagerTable MBeans.

After you configure your OC Monitor system to use the Custom MBeans and configure your monitoring system to use JMX as normal, uncomment the following line in the rtview.properties file:

# JMX TABLES
#
# Uncomment the line below to use the JMX tables custom mbeans
#maincollector.sl.rtview.cmd_line=-ocjmxtables

This sets the -ocjmxtables command line argument to be passed to the maincollector program (typically this is the Data Server), and the log file will then contain the following text at startup:
... using OC JMX Tabular Data

And at runtime, the previous JMX queries (as seen in the JMX Metrics Administration display in the MBean Query Key column of the RTView JMX Query Statistics table):

* Coherence:type=Cache,* 0 * -1 *-
* Coherence:type=Cluster 0 * -1 *-
* Coherence:type=Service,* 0 * -1 *-

become the following:

* Coherence:type=CacheTable,* 0 CacheTable -1 *-
* Coherence:type=ServiceTable,* 0 ServiceTable -1 *-
* Coherence:type=StorageManagerTable,* 0 StorageManagerTable -1 *-

The JMX queries should also have a reduced execution time leading to a reduced total (JMX Query) Execution time.

Direct Connection to Cluster as a Coherence Management Node
In this mode, the OC Monitor itself joins the cluster and establishes itself as a management node. As a management node, it is configured with local data storage disabled so that it does not store any cache data and serves only as a monitoring node. In this role, it creates the JMX MBean server in-process and collects JMX monitoring data from other Coherence node using fast internal Coherence protocols.

The primary advantage for this mode is speed. In practice, this performance improvement can range from 2 to 10 times faster, depending a number of factors, in particular the network configuration environment.

However, there are tradeoffs. In order to use the direct connection mode, one must have access to all of the important cluster configuration parameters that are used by other nodes in the cluster. These include the Coherence override file, or specific settings like cluster name, well-known address, multicast ports, and Coherence mode. Having limited access to this information can make the configuration process time-consuming.

RTView contains components licensed under the Apache License Version 2.0.

JMS, JMX and Java are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. They are mentioned in this document for identification purposes only.

SL, SL-GMS, GMS, RTView, SL Corporation, and the SL logo are trademarks or registered trademarks of Sherrill-Lubinski Corporation in the United States and other countries. Copyright © 1998-2015 Sherrill-Lubinski Corporation. All Rights Reserved.