Splunk Data Source

Splunk is a data source used in conjunction with RTView application monitoring solutions.

Splunk (www.splunk.com) offers the ability to perform high performance search queries on log files and then makes it easy to grab content out of a log file that is applicable to application performance or status. That information can then be fed into RTView dashboards, reports and alerts as any standard RTView data adapter.

The Splunk data source allows you to access data in local or remote Splunk servers and use that data in RTView displays. Within the Splunk Web interface you can define various filters, such as time range filters, embed them in the search and then copy and paste these search definitions directly into the “Attach to Splunk Data” dialog.

If you have a large amount of various log files containing complex information that might require specialized parsing to gather pertinent information, Splunk can greatly enhance access to and maintenance of this important source of application performance data.

This section includes:

§ “Splunk System Requirements” on page 644

§ “Attach to Splunk Data” on page 644

§ “Application Options - Splunk” on page 650

§ “RTView Deployment - Splunk” on page 652

Splunk System Requirements

See the README_sysreq.txt file in your installation’s home directory for the current Splunk version(s) supported.

Note: If you are using Windows, make sure JAVA_HOME system variable is set to the directory where the JDK is installed.

Attach to Splunk Data

Note: The Splunk data source may not be licensed in your RTView installation.

Right-click on the Property Name from the Object Properties window and select Attach to Data>SPLUNK to display the Attach to Splunk Data dialog, which is used to search the server in the defined Splunk connection and filter the results. The Attach to Splunk Data dialog provides several drop down menus that allow you to specify information. If the drop down menu does not contain the item you require, type your selection into the text field.

Field Name	Description
Connection	Connection name. You may define a connection on the “Splunk Connections Tab” of the Application Options dialog.
Search	Enter a search. Note: It is possible to copy and paste a search query directly from the search bar at the top of the Splunk Web dashboard into the Attach to Splunk Data dialog. Refer to your Splunk Documentation for further information on searches, including search syntax and how to define various filters and display them outside of the search.
Max Results	Set the maximum number of rows of data to be returned from the Splunk server. If the search results in fewer rows than specified, only those rows of data are returned. Default is 100. Note: If value is set to 0, then all available results are returned. This value should be used with caution as it is possible for a large amount of data to transferred from the Splunk server.
Earliest Time	Filters data by timestamp. Only objects that have a timestamp greater than or equal to earliest_time will be returned. If left blank, no limit will be set on the earliest object returned. Use in conjunction with Latest Time to return objects within a specific time range. Note: Time format is always ISO-8601 formatted. For example: 2014-06-05T00:00:00-0800 2014-06-06T00:00:00-0800
Latest Time	Filters data by timestamp. Only objects that have a timestamp less than latest_time will be returned. If left blank, no limit will be set on the latest object returned. Use in conjunction with Earliest Time to return objects within a specific time range. Note: Time format is always ISO-8601 formatted. For example: 2014-06-05T00:00:00-0800 2014-06-06T00:00:00-0800
Server Column(s)	Select which columns to search in the Splunk server. To bring up the Select Server Column(s) dialog, click on the ellipses button in the Server Column(s) field or right-click in the Server Column(s) field and click Select Server Column(s). This dialog should contain a list of Available Column(s) you can add to your search. See “Select Server Column(s) and Select Column(s)” for more information. Note: If this field is left blank, all columns in the Search will be returned.
Offset	Starting offset of the first object in the list of data returned from the Splunk server. Use in conjunction with Max Results to parse a large set of returned data, one small section at a time. For example, with Max Results set to 100 and Offset set to 0, you would get search results with a _serial column value of 0-99. Or with Offset set to 10, you would get search results with a _serial column value of 10-109. In both cases you would get a subset of 100 results.
Divide Data Column	Select to divide up data in a returned column, via Java Regular Expression, into numbered columns. Typically this is used to extract data that has an implicit token or field based positional location in the raw data. For example, to break up raw log line/HHTP request into fields/columns.
Divide Column Name	Name of the column to split into into individual numbered columns that contain matched occurrences of the selected Java Regular Expression, where the new column name will be the number of the occurrence (e.g. "1", "2", "3", etc). In most cases, the Divide Column will be the raw data column (i.e.: _raw) returned from the Splunk server.
Column Data Regexp	Enter or select the Java Regular Expression (Regexp) desired to split the data in the specified Divide Column. The drop down menu contains a number of predefined regular expression literals or you can enter your own. Note: Normally Splunk will recognize and extract fields and make them available for searching on, but sometimes you need to split up the raw data to extract meaningful information.
Column(s)	Select which columns to display. To display the Select Column(s) dialog, click on the ellipses button in the Column(s) field or right-click in the Column(s) field and click Select Column(s). This dialog should contain a list of Available Column(s) you can add to your table. See “Select Server Column(s) and Select Column(s)” for more information.
Filter Rows	Check box to indicate whether or not to filter rows. See “Row Filtering” for more information.
Filter Column	Name of the column to use as a filter. Multiple column names should be entered as a semicolon (;) delimited list (i.e. col1;col2;col 3). If your column name contains a space or a semicolon, then the entire name must be enclosed in single quotes.
Filter Value	Value that the Filter Column must equal. Multiple filter values should be entered as a nested list, where values for a given column are separated by commas within a semicolon (;) delimited list (i.e. val1,val2;val3,val4;val5,val6). If your filter value contains a space or a semicolon, then the entire value must be enclosed in single quotes. When * is entered as a filter field value, data for all values in the specified filter column will be used to update the object property. When "*" is entered, only the literal comparative value will be used. These are only allowed for objects which display tabular data.
Update Mode	Specify which mode to use to update your Splunk connection:
	Every Default Poll Interval - Update connection each Default Poll Interval. See “Application Options - Splunk” for information on setting the Default Poll Interval. This is the default Update Mode.
	Every Poll Interval - Update connection each Poll Interval. If this option is selected, you must specify a Poll Interval.
	Poll On Demand - Update connection each time a display that uses this data attachment is opened and each time a substitution string that appears in the data attachment has changed.
	Poll Once (Static Data) - Poll for data only once. Select if the data returned by this attribute or operation is static.
Poll Interval	Specify the interval (in seconds) to update your Splunk connection. This option is only available if the Update Mode selected is Poll Every Poll Interval. See “General Tab” for more information. Note: Because the Poll Interval is superseded by the General Update Period, the amount of time elapsed between updates may be longer than the value entered. For example, if the General Update Period is 2 seconds and the Poll Interval is 5 seconds, the connection will be updated every six seconds.
Data Server	Select to read data through your configured Data Server and not directly from the Splunk data source.
	Default - Select the default Data Server you configured in the Application Options>“Data Server Tab”.
	None - Bypass data being redirected through the specified data server(s) for this attachment and instead attach directly to the data source.
	Named Data Servers - Select a Named Data Server that you configured in the Application Options>“Data Server Tab”.
	Multi-Server Attachment - To configure multiple data servers, enter a semicolon (;) delimited list containing two or more Named Data Servers (e.g. ds101;ds102). Each name specified must correspond with a Named Data Server that you configured in the Application Options>“Data Server Tab”. It is also possible to specify __default and __none (e.g. __default;ds101;ds102). Note: The values __default and __none begin with two underscore characters.
	Alternatively, a value of * can be entered to specify all data servers, including __default and __none. When multiple data servers are specified, the data attachment will be directed to each data server in the list. For tabular data attachments, a column named DataServerName will be added as the first column of the table and contain the name of the server from which the data was received.
	A multi-server attachment will receive data independently from each of the servers it specifies, so in most cases it will be necessary to combine the tables received into a single table. This can be accomplished in two ways: 1. The multi-server attachment can be applied to a local cache that has the DataServerName column specified as an index column. The current table of that cache will contain the combination of the tables received from all servers. Note: It may also be necessary to configure cache row expiration settings to remove defunct rows. 2. The multi-server attachment can be applied as the Table argument of the RTView function named Combine Multi-Server Tables. See “Tabular Functions” for more information.

When an object property has been attached to data, the Property Name and Value in the Object Properties window will be displayed in green. This indicates that editing values from the Object Properties window is no longer possible. To remove the data attachment and resume editing capabilities in the Object Properties window, right-click on the Property Name and select Detach from Data. You will recognize that an object property has been detached from the data source when the Property Name and Value are no longer green.

Validation Colors

Fields in the dialog change colors according to the information entered. These colors indicate whether or not information is valid. Information entered into the dialog is validated against the selected server.

Note: Filters are not validated.

Blue	Unknown	Cannot validate entry.
White	Valid state	Entry is valid.
Red	Invalid state	Incomplete or invalid entry.

Substitutions

The Substitutions feature allows you to build open-ended displays in which data attachments depend on values defined at the time the display is run. For example, a generic substitution value such as $MATCH could be used in a search. Later, when the display is running, this generic value is defined and used to control which records are matched on the server. In this way, a single display can be reused to show data from a number of different sources. For more information on creating displays using substitution values, see “Substitutions”.

Select Server Column(s) and Select Column(s)

From the Attach to Data dialog you can specify which columns to search in the Splunk server as well as which columns to display in your resulting data and in what order they will appear. To view the list of Available Column(s), click on the ellipsis button to open the Select Server Column(s) dialog the or Select Column(s) dialog.

To add a column, select an item from the Available Column(s) list and click on the Add button. If the item you require is not listed, type your selection into the Enter Column field. Click the Remove button to delete an item previously added to the Selected Column(s) list. You can control the order of fields in a table by arranging the items in the Selected Column(s) list with the Move Up and Move Down buttons.

Validation colors indicate whether selected columns are valid. However if even one column selected is invalid, the Server Column(s) or Column(s) field in the Attach to Splunk Data dialog will register as an invalid entry.

If no data is available for a table row within a selected column, the table cell will display one of the following values: N/A, false, 0, or 0.0.

The following describes Attach to Splunk Data dialog commands:

Command	Description
OK	Applies values and closes the dialog.
Apply	Applies values without closing the dialog.
Reset	Resets all fields to last values applied.
Clear	Clears all fields. Detaches object from data source (once Apply or OK is selected).
Cancel	Closes the dialog with last values applied.

Application Options - Splunk

Select Tools>Options in the Display Builder to access the Application Options dialog. Options specified in the Splunk Connections tab can be saved in an initialization file (SPLUNKOPTIONS.ini). On startup, the initialization file is read by the Display Builder, Display Viewer, Display Server, Data Server, and Historian to set initial values. If no directory has been specified for your initialization files and SPLUNKOPTIONS.ini is not found in the directory where you started the application, then RTView will search under lib in your installation directory. See “RTV_JAVAOPTS” for more information.

Splunk Connections Tab

This tab allows you to add or remove connections and set your default connection. When you add a Splunk connection to the list it will be highlighted in yellow indicating that RTView has not connected to it. To attempt to connect to a Splunk connection, click OK, Apply, or Save. If the background remains yellow, then RTView was unable make a connection. Check that your Splunk connection was setup correctly and that the Splunk server is running.

Note: Regardless of which tab you are currently working from in the Application Options dialog, each time you click OK, Apply, or Save, RTView will attempt to connect to all unconnected connections.

Field Name	Description
Default Connection	Name of connection used as the default for data attachments. Select from drop down menu to change default setting.
Add Connection	Click to open the Add Splunk Connection dialog. To edit, select a connection from the list and double-click. Connections that are updating objects in a current display cannot be renamed.
	Connection Name - Name to use when referencing this Splunk connection in your data attachments. This should be a unique name used to identify the connection to your Spunk server (splunkd) along with associated account details:
	Host - Host name or IP address
	Port - Port number
	User Name - User name
	Password - Password. If you need to provide an encrypted password (rather than expose server password names in a clear text file, use the encode_string command line option with the following syntax:
	encode_string type mypassword where type is the key for the data source and mypassword is your plain text password. Note: The type argument is only required when you encrypt a string for a data source. For example, enter the following in an initialized command window (see “Initializing a Command Prompt or Terminal Window”): encode_string splunk mypassword and you will receive an encrypted password: encrypted value: 013430135501346013310134901353013450134801334 Copy the encrypted value, paste it into the password field and click Save to save this value to the initialization (.ini) file. Or, if necessary, manually edit the (.ini) file to include the encrypted value. Note: If you need to manually edit a configuration (*.ini) file, contact SL Technical Support at support@sl.com for information about supported syntax.
Remove Connection	Select a connection from the list and click Remove Connection to delete. Connections that are updating objects in a current display cannot be removed.

Splunk Options Tab

This tab allows you to set a wait time for polling search requests and a default poll interval to update your Splunk connection.

Field Name	Description
search job wait time (ms)	Enter the time in milliseconds to control the wait time between polls for search data until a search job is complete. Default is 1000.
Default Poll Interval (Seconds)	Enter the time in seconds to control how often your Splunk connection is updated or operations in data attachments are executed if no Poll Interval is specified in the data attachment. Default is 0, which updates connections and operations according to the General Update Period specified in Application Options on the “General Tab”.
	Note: Because the Default Poll Interval is superseded by the General Update Period, the amount of time elapsed between updates may be longer than the value entered. For example, if the General Update Period is 2 seconds and the Default Poll Interval is 5 seconds, the connection will be updated every six seconds.

RTView Deployment - Splunk

This section contains details about the deployment process that are specific to your data source. Please go to the Deployment section of this documentation for instructions on how to implement your RTView deployment option. Return to this page whenever you are instructed to refer to deployment information that is specific to your data source.

System Requirements and Setup

The Splunk data source has additional System Requirements and Setup. See “Splunk System Requirements” for more information.

Data Source Configuration File

RTView saves general application settings as well as data source configuration options in initialization files that are read at startup. If no directory has been specified for your initialization files and files are not found in the directory where you started the application, then RTView will search under lib in your installation directory. See “Application Options”, “Application Options - Splunk”, and “RTV_JAVAOPTS” for more information.

Include the following initialization file when you deploy RTView with this data source:

File Name	Description
SPLUNK.ini	Contains data source options for Splunk.

Note: Options specified using command line and applet parameters override values set in these initialization files.

Rich Client Browser Deployment Setup for Direct Data Connection

This deployment is supported by your data source. No additional client setup is required.