acktomic.com

Making use of time series trending for network security monitoring. A short history with examples.

Time series trending tools like Cricket, Cacti and Torrus focus on performance and availability.

Operational teams use these day in day out, the tools are there to organize and display reams of data in a quick and painless way.Time series trending is a source of indicators.

Strengths of time series trending

• detailed baseline
• displays seasonality
• highlights anomalies
• illustrates subtle changes
• provisioning planning

Strengths of time series trending using RRD databases

• fast visualization - anyone having had to suffer SQL based trending tools can attest
• low maintenance
• graphing flexibility
• capabilities can be extended
• no vendor lockin or annual maintenance fees

Making use of these strengths to identify threats is a secret recipe. Yeah, reaaally.

Here are some DNS based trends that can help quantify, understand and defend or cleanup against extrusion attemps or malware/botnet command and control communications.

• Trending the number of hits against security related blacklisted entries
• Trending the number of hits against .cn, .ru, etc. specific domains
• Trending the number of hits against honeypot entries
• Trending the number of hits by source IP

The data value can be extended by doing a bit of munging on the interesting bits.

The information could also be reported in dashboards on a Splunk server for operational teams or management. Trending data over the long term also unshackles the administrator from defining hard limits in time of day use for what is normal and what is not. You can still define hard limits which can generate security events, but analysis is made much easier by seeing the whole time series.

Once identified the vector of the outbreak needs to be cleaned before nasty malware can move in (think Zeus variant).

RRDs are fixed size databases for storing time series data. They collect information given to them and normalize it to permit trending over long periods of time.

Spurious data may inadvertently make it’s way into a database. Treating this data is possible using the following means:

• Set the rrd-min and/or rrd-max variable(s) for each datasource when creating new RRD databases
• Use rrdtool dump to export the RRD database to XML format, edit out the spurious values and import the data back into the RRD database
• Use rrd tune to apply rrd-min and/or rrd-max variable(s) to an existing RRD database. All values outside the minimum or maximum defined bounds will be set to NaN.

rrdtool tune <file> --maximum <ds>:<value>

• Use the perl script removespikes.pl. This would remove all spikes within 1% of the datapoints in the rrd file. If 1% does not fix them, modify the % value up until all the spikes are removed. Though this may eat up some valid values in the process, use with caution!

perl removespikes.pl -l 1 fastrouter_ethernet0_1.rrd

• Use rrd_editor, a cross platform win32 or perl/tk tool to seek and remove spikes in an RRD. I have not used the tool, but according to comments it works as advertised. It also lets you easily add or remove RRAs and datasources from an RRD, which is a golden feature for many of us.
• Use killspike2 an RRD spike removal script distributed as part of the Cricket network management system. I have not used the script, but it is known to work.

As with any solution, automation and prevention are the keys to a fluid system.

genDevConfig will automatically set rrd-min and rrd-max values for all config-tree targets it creates for Cricket.

Download the latest release candidate version of genDevConfig.

genDevConfig 2.0 RC2

Or, download the latest stable version of genRtrConfig.

genRtrConfig 1.5.50

To learn more about the Cricket configuration generator for SNMP managed devices, please consult the genDevConfig reference manual.

To learn more about high performance trending of time series using Cricket, please visit cricket.sourceforge.net.

Also of interest, an older configuration generator project, CHIRP, has support for some equipment that is not currently fully supported in genDevConfig (Fore ASX, NetScaler, HP AdvanceStack, Riverstone, Foundry BigIron). The perl modules for those device classes included in the CHIRP tool can be easily converted to genDevConfig modules. See the genDevConfig reference manual linked above.