Riemmann as a general purpose monitoring system for Microsoft Windows

Riemann is an up and coming event stream processor. It is one of those projects that you take a look at and think this is nice, but not ready yet. A year has passed since I made that observation. This is a project followed by high profile monitoring folks. This very dynamic system  has a similar philosophy as Graphite. Riemann filters, combines and acts on flows of events.

I like elegant. Riemann is very elegant, it acts as a stream processor like ESPER, but built with a monitoring angle to make it more intuitive and easy to work with. Riemann listens and acts, it does not have a scheduler like Shinken, Zabbix or OpenNMS. It depends on application libraries, clients, scripts or agents to forward data to it.

A lot of the things I have privately ranted about in Shinken, which don’t take me wrong,  I am a great fan of, are gracefully handled in Riemann. Notably the way rules are defined/managed, including tags as part of the event stream, full use of all cpus, not having to predefine all possible events to monitoring them, not interrupting monitoring flow  to load new rules, streamed updates to dashboards, flexible dependency model and more.

The open-source Swiss knife of the future is really shaping up with Riemann dealing with events, Graphite for time-series and logstash for logs (I am a fan of Splunk, but there is no current integration between the two).

For internal linux/UNIX, applications and cloud-type application monitoring Riemann is IMO a viable proposition with version 0.2. For general purpose monitoring, not so much for network devices and Windows.

What Riemann needs to succeed in general purpose monitoring

For it to succeed in general purpose monitoring I believe it needs :

• Integration with a solid Windows agent to receive streamed data (like statsd or diamond) but for Windows; update: Riemann-health for windows Seems like a good start, but modifying it requires to modify C# code, recompile and redeploy. :-/ Support for things like process monitoring, executing local commands, scripts and more
• Integration with a solid SNMP polling system that would use Riemann as its back end; Collectd is there, but alternatives would be welcome. Something like a stripped down version of Cricket that just sends data to Riemann and Graphite.
• Professional full time backing (Kyle Kingsbury, the main author just accepted a full time job…) Good for him, but so-so for the project. More developers are contributing to the project which is good. I could envision starting a company to implement Riemann based systems
• A dashboard element for displaying streamed text events/logs (On the road map, per Kyle K.)
• A dashboard element or alternate dashboard for displaying graphical status data. Based on vector graphic backgrounds and status widgets sitting on top of the image to show changes. Think of Nagvis and the plethora of industrial HMIs like Wonderware, OSIsoft PI processbook, etc.
• Ability to drill down into dashboards to view dependency relations (Think Big Brother/Xymon dashboard drill downs) Though this may be impossible simply due to the nature of how Riemann works?!
• A security model for the data acquisition and API
• Cookbook type recipes that can be used to to understand how a typical monitoring workflow might handle calculations, thresholds, rate limiting, dependencies(parent/child), state changes and finally output to graphite/request-tracker/email/pagerduty. Single examples are nice, but how does it *really* come together and how to maintain it.
• Having some place to store and retrieve textual events
• Role based controls for the dashboard
• An HA strategy

Side note: Riemann is a project with immense potential for lightweight industrial controls monitoring. Having a listening module for modbus or even OPC UA  would really light a fire under the archaic beasts that the industrial monitoring/controls suppliers are using. Ok, getting back to Windows.

Windows sources of events

Windows operating system status and events are typically monitored via these methods

1. Windows event log
   1. Can be handled via agents like NSCA++, OSSEC, SNARE, custom agents for SIEM/logging software which would need to be able to forward the data to Riemann in a second step
2. Windows PDH API (performance monitor counters)
   1. Handled by local agents (riemann-health-windows) or via WMI, but this becomes polling based monitoring
3. Custom event logs
   1. Can be handled via agents like NSCA++, OSSEC, another SNARE agent, custom agents for SIEM/logging software which would need to be able to forward the data to Riemann in a second step
4. External monitoring via the network of services
   1. Centralized polling from collectd could do the job but alternatives would be nice;
   2. Shinken/Munin/other as the active poller and then streaming events to Riemann to dashboard??
5. Scripted monitoring via a local agent
   1. Local agent, like ???

A great project needs a leader and contributors

This project has great potential and I sincerely hope that the main developer continues to work on it and that companies continue to contribute (money, resources, dev offers, etc.) to his effort to develop it.

OSSEC is a pretty nifty tool to collect, parse and categorize events. Though it sometimes feels like a party entering the second level of the Proving Grounds of the Mad Overlord. You feel like your are getting there but you know some crazy vorpal bunny is out for you. You might not be wrong..

Milwa my Regex

First, the regex library has the bare minimum using the following tricks will help along the way.

1. Use the after_parent, after_prematch inside your pattern match directives.
2. Use the your ^ $ anchors
3. The | (logical OR) is globally scoped and can’t be used within a field extraction ()
4. If you think of doing anything fancy, think again, it probably isn’t directly supported
5. If you bother to extract fields, use them in the rules.. It is more efficient and readable.

Dialma my Decoder

This is my list to help me get around to making good Decoders. I am by no means an expert in this stuff, but I will add more as I go along. The OSSEC book is great, but lacks in lighting some of the darker halls.

1. Decoded_as is really nifty, but cannot be used to reference child decoders
2. More than one decoder can be used for the same root decoder
   1. The child decoders can be used to deal with different events
   2. The child can also be used one after the other to deal with an event that may have different forms. Due to fields may not in the same place depending on the software version or features enabled.
   3. It also may be hard to create a single regex to extract different fields of the same event
3. If you have a commonality between events you should structure your decoders in a tree like manner
4. madirish.net has a great post on writing a new decoder step by step
5. You cannot arbitrarily create new fields, you have to work with what is there. (If anyone can show me the contrary I will be happy document how to add them (other than hacking the code)

Tiltowait the Rules

1.  If you wish to monitor a log file for which no decoder exists, add the path to the file in your ossec.conf and simply create a rule that does a <match>blah</match>.  Of course, this means, ALL your logs will hit his rule. So consider this a quick fix for a critical issue, that will need to be addressed down the line.
2. Use your decoded fields (as mentioned to the regex section)
3. If you have any rules that need to match against a list of things, use the lookup tables as described in the book and documentation. This comes up a lot on the mailing list, but then again, some people don’t read the documentation.

Dumapic for the human

The OSSEC server is the workhorse, but humans need a method to interact with the data from OSSEC, here are the IMO best choices for doing this at this current time and date. Pre-built dashboards, reports, field extractions, etc.

1. Splunk with the Splunk for OSSEC application
2. OSSEC Web UI

Some parsing and field extractions work may be needed for commercial offerings

1. Commercial SIEM tools (Batteries not included)

Base building blocks for search and reporting on the collected data, nothing pre-built.

1. Elastic search + Kibana
2. Logstash + Kibana

OSSEC out of the box does not include any decoders or rules for Palo-Alto Networks Next-Generation firewalls. This is part 2 of PaloAlto firewall integration with OSSEC. Read part one to learn how to create the basic decoder, a threat decoder and rules. In this part we will learn how to decode SYSTEM and CONFIG event logs.

Decoding Palo-Alto syslog messages

The general decoder for PaloAlto firewalls is defined in part one. Now we need to decode SYSTEM and CONFIG event logs. Each type of logs has different arguments, but they still follow the same comma delimited format. We will use the following two log messages for our decoders.

Mar 19 18:05:17 192.168.1.1 1,2013/03/19 18:05:17,0016A102121,
CONFIG,0,0,2013/03/19 18:05:17,192.168.1.120,,commit,username,Web,Submitted,,993,0x0

Mar 19 18:34:27 192.168.1.1 1,2013/03/19 18:34:27,0016A102121,
SYSTEM,general,0,2013/03/19 18:34:27,,general,,0,0,general,high,
System restart requested by username,1261490,0x0

Identify the types of event and extract fields

Add the following decoders to your $OSSEC/etc/local_decoder.xml file.

CONFIG events:

<!-- Custom decoder for PaloAlto Firewalls CONFIG Events -->
<decoder name="paloalto-config">
 <parent>paloalto</parent>
 <prematch offset="after_parent">^CONFIG,</prematch>
 <regex offset="after_parent">^(CONFIG),\.*,\.*,\.*,(\d+.\d+.\d+.\d+),\.*,(\w+),(\w+),</regex>
 <order>id,srcip,action,srcuser</order>
</decoder>

Extracted fields are based on what can help for policy and correlation rules. The decoder will extract the required information and fill the OSSEC variables.

• id of the event (CONFIG)
• Source IP
• Configuration action that was submitted
• Source user of the action

<!-- Custom decoder for PaloAlto Firewalls SYSTEM Events -->
<decoder name="paloalto-system">
 <parent>paloalto</parent>
 <prematch offset="after_parent">^SYSTEM,</prematch>
 <regex offset="after_parent">^(SYSTEM,\w*),\.*,\.*,\.*,\.*,\.*,\.*,\.*,\.*,(\w+),(\.*),\.*,</regex>
 <order>id,status,action</order>
</decoder>

Extracted fields:

• The id of system event (SYSTEM,sub-id)
• The severity/status of the event (informational, low, medium, high, critical)
• The action message including the username that generated the event

This last one is a bit problematic as both the message and the event are important, but they cannot easily be parsed, so we just group ‘em together.

Creating rules for CONFIG events

Configuration events are more unique and need rules to target events that are matched to policies.

Add the rules to your $OSSEC/rules/local_rules.xml file.

AUTHORS NOTE : Rules cannot use the decoded_as to refer to a child decoder. argh. Feature request to come… I have updated the rules to reflect this and use the id to include the event identifier (CONFIG, SYSTEM or THREAT)

<group name="syslog,paloalto,config,">
  <rule id="101200" level="3">
    <decoded_as>paloalto</decoded_as>
    <id>^CONFIG</id>
    <description>PaloAlto Firewall Config Event</description>
  </rule>

  <rule id="101201" level="3">
    <if_sid>101200</if_sid>
    <action>commit</action>
    <group>config_changed</group>
    <description>Configuration changed and commited</description>
  </rule>

  <rule id="101202" level="10">
   <if_sid>101200</if_sid>
   <time>7:00 pm – 8:00 am</time>
   <description>Configuration changed outside business hours.</description>
   <group>policy_violation</group>
  </rule>

Other interesting events to consider :

• Add/remove users
• Change administrative privileges
• Source IP doing the changes
• Expected user doing changes (and type of changes)

 Creating rules for SYSTEM events

System events have a severity, let’s start by classifying them by severity.

Add the rules to your $OSSEC/rules/local_rules.xml file.

<group name="syslog,paloalto,system,">

  <rule id="101100" level="3">
    <decoded_as>paloalto</decoded_as>
    <id>^SYSTEM</id>
    <description>PaloAlto Firewall System Event</description>
  </rule>

  <rule id="101101" level="8">
    <if_sid>101100</if_sid>
    <status>critical</status>
    <description>Critical system event!</description>
  </rule>

  <rule id="101102" level="3">
    <if_sid>101100</if_sid>
    <status>high</status>
    <description>High system event!</description>
  </rule>

  <rule id="101103" level="2">
    <if_sid>101100</if_sid>
    <status>medium</status>
    <description>Medium system event!</description>
  </rule>

  <rule id="101104" level="2">
    <if_sid>101100</if_sid>
    <status>low</status>
    <description>Low system event!</description>
  </rule>

  <rule id="101105" level="0">
    <if_sid>101100</if_sid>
    <status>informational</status>
    <description>Informational system event!</description>
  </rule>

Some events are categorized as medium with a level=2, but really we want to raise their level.

  <rule id="101106" level="8">
    <if_sid>101103</if_sid>
    <id>server-no-free-ip</id>
    <description>DHCP server ran out of IPs in pool</description>
  </rule>

We used one of the extracted fields , “id”, from the decoding run to speed up processing instead of doing a match.

Discussion on rules

Up to now this has been very rule based. We did introduce time based policy validation. It gets interesting when combining multiple indicators. This is where the extracted fields gain their power. Attackers are more sophisticated and it is only when combining the network view (connections, data flows, duration, data access, users, time based controls, etc.) that one can catch bad behaviour or attackers soon enough to be able to respond to the attack.

In our next part we will see how to create some more advanced behavioural rules for our PaloAlto boxes.

OSSEC currently does not include any decoders or rules for Palo-Alto Networks Next-Generation firewalls. I just happen to have access to one of these devices and wished to get some logs into OSSEC. A little search on the interwebs only turned up a single relevant link. Xavier Mertens wrote a quick blog post during the second edition of OSSEC week in 2010. Thanks Xavier for putting it out there and getting people to see how it can be done. So let’s go about making it a cleaner and more comprehensive example.

OSSEC is a LIDS that acquires, categorizes, filters, correlates and takes action on logs. It is very elegant in solving many thorny problems related to cross-platform lightweight secure log acquisition. It also integrates wonderfully with other tools for search (ElasticSearch, Splunk for OSSEC) or even full blown SIEM solutions.

Having used these firewalls for a while now (and free time away from reading vintage Dragon magazines) I wanted to get these things truly integrated with a security process. It pings, it forwards, it filters! The rules, filters, vulnerability/intrusion/threat detection and other nobs are tuned. This things is impervious as it can be now. All done, right? Not really…

Your mind wanders thinking about Kelly in accounting hmmm.. Sweet thought. All of a sudden, the office romance squirts out of mind and is replaced with a vague sense of unease. You remember something about security policy documents, best practices and internal auditor visits for one of those Pissy Eye or NERD compliance audits. oh yeah… answering questions like

• Who last changed something in the firewall
• Do threats get flagged to administrators
• Who tried to login to the firewall and succeeded (or failed)
• Have I just run out of IP addresses in the DHCP pool
• Did an internal client just port scan the firewall
• Did that internal client also do failed logins against another device
• Is a domain admin account surfing on the internet?

Logging in to the device and answering the questions is a fail. If it is not automated, the process is doomed to fail or be inconsistently followed. Lucky for us, OSSEC can do the hard work, keep the bosses (and auditors happy) and have the security admin on top of things.

Decoding Palo-Alto format syslog messages

The first step is to look at what type of log message would be generated by the device. Let’s use the log entry from the linked article.

Aug 26 13:56:07 192.168.0.5 1,2010/08/26 13:56:07,0003Z100245,THREAT,\
   vulnerability,8,2010/08/26 13:56:01,10.0.0.1,10.0.0.2,0.0.0.0,\
   0.0.0.0,rule2,domain\user,,netbios-ns,vsys1,TAP-ZONE,TAP-ZONE,ethernet1/1,\
   ethernet1/1,Logger,2010/08/26 13:56:07,136674,3,137,137,0,0,0x8000,udp,\
   alert,"",NetBIOS nbtstat query(31707),any,low,client-to-server

Okay, it is a comma delimited, the date format is okay, it has fixed numbers of fields and lots of information. The Palo-Alto can also be customized to add or substract fields in the syslog profile settings. Do not do this unless you want to customize all your rules!!! So this is actually a pretty easy format to work with in OSSEC.

Second is to create a generic decoder for all Palo-Alto devices. To do this create a local_decoder.xml file in your ./ossec/etc/ directory. This file will contain anything that is specific to your installations. This file will not be overwritten during an upgrade!

OSSEC decoder tree entry point – it’s a Palo Alto Firewall

Lets go and identify what is unique about PaloAlto devices. Add this to your local_decoders.xml file.

<!-- Custom decoder for PaloAlto Firewalls Events -->
<decoder name="paloalto"> 
  <prematch>^\d,\d\d\d\d/\d\d/\d\d \d\d:\d\d:\d\d,\w\w\w\w\w\w\w\w\w\w\w,</prematch>
</decoder>

The unique key here, is the series of \w, which are the serial number of the device.

For regex purists, OSSEC only has the basics and is not as greedy. Learn more about OSSEC regex syntax here.

Identify the type of event and extract fields

Next, we want to identify  a type of event which in our example above is THREAT.

<!-- Custom decoder for PaloAlto Firewalls THREAT Events -->
<decoder name="paloalto-threat">
 <parent>paloalto</parent>
 <prematch offset="after_parent">^THREAT,</prematch>
 <regex offset="after_parent">^(THREAT,\w+),\.*,\.*,(\d+.\d+.\d+.\d+),(\d+.\d+.\d+.\d+),\d+.\d+.\d+.\d+,\d+.\d+.\d+.\d+,\.+,(\w*\\*\w*),(\w*\\*\w*),(\w*),\.+,(\.*),\w*,(\w*),\w*$</regex>
 <order>id,srcip,dstip,srcuser,dstuser,protocol,action,status</order>
</decoder>

Extracted fields are based on what can help for policy and incident response. The decoder will extract the required information and fill the OSSEC variables.

• The type of threat
• The source IP address
• The destination IP address
• The source user (when linked to the Active Directory or auth portal)
• The destination user (when linked to the Active Directory or auth portal)
• The reported protocol implicated
• The action taken by the firewall (alert, deny, block-ip, allow, drop-all-packets)
• The severity of the detection threat (informational, low, medium, high, critical)

Classify events based on severity of the event

Now we want to classify some of these events. We will use the severity of the alert for this. This makes it easy to capture all events and filter as desired. Notice we go from the general to the specific.

Syslog (not shown) -> Device Decoder -> Event Type Decoder -> Categories for that type

<group name="syslog,paloalto,threat,">
  <rule id="101000" level="0">
    <decoded_as>paloalto</decoded_as>
    <id>^THREAT</id>
    <description>PaloAlto Firewall Event</description>
  </rule>

  <rule id="101001" level="12">
    <if_sid>101000</if_sid>
    <status>critical</status>
    <description>Critical threat event!</description>
  </rule>

  <rule id="101002" level="12">
    <if_sid>101000</if_sid>
    <status>high</status>
    <description>High threat event!</description>
  </rule>

  <rule id="101003" level="8">
    <if_sid>101000</if_sid>
    <match>medium</match>
    <description>Medium threat event!</description>
  </rule>

  <rule id="101004" level="8">
    <if_sid>101000</if_sid>
    <match>low</match>
    <description>Low threat event!</description>
  </rule>

  <rule id="101005" level="5">
    <if_sid>101000</if_sid>
    <match>informational</match>
    <description>Informational threat event!</description>
  </rule>

Using security policy to drive creation of further rules

Next steps would be based on your security policy. What critical assets are being protected, what systems are involved, what data and how can it be logged or tracked. Use OSSEC to identify anomalies to your normal baseline. What events can also be dropped from being alerted on for things liked low severity blocked threats from outside to inside.

Example anomalies that would differ from an expected baseline :

• Time based anomalies for data or system access (success or failure)
• Credential based anomalies for data or system access (success or failure)
• Quantity based anomalies for data or system access (success or failure)
• Protocol based anomalies
• Traffic direction based anomalies
• Statistical anomalies of network traffic or system characteristics or data access patterns
• Policy based anomalies (ex. a domain admin account browsing the internet)

Composite rules to alert on actionable events

Finally composite rules of anomalies with external complimentary data are the automation golden cup. Covering sources like vulnerability data, server data, known bad domains or bad IPs. For example these sources can be correlated to attack vectors seen by the firewall so that they are raised as very critical events (if a successful attack is detected or possible).

Next steps – Decoding more Palo-Alto firewall event types

OSSEC and PaloAlto integration Part 2 covers other types of logs that a Palo-Alto can generate like CONFIG and SYSTEM events. As well as identifying a general categorization and specific events of interest that need to be tracked for actionable security.

• SANS Instructors and speakers are top rate (once again)
• The invited vendors were pertinent to the conference and added to the experience
• SANS @ Night and special conferences were very much worth it and provided up to date actionable material.
• Lots of senior technical personnel from military, industrial, public sector, critical infrastructure and commercial interests bring great depth to the discussions in class and out of class.
• NetWars, how awesome is that stuff!! Sign me up for next time.

The standout speakers were Dr. Eric Cole, Joshua Wright, Mike Poor and Rob Lee. Their talks all brought unique insight, usable knowledge and were entertaining as well. Special props go out to Alissa Torres from Mandiant on Finding Unknown Malware, she dished out a no holds barred great presentation, you rocked. (You did not have to be stressed, things worked out. :-)

Dr. Eric Cole was a real treat as an instructor, he knows what he is talking about and has real knowledge of how to handle incidents. David Hoelzer was a super instructor and delivered a well rounded course and good insight. He even threw in some custom Net Wars for the class, awesome stuff.

In a few words:

• Identify your critical assets
• Identify the systems and data
• Identify how to monitor them

Prevention is good, but detection is king. Also, make sure that behavioural and anomaly detection are key components of the detection in addition to traditional signature/known-bad based detection.

This gave me some ideas for new statistical rules to implement with flow based analysis and SIEM type correlation rules. I really focus on NSM (Network Security Monitoring) approaches which should also be combined with data access monitoring to really highlight bad behaviours. Data access monitoring is a new field that I will be exploring.

This means, you can easily add more devices, submit corrections and  improve this old school tool reborn for use with the up and coming Shinken monitoring system.

Yup. genDevConfig 3.x, dedicated to creating Shinken configuration files is ready to rock. What does this mean?

1. genDevConfig creates a configuration file formatted for Shinken with Host and Services
2. Automatically creates a dependency to the chassis Service
3. Can be assigned user defined variables
4. Can be assigned user assigned templates
5. Supports all the same devices that it used to do with the addition of more comprehensive Nortel/Avaya ES and ERS switches

First question on your mind… Can I migrate my Cricket installation to Shinken?!

Short Answer:  Yes, but as a proof of concept. Not for production.

Shinken’s various web interfaces differ a little bit. The closest to Cricket are WebUI with native Graphite support, Pencil, Multisite+Graphite Dashboard and Multisite+PNP4Nagios. Cricket’s web interface is an automatic hierarchical dashboard, designed with a single purpose in mind. While Shinken processes state and performance data so it arranges data with different priorities.

Second, Shinken has a discovery management console to provide an interface for discovery scripts like genDevConfig. It has not been tested and integrated to be used with Shinken’s discovery console.

Third, Shinken’s SNMP poller module that is meant to be used with genDevConfig configurations only presently supports SNMPv2c getBulk. While great, will not collect data from older devices, UPSs or anything that requires SNMP v2c get/get-next or SNMP v1.

Fourth, Shinken’s SNMP poller module is in public beta and has not been officially released.

Question: Okay, but can I try it?

Answer: By all means. genDevConfig is ready. Shinken’s SNMP poller module (SnmpBooster) is also ready.

Learn more about it here:

• Shinken SNMP module presentation and installation links
• Shinken SNMP module design specification and development status 
• Shinken 10 minute installation guide
• genDevConfig on github

Hope to see you around and contributing to the project.

Nagios does not have any true discovery frameworks that generate configuration trees. It is more of a piecemeal affaire, where each acquisition check is responsible for knowing what it gets and how. Basic information is passed from Nagios to the check. As you can imagine, for snmp polling, this is a very inefficient method. Calling an interpreter, loading a bundle of modules, executing a tiny script, exit and repeat. Even if they are C or scripted checks. Ouch.

Some innovation has happened in the check_mk plugin which does a good job of centralizing all snmp polling and data normalization before sending the data back as a passive check response. Other means of scaling snmp checks are by using collectd to do the polling and data normalization and just having Nagios check for updated values. Meh.. Fortunately, the collectd guys are supposed to be working at converting this to sending data back as passive check responses.

Nagios, is a real monster to hack. As luck would have it, a pair of Nagios professional admin, book writers and developers took the beast by its horns. They created Shinken, it is a Nagios compatible rewrite in Python. It offers a well thought out design offering scalability, modularity and worlds of flexibility.

Shinken offers distributed poller daemons  that can load modules. These modules make it easy to extend its feature set and scope. The aim is to create a python based SNMP poller module that can execute SNMP checks in a logical and efficient manner using the native python library that supports SNMPbulkwalk and SNMPbulkget. It has its drawbacks, but it does not require installing Net-SNMP and it is actively developped.

The distributed nature of the scheduling and polling in Shinken, the Bulkwalk efficiency and simpler interface offset the slower performance of pySNMP. It is noted that the error handling and corner cases for tables do suffer some inconsistencies, so these need to be accounted for. Performance is estimated around 5000 checks per second using the python module with high cpu usage, while net-snmp goes for over 10000 checks per second with low cpu usage.

So there you have it. There may eventually be an alternative for Cricket users used to the automation offered by genDevConfig and the simplicity of the Cricket config-tree that are looking for data gathering and monitoring in a single core.

More details will be provided when the beta version is released. I expect the configuration output to have a few iterations to find the right mix between what is created by genDevConfig and what is included in the templates. As most of you will note, Cricket’s and Nagios’ configuration inheritance model are not exactly the same.

• SANS Instructors and speakers are top rate
• The invited vendors were pertinent to the conference and added to the experience
• SANS @ Night and special conferences were very much worth it and provided up to date actionable material.
• Lots of senior technical personnel from military, industrial, public and commercial interests rounded out the experience.

The standout speakers were Lenny Zelster, Jason Fossen and Ed Skoudis. Their talks all brought unique insight, usable knowledge and were superb speakers. Eric Conrad was an enjoyable instructor, he provided no nonsense feedback on what worked for him in his various jobs.

Network security monitoring and extrusion detection have really gained in the general mindset. As there are no silver bullets, the intruders will gain a foothold and only well varied system and network indicators can highlight anomalous behaviour that can permit a response. It is also clear that response now needs to be more coordinated and all encompassing to lockdown a site before advanced intruders react to partial measures and deploy more advanced malware or burrow deeper.

Main attack vectors where directed Spear-fishing and combination’s of drive-by exploits or business document exploits. For local attacks, any shared layer 2 network is a sitting duck to MiTM attacks which impress by their breadth, variety and tool automation.

On the SIEM side of things, Qradar, LogRythm and Splunk* all had products that were well received by various network and security administrators. (ed. Splunk is an analytical tool and does not have a state machine like a SIEM, I see it as complimentary to a SIEM)

On the NSM side of things, Sourcefire had a compelling commercial solution that is similar to SGUIL but with more setup automation and polished approach.

Advanced network monitoring

• Forward the black hole routes to a passive Honeypot such as Nepenthes
  • This would entail having either the black hole router or the core router forward the traffic to the Honeypot as the next-hop for all the black holed routes.
  • The honey pot will accept connections and will let the operator know what the evil doer was up to. Ex. C2 channels, File transfers, protocol tunnelling, scanning, exploiting, etc.
• Reconfigure or implement a DNS black list to point the blocked traffic to specific black hole networks. Traffic can be categorized using pre-built filters in the analysis station.
  • Known malware+spyware black list resolves to x.x.x.A
  • Domains, sub-domains or FQDNs that are unrelated to business needs (.tv, .sex) black list resolves to x.x.x.B
  • Domains, sub-domains or FQDNs considered security risks* (.ru, .cn, .ws, .cm, .info, etc.) black list resolves to x.x.x.C. Mcafee publishes a list of the riskiest TLDs, Mapping the mal web.
  • Blocked web sites due to company policy resolves to x.x.x.D
  • In maintenance or no longer available internal servers resolves to x.x.x.E
  • Other blocked content that is of no interest to network or security analysts resolves to 127.0.0.1
• Advertise specific black hole routes that target nasty netblocks.
  • If there is a requirement in splitting traffic reaching specific routes, convert the link between the core router and the black hole router to an 802.1Q trunk with sub-interfaces. This way you can advertise each group of nasty routes on it’s own sub-interface. Reporting would know which 802.1Q tag is associated with which group of nasties.
  • The risk with explicitely blocking network blocks or domains, is the user community may not be aware they HAVE been blocked administratively. Such that a web page or email track back saying, hey the ressource you are trying to reach is blocked! Some type of advisory should be published that some domains or netblocks may be blocked, check list or contact your help desk.
• Graph the traffic hitting each sub-interface or interface on the black-hole router using SNMP and Cricket or Cacti.
• Run ntop, sancp or argus on the analysis station to get session data.
• Find a way to generate tickets or events back into analysts consoles for interesting traffic hitting the black hole router.
  • Cleaning most of the above classes of traffic will be beneficial for fixing operational problems.
  • Storing this traffic in your network analyzer can serve as an easy to maintain forensic archive for infection vectors.
  • A very cheap alternative to a honeypot.
  • Forensic history of problematic sources in the managed network.
  • Provides a list of workstations and their associated users that are risks to the managed LAN.
  • An easy way to archive the presence of traffic that was marked for deletion by other systems. (Policy based routing, DNS, proxies)
  • Can serve as a source of automated tickets for ops and events for security folks. As what needs to be fixed is always on the managed side of the LAN or WAN.
  • I would love to hear about other benefits

Recap of the benefits

• Provides a looking glass into misconfigured, unauthorized, unexpected or inexistant traffic or destinations.
• Cheap and easy to setup
• Maintenance and operation can be very automated
• Integrates with existing processes (ticketing, change management, backups, monitoring, logging)

Executive summary

Static black routing and policy based black hole routing are great ways to securely offload traffic from your core routers or firewalls. The traffic that ends up being dropped could be known source or destination networks that should not be routed, but they could also be unexpected, misconfigured or malicious traffic that is being dropped. Keeping an eye on this traffic could pay off big time. NSM at work! But first how to do the basic setup.

Prerequisites

• A spare router running Cisco IOS 12.2.x and up
• A spare network tap or mirror port on your core route
• A spare distributed network analyzer or workstation with wireshark
• Console port access or out-of-band access to the router via a secure terminal server

An alternate way of processing the incoming traffic directly in a honeypot described here.

Physical setup

• Connect the network tap to your core-router
• Connect your black-hole-router to your network tap.
• Connect your analyzer to the replication port of your network tap.
• Connect your router to your out-of-band access

Logical setup on a cisco IOS router with a test route

• Apply secure configuration guidelines to protect your black-hole-router. *
• Configure a point to point routed interface on black-hole-router and your core-router.
• Create the appropriate routing process
• Make the black-hole-router a sub that will only advertise static routes
• Add a redistribute static statement to your routing process
• Create a null0 interface
• On null0 configure no ip unreachable
• Now create a test black hole route: ip route 10.255.255.1 255.255.255.255 Null0

! Example using eigrp IGP   

!   

ip classless   

!   

interface null0   

 no ip unreachable   

 no shutdown   

!   

router eigrp 1   

 redistribute static   

 eigrp stub static   

 network x.x.x.x # replace x.x.x.x with the network of the point-to-point interface   

 no auto summary   

!   

ip route 10.255.255.1 255.255.255.255 null0

• Verify the route is showing up in the core router.
• Send traffic to it from another network location. This traffic should be blackholed on the router. Check with your traffic analyzer, with a debug statement or a logged ACL on your router.

Applying the black hole route on a cisco IOS router Here you need to choose what type of black hole route to use. Can you inject a default black hole route, specific black routes routes or both. Your network may be architected so that you already have a black hole route somewhere that could be migrated to your new setup. You may wish only to black hole routes to certain networks (ie. bogon networks, known “bad net blocks” or AS). Use a bit of DNS foo to blackhole “bad” domains, sub-domains or FQDNs.

! Example using eigrp IGP   

!   

ip route 0.0.0.0 0.0.0.0 null0   

!

By creating a black hole route, all traffic that is now destined to networks with no explicit routing entries will end up on your new black hole router. After applying the route, the first order of business is making sure you have not broken anything during your maintenance window. You are doing this work in your lab or at least in a maintenace windows right, ;-). Make sure you can still reach your application servers, your proxies, external NATs, etc. Once you are satisfied save your configuration on the router.

Now, fire up your network analyzer. Traffic can now be categorized

• There should be your typical misconfigured hosts, moritoring servers, deprecated DNS entries sending traffic to networks that do not exist.
• Workstations trying to update software using non-proxied ports from internet vendor servers.
• Management/monitoring software scanning ports of IPs for auditing purposes
• And the famous other category

Yeah, fun with “other”. Non browser based applications trying to send web or ftp traffic without going through the designated proxies. Applications trying to reach internet addresses for which there are no proxy services. Having worked hard at designing your network to make use of NAT and proxies, you get to have a much easier time tracking down the above classes of traffic.

Benefits of this easy win approach

• Cleaning most of the above classes of traffic will be beneficial for fixing operational problems.
• Storing this traffic in your network analyzer can serve as an easy to maintain forensic archive for infection vectors.
• A very cheap alternative to a honeypot.
• Forensic history of problematic sources in the managed network.
• Provides a list of workstations and their associated users that are risks to the managed LAN.
• An easy way to archive the presence of traffic that was marked for deletion by other systems. (Policy based routing, DNS, proxies)
• Can serve as a source of automated tickets for ops and events for security folks. As what needs to be fixed is always on the managed side of the LAN or WAN.
• I would love to hear about other benefits

* – SANS IOS security guidelines or other security focused templates. Any mistake on this router can have nasty consequences. All your routers should be equally secure, audited, logged, change controlled, monitored and backed up.