acktomic.com

Black hole routing is now your friend. You are forwarding traffic not meant to be routed to a termination point on your network. Let’s look at some advanced ways to get more focused network security data from this traffic. For part 1, skip to the next post.

Advanced network monitoring

• Forward the black hole routes to a passive Honeypot such as Nepenthes
  • This would entail having either the black hole router or the core router forward the traffic to the Honeypot as the next-hop for all the black holed routes.
  • The honey pot will accept connections and will let the operator know what the evil doer was up to. Ex. C2 channels, File transfers, protocol tunnelling, scanning, exploiting, etc.
• Reconfigure or implement a DNS black list to point the blocked traffic to specific black hole networks. Traffic can be categorized using pre-built filters in the analysis station.
• Known malware+spyware black list resolves to x.x.x.A
• Domains, sub-domains or FQDNs that are unrelated to business needs (.tv, .sex) black list resolves to x.x.x.B
• Domains, sub-domains or FQDNs considered security risks* (.ru, .cn, .ws, .cm, .info, etc.) black list resolves to x.x.x.C. Mcafee publishes a list of the riskiest TLDs, Mapping the mal web.
• Blocked web sites due to company policy resolves to x.x.x.D
• In maintenance or no longer available internal servers resolves to x.x.x.E
• Other blocked content that is of no interest to network or security analysts resolves to 127.0.0.1
• Advertise specific black hole routes that target nasty netblocks.
  • If there is a requirement in splitting traffic reaching specific routes, convert the link between the core router and the black hole router to an 802.1Q trunk with sub-interfaces. This way you can advertise each group of nasty routes on it’s own sub-interface. Reporting would know which 802.1Q tag is associated with which group of nasties.
  • The risk with explicitely blocking network blocks or domains, is the user community may not be aware they HAVE been blocked administratively. Such that a web page or email track back saying, hey the ressource you are trying to reach is blocked! Some type of advisory should be published that some domains or netblocks may be blocked, check list or contact your help desk.
• Graph the traffic hitting each sub-interface or interface on the black-hole router using SNMP and Cricket or Cacti.
• Run ntop, sancp or argus on the analysis station to get session data.
• Find a way to generate tickets or events back into analysts consoles for interesting traffic hitting the black hole router.
  • Cleaning most of the above classes of traffic will be beneficial for fixing operational problems.
  • Storing this traffic in your network analyzer can serve as an easy to maintain forensic archive for infection vectors.
  • A very cheap alternative to a honeypot.
  • Forensic history of problematic sources in the managed network.
  • Provides a list of workstations and their associated users that are risks to the managed LAN.
  • An easy way to archive the presence of traffic that was marked for deletion by other systems. (Policy based routing, DNS, proxies)
  • Can serve as a source of automated tickets for ops and events for security folks. As what needs to be fixed is always on the managed side of the LAN or WAN.
  • I would love to hear about other benefits

Recap of the benefits

• Provides a looking glass into misconfigured, unauthorized, unexpected or inexistant traffic or destinations.
• Cheap and easy to setup
• Maintenance and operation can be very automated
• Integrates with existing processes (ticketing, change management, backups, monitoring, logging)

The average Network Administrator has control over all the tools to put in place effective black hole routing and reporting. The simple steps below should get you started and the prerequisites to getting there.

Executive summary

Static black routing and policy based black hole routing are great ways to securely offload traffic from your core routers or firewalls. The traffic that ends up being dropped could be known source or destination networks that should not be routed, but they could also be unexpected, misconfigured or malicious traffic that is being dropped. Keeping an eye on this traffic could pay off big time. NSM at work! But first how to do the basic setup.

Prerequisites

• A spare router running Cisco IOS 12.2.x and up
• A spare network tap or mirror port on your core route
• A spare distributed network analyzer or workstation with wireshark
• Console port access or out-of-band access to the router via a secure terminal server

An alternate way of processing the incoming traffic directly in a honeypot described here.

Physical setup

• Connect the network tap to your core-router
• Connect your black-hole-router to your network tap.
• Connect your analyzer to the replication port of your network tap.
• Connect your router to your out-of-band access

Logical setup on a cisco IOS router with a test route

• Apply secure configuration guidelines to protect your black-hole-router. *
• Configure a point to point routed interface on black-hole-router and your core-router.
• Create the appropriate routing process
• Make the black-hole-router a sub that will only advertise static routes
• Add a redistribute static statement to your routing process
• Create a null0 interface
• On null0 configure no ip unreachable
• Now create a test black hole route: ip route 10.255.255.1 255.255.255.255 Null0

! Example using eigrp IGP   

!   

ip classless   

!   

interface null0   

 no ip unreachable   

 no shutdown   

!   

router eigrp 1   

 redistribute static   

 eigrp stub static   

 network x.x.x.x # replace x.x.x.x with the network of the point-to-point interface   

 no auto summary   

!   

ip route 10.255.255.1 255.255.255.255 null0

• Verify the route is showing up in the core router.
• Send traffic to it from another network location. This traffic should be blackholed on the router. Check with your traffic analyzer, with a debug statement or a logged ACL on your router.

Applying the black hole route on a cisco IOS router Here you need to choose what type of black hole route to use. Can you inject a default black hole route, specific black routes routes or both. Your network may be architected so that you already have a black hole route somewhere that could be migrated to your new setup. You may wish only to black hole routes to certain networks (ie. bogon networks, known “bad net blocks” or AS). Use a bit of DNS foo to blackhole “bad” domains, sub-domains or FQDNs.

! Example using eigrp IGP   

!   

ip route 0.0.0.0 0.0.0.0 null0   

!

By creating a black hole route, all traffic that is now destined to networks with no explicit routing entries will end up on your new black hole router. After applying the route, the first order of business is making sure you have not broken anything during your maintenance window. You are doing this work in your lab or at least in a maintenace windows right, ;-). Make sure you can still reach your application servers, your proxies, external NATs, etc. Once you are satisfied save your configuration on the router.

Now, fire up your network analyzer. Traffic can now be categorized

• There should be your typical misconfigured hosts, moritoring servers, deprecated DNS entries sending traffic to networks that do not exist.
• Workstations trying to update software using non-proxied ports from internet vendor servers.
• Management/monitoring software scanning ports of IPs for auditing purposes
• And the famous other category

Yeah, fun with “other”. Non browser based applications trying to send web or ftp traffic without going through the designated proxies. Applications trying to reach internet addresses for which there are no proxy services. Having worked hard at designing your network to make use of NAT and proxies, you get to have a much easier time tracking down the above classes of traffic.

Benefits of this easy win approach

• Cleaning most of the above classes of traffic will be beneficial for fixing operational problems.
• Storing this traffic in your network analyzer can serve as an easy to maintain forensic archive for infection vectors.
• A very cheap alternative to a honeypot.
• Forensic history of problematic sources in the managed network.
• Provides a list of workstations and their associated users that are risks to the managed LAN.
• An easy way to archive the presence of traffic that was marked for deletion by other systems. (Policy based routing, DNS, proxies)
• Can serve as a source of automated tickets for ops and events for security folks. As what needs to be fixed is always on the managed side of the LAN or WAN.
• I would love to hear about other benefits

* - SANS IOS security guidelines or other security focused templates. Any mistake on this router can have nasty consequences. All your routers should be equally secure, audited, logged, change controlled, monitored and backed up.

Making use of time series trending for network security monitoring. A short history with examples.

Time series trending tools like Cricket, Cacti and Torrus focus on performance and availability.

Operational teams use these day in day out, the tools are there to organize and display reams of data in a quick and painless way.Time series trending is a source of indicators.

Strengths of time series trending

• detailed baseline
• displays seasonality
• highlights anomalies
• illustrates subtle changes
• provisioning planning

Strengths of time series trending using RRD databases

• fast visualization - anyone having had to suffer SQL based trending tools can attest
• low maintenance
• graphing flexibility
• capabilities can be extended
• no vendor lockin or annual maintenance fees

Making use of these strengths to identify threats is a secret recipe. Yeah, reaaally.

Here are some DNS based trends that can help quantify, understand and defend or cleanup against extrusion attemps or malware/botnet command and control communications.

• Trending the number of hits against security related blacklisted entries
• Trending the number of hits against .cn, .ru, etc. specific domains
• Trending the number of hits against honeypot entries
• Trending the number of hits by source IP

The data value can be extended by doing a bit of munging on the interesting bits.

The information could also be reported in dashboards on a Splunk server for operational teams or management. Trending data over the long term also unshackles the administrator from defining hard limits in time of day use for what is normal and what is not. You can still define hard limits which can generate security events, but analysis is made much easier by seeing the whole time series.

Once identified the vector of the outbreak needs to be cleaned before nasty malware can move in (think Zeus variant).