- Nate King and Errol Weiss
Surgeon Michael Nusbaum knows a thing or two about hacking. Many years after reaching the "mandatory retirement age of 13" as a computer hacker, he commands the infosec defenses as COO at Hamilton Scientific Ltd., an ASP that provides Web-based medical software.
Atop Hamilton's multilayered defense system sits one of a new class of network forensics analysis tools (NFATs): Niksun's NetDetector. These products -- which also include SilentRunner and Sandstorm's NetIntercept -- combine robust network traffic capture with sophisticated analysis and forensics capabilities.
Before he would take Hamilton Scientific live on the Internet, Dr. Nusbaum searched for a product that monitored network traffic flow and had the forensics capabilities to enable his IT staff to trace back to any machines that attempt to log in or access its systems. That kind of information arms him to counter possible threats to the sensitive medical records databases accessed by physicians and administrators.
"You have to be able to adjust your defense mechanisms and parameters based on the attacks that occur," says Nusbaum, still a very active vascular surgeon (his interview with Information Security was planned to fit into his surgery schedule). "You can recreate what happened, and based on that make assumptions and maybe throw some more resources in that direction, if need be."
Sysadmins can use NFAT products to perform complex analysis that in the past was performed using a painstaking brute-force approach or was somewhat automated using creative scripting.
A case in point is Van Nguyen, director of global security for cargo shipping giant American Presidential Lines (APL), who uses SilentRunner, an NFAT that features powerful visual analysis capabilities, to bolster security significantly with existing staff.
"My staff can be trained to use the tool to do investigations, the forensics, [and] the network traffics analysis," says Nguyen. "The tool provides all that view without additional head count."
Putting NFAT to Work
As a logical next step beyond IDSes and the new breed of DoS mitigation tools which use signature- and/or anomaly-based methods to detect and report an attack, NFAT products capture and retain all network traffic and provide the tools for forensics analysis. While an IDS may flag a particular attack, an NFAT user can replay, isolate and analyze an attack or suspicious behavior, then bolster network defenses accordingly.
In fact, the three NFAT vendors interviewed for this article emphasize that their products should complement IDSes and firewalls. The playback features in all these tools show how and what has happened on the network. In times of a successful attack, the products reveal exactly which hosts were broken into, which were compromised and then what happened next.
"If you're able to thwart an attack, you'd like to go back and see what they are doing," Nusbaum says. "What attempts did they make? What probes did they use? Where did they go? What were they using to get in?"
It's hard to say exactly what we've been living without all these years. NFAT products defy narrow definition, in part because of the wide range of uses suggested by both vendors and customers. An admin might discover, for example, that an employee in the engineering department was e-mailing plans for the next major product to a competitor or that someone was tunneling Telnet sessions out of the company network to a host in Korea using HTTP. Discussions with all three vendors and several customers suggest several uses for this new class of product:
- Intellectual property protection
- Detection of employee misuse/abuse of company networks and/or computing resources
- Risk assessments
- Network forensics and security investigations
- Exploit (break-in) attempt detection
- Data aggregation from multiple sources, including firewalls, IDSes and sniffers
- Incident recovery
- Prediction of future attack targets
- Anomaly detection
- Network traffic recording and analysis
- Network performance
- Determination of hardware and network protocols in use
One of the innovative uses of these tools was mentioned by Niksun: using a replay of captured traffic to test new IDS signatures. After IDS attack signatures are updated, a replay would show if the newly installed signatures would catch the attack the next time around.
The three NFAT products examined in this article differ substantially from one another, but all combine the ability to passively monitor and capture all network traffic, use forensics tools to analyze traffic, track down security violations and protect against future violations and attacks.
SilentRunner is the analysis and visualization powerhouse of the NFAT universe. As SilentRunner churns information, the network comes to life as the tool paints a vivid 3-D network diagram. Packets scream around the network, lighting it up as they flow. Using a specialized "six-degrees-of-freedom" input device, the user can manipulate, rotate and fly through the data, re-orienting to new views to get a clearer picture of what's happening on the network. Packets are played back as they happened, and SilentRunner keeps enough time-stamp granularity to track them down to a millisecond. Things happen so quickly that the playback looks like cameras flashing in a paparazzi feeding frenzy. Fortunately, you can adjust timing and playback speeds, just like the frame controls on your VCR.
Bet you didn't think you could spend more than $500 on a mouse? For the special 3-D effects and for the ability to "fly through" your data, SilentRunner recommends several products from Logitech's 3Dconnexion SpaceMouse and SpaceBall product lines.
The 3-D visualization is a large part of SilentRunner's claim to fame and success in the marketplace. The idea is that while powerful algorithms cut through a lot of analysis, synthesis, categorization and interpretation of multiple data sources, the human analyst can put it all together by grasping patterns that emerge on the screen.
What started out as a tool to help spies weed out nuggets of intelligence from a massive amount of unrelated data has evolved to a commercial tool with lots of applications, especially in network forensics. Raytheon spun off the SilentRunner business unit as a wholly owned subsidiary in October 2001.
Infosec professionals will use SilentRunner in numerous ways. The SilentRunner 3-D rendering makes it easy to spot malicious and anomalous network behavior. We saw examples on finding insiders gone bad, watching the spread of macro viruses, identifying sources of internal attacks, and seeing a "low and slow" attack on the network from two seemingly unrelated sources.
SilentRunner can help identify network traffic bottlenecks, unauthorized network connections and back doors. We also could quickly see the Web sites that people were visiting. The lunch hour was a popular Web browsing time. By looking at a quick mosaic tiling of recent graphics grabbed from Web sites, we could see people house-hunting, checking their stocks and catching up on the latest sports scores.
Although all the NFAT tools will track Web page viewing to one degree or another, none does a great job of really rendering what the Web user actually viewed. NetIntercept and SilentRunner allow analysts to see all images viewed, but none of the tools can detect steganography in images.
Steep learning curve. SilentRunner packs a lot of potential, but that potential comes with a big price -- not just in dollars.
Be prepared to invest time learning how to use this very powerful and complex tool. Users will need a five-day introductory course and are well-advised to follow up with a three-day advanced analysis class – feedback from people who attended SilentRunner training said they didn't get much out of the basic course because the analysis is so complex.
Want to take a test drive? SilentRunner requires potential customers to have clearly defined objectives on how they're going to use it and commit to a three-day training boot camp -- just to use a 30-day demo!
We found it impossible for even seasoned computer security experts to jump into a SilentRunner console and get anything meaningful done. We know, because we tried. We were flying through the data, just as the marketing literature promised, but had no idea what we were looking at.
"SilentRunner is not for the novice. It's not for someone who doesn't have experience," says Steve Chapin, director of the Center for Systems Assurance at Syracuse University.
What makes SilentRunner run? SilentRunner's desktop is logically broken into three major components: Collector, Analyzer and Visualizer. Collector can be used locally or placed on a remote network segment to capture and record packet and session information. Collection files are played through the Analyzer and Visualizer tools. Analyzer has a slew of algorithms with front-end interactive tools and graphical displays to help the security analyst derive useful information from all those captured network packets. The visualizer is a 3-D rendering tool that helps the analyst recognize trends or patterns by flying through views of the interactive network.
Collector. Collector is essentially a sniffer placed on the target network. It runs on a dedicated Windows 2000/XP platform that requires a SilentRunner-modified Network Driver Interface Specification (NDIS) packet driver. Major components include a sensor manager, a collection engine and a network viewer.
Remote management of the Collectors appears to be an afterthought. Communications to the sensor manager are not encrypted, so confidentiality and integrity between the Collectors and analysis workstation depends on the user creating a secured management network. For example, you could have the Collector monitor the target network with the primary NIC and use a second NIC on a physically separate and secured network (such as a VPN) to remotely manage and administer the Collector.
Though SilentRunner does well on Ethernet segments, the collector's raw packet capture capability didn't seem as robust as NetDetector's or NetIntercept's. While running it on Windows reduces complexity and costs for hardware, it limits its ability to capture packets on high-speed (1 GB-plus) networks.
An inference engine, which uses proprietary algorithms, draws conclusions based on network data that can be displayed to the user. The display uses varying colors to represent what has been taken as fact versus what has been inferred, and to what degree.
The collection engine captures and stores packets from the local network segment. Accessing displays based on Collector data, users can drill way down to hex readouts and see the guts of the Ethernet packets, very much as they would with NetDetector and NetIntercept or a typical network analyzer, for that matter.
Analyzer and Visualizer
The Analyzer creates a two-dimensional network diagram showing communication relationships between nodes based on Collector files or external sources, such as IDS or firewall logs. Analyzer's algorithms establish and display the relationship between nodes based on communication patterns.
Behind Analyzer, a Context Analyzer -- a key to Silent-Runner's robust analytical capability -- uses N-gram analysis to determine relationships between similar types of information processed from SilentRunner source files. Context Analyzer determines the similarity between files and groups them in a technique referred to as "blind clustering." The N-gram analysis isn't keyword-driven and/or dependent on language (such as English vs. German or "techno-speak" vs. "legalese") or file type.
Since we were bouncing back and forth between analysis and visualization during the demonstrations, we're not sure where one begins and other takes over. It's perhaps best explained through examples.
The first demonstration we saw was a capture of e-mails from an internal company network. In this scenario, the analyst reviewed e-mails to make sure company propriety information wasn't at risk.
As Analyzer processes SilentRunner source files, a virtual network diagram begins to emerge. Be prepared, though, because it may take some time to load up.
Through Visualizer, it was quick and easy to tell who was e-mailing whom based on the clustering. When it was able to figure out the context of each message, the display depicted boxes representing each person and produced clusters of conversations through N-gram analysis.
In our canned scenario, three large clusters were apparent:
- A group of people in engineering were exchanging e-mails about the status of a project they were working on;
- The billing department had e-mailed an outside third party about the status of an equipment order; and
- People in marketing were talking about the movies they had seen over the weekend.
Because it was the only box talking to a particular outside entity, instead of the usual clustering, it was obvious a lone insider was talking with someone outside, and the discussion was beyond the bounds of what was considered a "normal" topic. It turned out to be about a sensitive new intercompany partnership that was on the drawing board. Contrived? Yes. But it does make the point.
Another demonstration contained about 95,000 security events from a Cisco Secure IDS. Analyzer completed the load in a few minutes-a good reason to have lots of disk space, lots of RAM and multiple fast processors. The Visualizer displays made it obvious what systems were the big talkers on the network and the degree of interaction between each of the systems on the network.
SilentRunner Analyzer was designed to be source-independent-it can look at log files from just about any device that can produce a predictable delimited format. Analyzer uses "Spec Templates" to parse through source files (captured packets from Collector) or input log information from other sources. SilentRunner comes with a number of spec templates for the popular IDSes (Dragon, RealSecure, Cisco Secure IDS) and firewalls (Check Point, Cisco PIX). So, you can take log files from an IDS or firewall and manually feed them into the SilentRunner Analyzer.
Pricing starts at $20,000 for the "Lite" version and $65,000 for the standard package. Unlike NetDetector and NetIntercept, which include hardware and software as a package, these costs are for software licenses only. SilentRunner Lite is a greatly reduced offering and does not include the major features that set the full product apart from the others we reviewed. If your budget is limited to the $20,000 range, you're better off choosing between NetIntercept and NetDetector.
Now that Hamilton Scientific's Nusbaum, the ex-hacker turned surgeon, is on the side of the angels, he appreciates what NetDetector can do. Nusbaum uses the NFAT tool to analyze potential attacks and security violations that could expose Hamilton's sensitive medical data. Nusbaum uses NetDetector in tandem with Niksun's first product, NetVCR, a network performance and quality-of-service measurement tool.
"NetVCR is the camera in the bank that's recording all the activity that's going on. NetDetector is like the alarm system," says Nusbaum. "Having them in combination really brings up the level of security to that of a major government agency."
That's important when the government mandates under the Health Insurance Portability and Accountability Act (HIPAA) that electronic medical records be secured. Among other things, Niksun's products create an audit trail, which HIPAA requires.
Niksun, founded in 1997, made a name for itself with NetVCR. In December 2000, the company introduced NetDetector, which uses the same high-end packet capture engine as NetVCR. So it's no surprise that NetDetector's biggest strength is its robust network capture ability.
NetDetector is also the only one of the three NFAT tools with a sophisticated alert system. However, its analysis capability is quite limited, and there are questions about the security of the connections over which it's managed.
Unlike SilentRunner -- and like NetIntercept -- NetDetector is a dedicated dual-processor hardware appliance running a modified version of FreeBSD. The modified FreeBSD kernel prioritizes I/O and allows users to simultaneously collect, analyze, store and query data on multiple network interfaces.
A Java-based Web interface allows the analyst to configure, administer, monitor and use the forensic tools. The interface is intuitive, with toolbar buttons providing quick access to the collection, analysis, alerting and reporting tools. Power users will appreciate a command line option to perform analysis and potentially to integrate NetDetector with their own custom tools, scripts and utilities.
"It's a very intuitive product," says Nusbaum. "Looking at the pieces to reconstruct an event or activity that occurred on the system doesn't take a computer superstar to figure out how to do it. You really can have basic security and computer knowledge and be able to do very sophisticated things."
Text-based documents reveal patterns, if you know how to look.
SilentRunner is unique among the three NFAT products discussed in its use of N-gram analysis to parse the contents of files to determine possible relationships, which it then displays using its powerful visualization capabilities.
N-gram analysis has long been a field of study in computing automation theory and natural language processing. N-gram analysis is a method of breaking up text-based documents into n-number long character words. The statistical similarity of occurrences of N-grams in the source texts ultimately leads to similarities in the source documents.
For example, a document about coyotes will have occurrences of the tri-grams "coy," "oyo," "yot" and "ote," which will be uncommon in other documents. Through N-gram analysis, you can determine the similarity of many documents by looking for a statistically significant number of matching N-grams.
Not-so-in-depth analysis. Although NetDetector's user interface was much simpler to operate than SilentRunner's, its analysis capability is limited to examining packet flow between hosts through the Web interface. You can quickly see what protocols are in use, what hosts are talking up a storm, and which pairs of connections are more popular than others. However, it takes a while to drill down to the packet level.
NetDetector's analysis function, which starts by selecting a time frame to examine, is most useful when you know what you are looking for (e.g., activity from a particular IP address or traffic using a specific protocol). TCP streams can be reassembled into the resulting FTP or Telnet sessions, for example, and then viewed in plaintext.
The interface makes ferreting out anomalies a challenge -- the information is presented as line upon log file-style line of traffic. While the ability to create the various TCP streams is helpful, the Web playback feature lags far behind SilentRunner and NetIntercept.
NetDetector comes nowhere near the ability of SilentRunner to graphically represent network traffic, which is significant when you don't know exactly what you're looking for. Although Net-Detector does provide the ability to click on the individual log entries for more detail, SilentRunner visualization eliminates the need to plow through all those lines of text.
During the analysis phase, DNS names in captured traffic can be resolved offline using the management interface -- a nice feature for minimizing network noise and avoiding tipping off a potential intruder.
How alarming. Unlike the other two products, NetDetector has powerful alerting capabilities. Alerts must be manually configured within several categories: utilization, TCP count, invalid address, host flood, host scan, host pair bytes and port scan. The alerts can be used to warn of distributed denial-of-service attacks, IP address spoofing, broadcast amplifications, port scans and to identify the use of unknown protocols.
In addition, the threshold and time frame are also configurable to allow extremely slow port scans to be detected, for example. This capability can also be useful to allow notification when traffic is seen from a particular IP or MAC address. Users can be alerted by popup screen message, e-mail, cellphone, pager or SNMP trap.
Packet capture. The NetDetector appliance is installed on a target network segment and supports almost every network interface and method imaginable, including T1, Ethernet (10/100/Gig-E), FDDI, PPP, frame relay and OC-3, with support for OC-12 in development. NetDetector can even capture traffic on multilink PPP T1 lines, then reassemble the information for analysis.
The product also can capture traffic in several ways, including taps, splitters and mirrored ports, depending on the network interfaces used and your network architecture.
The tool employs the Berkeley Packet Filter (BPF) syntax for filtering traffic, along with a Network Associates Sniffer-like interface.
NetDetector continuously records network traffic into a circular buffer database, which is large and scalable. The base system starts with 150 GB of storage, which we were told is about two weeks' worth of full capture for a "typical" customer on a "typical" T1 connection. Data can be archived and exported in several formats, including a proprietary Niksun format, Sniffer, libpcap and TCPdump. When the storage on the appliance fills up, the statistics remain, but the oldest data will be overwritten. Data must be offloaded from the appliance using FTP, HTTP or SCP. NetDetector, unlike NetIntercept, does not have an integrated CD-ROM-burning capability.
Feeling a bit insecure. The fact that NetDetector's background is that of a network and performance company rather than a security company was apparent in a few instances.
Although the NetDetector appliance supposedly supports management over secure connections using SSL and SSH, the demonstrations we saw showed only HTTP, using basic authentication and Telnet. The sales engineer said that secure connection had to be turned on, but he didn't know how to do it. Although the system does include an easily configured firewall that can restrict access to the management interface, it's only necessary because of the insecure protocols running by default, including Telnet, FTP and HTTP.
One-day NetDetector training is available at the company's New Jersey facility. Niksun says that customers need only a few hours of training to become familiar enough with the tool to hit the ground running.
NetDetector's price ranges from $20,000 to $80,000, depending on the amount of storage and the type of network interfaces.
Sandstorm Enterprises' NetIntercept
NetIntercept, released last October, is the new face in the NFAT lineup. NetIntercept is a solid entry-level contender that provides a nice balance of price, network traffic capture performance and forensics analysis capability, all wrapped up in an appliance that was designed with security in mind.
NetIntercept identifies content by actually looking at it, rather than relying simply on the port it ran on. For example, the appropriate content type would identify Web traffic on port 25, or an FTP server running on port 2002, even though those aren't the standard ports for those services. This also enables alerts on packets whose content did not match the protocol type. For example, if someone were tunneling out a Telnet session over HTTP, the alert would be reported.
Unlike NetDetector, NetIntercept lacks robust alarm capabilities. It has an alert function, but the user can only set priorities for a limited set of choices. Alerts can only be viewed from the section of the interface in which they are set, and there's no remote notification capability.
A major plus is the way security was designed into the product -- in contrast to security questions that may haunt NetDetector and possibly SilentRunner. We especially liked that remote management is performed only through SSH on port 22 and how the listening interface is hardened. NetIntercept won't broadcast a single packet -- it won't even respond to ARPs on the local subnet.
NetIntercept only supports fast Ethernet network interfaces, since Sandstorm envisioned the product being deployed near a firewall or in a DMZ. Sandstorm did performance testing using all BSD flavors and a variety of Fast Ethernet cards and found the best capture performance using a combination of modified FreeBSD with Intel EtherExpress Fast Ethernet cards. NetIntercept can import network traffic data from TCPdump, Net-X-Ray and LanWatch.
Drilling down. NetIntercept's intuitive user display easily shows session-level activity from the network, including views at the IP, TCP and session levels. The NetIntercept desktop can be toggled among several tabs, including traffic capture, forensics and summary. Users of Sandstorm's first product, PhoneSweep, a modem-scanning tool, will immediately recognize the look and feel of the NetIntercept user interface.
The display shows a graphical line chart representing the number of packets captured over time. NetIntercept can determine DNS information based on traffic that was captured, and doesn't have to do active DNS lookups on the management interface like NetDetector. The ability to turn on and off the active DNS lookups is a useful tactic to avoid discovery of NetIntercept on your network.
While analysis isn't as sophisticated as SilentRunner's, we found the forensic tools to be useful and intuitive to operate. Like NetDetector, NetIntercept requires a block of time to be selected for analysis, so it can reassemble packets into TCP streams. All three tools provide this protocol reconstruction capability, which is quite useful in playing back traffic to review network sessions.
We spent most of our time using the forensics tab (see screen, right). We would enter a keyword, or select a criteria or combination of criteria -- for example, port numbers and their associated services, IP addresses (source and destination), username and content type (AOL_IM, ASCII, HTML, etc.) -- each of which would be displayed in a column. We could drill down for more information by clicking on individual entries.
Like SilentRunner, NetIntercept can extract and display all the images files that traveled across the network. This can be revealing if you want a quick overview of the types of Web pages and images that are circulating.
Archiving features. One of the differentiators between Net-Intercept and the other products is its built-in CD-ROM burner. This allows data to be archived directly from NetIntercept user interface, which may be useful in a forensics investigation. In addition, forensics markers (placeholders) can be set when examining network traffic to help keep track of large amounts of data that may be examined. SSH can be used to securely transfer data from the appliance for archiving by other media (e.g., a tape drive).
Like NetDetector, NetIntercept uses a circular buffer. When the hard drives are filled, the oldest data is automatically overwritten. However, if a new analysis session is started, that session is retained indefinitely. NetIntercept utilizes "file objects" for efficiency. If NetIntercept sees the exact same item -- such as a graphic file on your own Web page that people are constantly loading -- it references it instead of saving it multiple times.
Training is informal, and available on an hourly basis. Technical support is available weekdays, 9 a.m. to 5 p.m. EST.
NetIntercept starts at $15,000 and includes 80 GB of storage with Fast Ethernet interfaces. The "plus" version costs $21,000 and increases storage space to 300 GB. Both configurations come with dual processors and dual NICs.
If you're limited by budget and are content to capture traffic on Fast Ethernet segments, NetIntercept is a choice worth considering.
One of the potential issues that will arise from the ability to capture all network traffic is how to manage and archive the captured data. At least one company is developing a network traffic capture device with this in mind. A product like this will have full network capture capability in addition to supporting fully automated streaming of all captured network traffic data to disk storage and then to tape storage without any manual intervention. The technology to do this will utilize storage area networks (SANs), Scalable Coherent Interface (SCI) and tape robotic library hardware.
While NFAT products fill security gaps left by IDSes and network monitoring tools, they are all somewhat immature. Ideally, we would like to see a tool that does it all:
- Captures network traffic indefinitely on multiple interfaces without missing any data.
- Performs advanced graphical analysis.
- Sends alerts based on user-defined events.
- Employs secure remote administration.
- Archives data.
In addition, since we're still dreaming, it would be reasonably priced and easy to learn and operate.
Until this dream tool exists, the best solution is to determine your specific needs and objectives and choose the tool that's the best match. If an organization had enough money and resources, the most powerful solution might be to use NetDetector to capture network traffic data and then use SilentRunner operated by a dedicated analysis guru for the powerful analysis capability.
About the authors:
Nate King is a managing consultant with Predictive Systems' ethical hacking practice.
Errol Weiss is currently the vice president of technical services at MSSP Solutionary. At the time of this writing, Weiss was vice president of services strategy for managed security at vendor-neutral Predictive Systems.