How to analyze internal security threats: Network security from the inside out

Learn to effectively assess threats from internal clients, detect anomalies in outgoing traffic, architect networks to resist internal attacks, respond effectively when attacks occur and more.

About the speaker:
Richard Bejtlich is the founder of consulting firm Tao Security.

Return to the Data Protection Security School lesson, Preventing data leaks.

Read the full transcript from this video below:  

Securing your network from the inside out

Eric Parizo: Hello and welcome to our presentation, "Extrusion Detection and Prevention:  Reassessing Web Protection Amid the New Threat Landscape," with guest speaker Richard Bejtlich. This is part of's Data Protection Security School, which you can visit at any time via My name is Eric Parizo and it's great to have you with us.

In the past, traditional enterprise web security has placed an emphasis on identifying and thwarting threats that originate outside the perimeter. However, today's threat landscape shows that information security threats can just as easily originate from inside an organization. This presentation will help enterprise security pros learn to assess threats from internal clients, detect anomalies in outgoing traffic, architect networks to resist internal attack, and respond effectively when attacks do occur. 

Our speaker today Richard Bejtlich is the founder of Tao Security, an information security consulting firm based in Northern Virginia. He is an expert on data protection and integration leakage, having authored several books including, The Tao of Network Security Monitoring and Extrusion Detection, and co-authoring the book Real Digital Forensics. He's a frequent speaker at industry events, is perhaps best known for his popular TaoSecurity blog. Thanks for joining us today, Richard.

Richard Bejtlich: Thanks, Eric, happy to be here.

Eric Parizo: After the presentation, Richard will answer a few questions in a brief Q&A segment. And now, without further ado, ladies and gentlemen, Richard Bejtlich.

Richard Bejtlich: Thanks Eric. I'd like to thank everyone for attending this morning and I hope that you enjoy, hope you find something about it useful. Before I get started, I just thought it would be useful to give you a sense of what my background was, so you could get an idea of if I've dealt with these sorts of problems before. I worked for a variety of organizations with military, government flavor, also commercial and over the last two years I've been doing consulting for pretty much every industry that you could imagine. I cover the whole map, in terms of, who's out there.

This issue of finding and potentially stopping what is leaving your company or your organization is one of the top three items I hear from all of my clients. Everyone seems to be worrying about it these days, some people approach it from the perspective of, it must be an insider threat issue, other people see it as an issue of, "I've got intruders who can get into my company. What do I do now to try to remove them," and so forth. I've written about this as well, in 2005, I published a book called, Extrusion Detection, and the idea behind that book was to talk about ways that you could look at what was leaving your organization to find machines that were compromised, and we'll use some of those ideas today, and then we'll also talk about other terms that are being used to discuss this problem.

Just to give you an overview of what we'll be discussing, I'd like to start off first by talking about the problem, because while this is a hot topic, many people immediately jump to some sort of a tool based approach and they don't even spend the time to figure out what it is that they're trying to solve. After that, we'll talk about the importance of having a policy and a policy is needed because if you don't even know what an incident is, it's very difficult to try to architect a solution. After that I'll talk about an idea called the Central Network Architecture, and these will be some simple steps or simple concepts that will give you the best chance of resisting intrusion and dealing with it once it happens.

After that, I'm going to talk about the three major approaches that you'll see associated with extrusion type approaches, namely, retention, detection and resistance. Each one of those escalates the involvement that you have with potential intruder and I'll talk about some ways you can approach those. After that, I'll talk about ways that some of these products are specialized and I'm going to give a little attention to insider threats. I'm also going to talk about some open source approaches, including a look at a tool called Flop that Michael Zaluski wrote. I'd just like to give you this as an idea of some other ways that you might approach this problem. Then finally I'll talk about some commercial products and what they might be able to do for you.

Well, let's start out talking about what the problem that we're trying to solve is. You can see there in the diagram that I've written that you've got these concentric rings of resources that you care about. You start out with a piece of hardware, on top of that hardware there's running an operating system, whether it's an embedded operating system or a regular general purpose operating system, above that you have applications, and the applications and the operating system and hardware are all there to support the processing of data.

If you think about classic information security principles, the events that we're trying to avoid as a security professional are a disclosure of information, the modification of information or a denial of service against information. You think about those three items, they're the inverse of the confidentiality, integrity, availability triad that we've all come to appreciate as being a core of security. What we've found over the last few years is that people who deal with these problems realize that securing hardware and operating systems is not sufficient. I put "securing" in quotes there because, there's really no such thing as a secure anything. Every level of security is simply relative to something else, so there is no point which you're secure. No one is secure today, no one is secure tomorrow. It's simply a state of accepting a perceived level of risk.

Because we've seen a proliferation of connectivity and devices that process data, we find that people are realizing that it's just not simply sufficient to protect hardware, to protect an operating system, or even to protect an application these days. It seems like all the defensive measures that people are talking about are crashing down around the data and this presents some interesting problems because data doesn't exist in a vacuum, it's processed by an application, it resides in the file of some kind or perhaps it resides in memory before being written to disk.

So the idea of protecting data sounds nice, but in terms of actually implementing, it could be very difficult. That's why we seem to see this once removed problem of watching data as it's transferring across the network, so the data in motion problem, as opposed to data at rest. For example, encrypting file systems on laptops, but if the laptop is stolen then you're not quite as concerned about the disclosure of the data to an authorized party because it's in an encrypted form. We're going to be talking today about the data in motion problem, and specifically data in motion across a network.

Whenever I do consulting or I teach, I always begin with some sort of issue of policy. Just this morning I was idling in my favorite RST channel, I idle on several, one of them is the Snort Channel and someone dropped into the channel and said, "How can I use Snort along with Squid?" We know Snort is an open source intrusion detection system; Squid is an open sourced web application cache and proxy. The person had jumped straight into the tool approach, and I said, "What do you want to do with these?" The person said, "I don't know. I simply want to know how I can use these on my network." I said, "Do you have a policy?", and the person said, "No."

You have to figure out what it is you're trying to do with your company, and what it is that you care about and what you're allowed to do on that network. If you go straight to a tool based solution, then you really have no standing to evaluate whether the tool could even address something that you care about. I think this might be one of the reasons why so many people buy products, and then several months or maybe a year later, they're very disappointed with the way the product has performed. They didn't go up front saying, "What does our enterprise look like? What is the problem we're trying to solve? Can this potential tool or tactic address that problem?"

Instead they hear that extrusion detection is a hot topic, they look at products that claim to do extrusion detection, they talk to a vendor and the vendor sounds good, they bought a product and then several months later, they're wondering, "What did we even get this for? Is it working to the way we expected it to?" I recommend that the first thing you do is figure out what your policy is. What is an incident? Just as a sample, in this case, you may have a policy that says, "An incident would include disclosure of information to a party that's not authorized to obtain it."

That would be a violation of confidentiality. You could just as easily have an incident be defined as the modification of information by a party that's not authorized to interact with it. Of course, there's the denial of service option, as well. Within that space, you're going to hear of many terms associated with this problem. I tend to use the extrusion problem, honestly you could just as easily say one of these others, for example, intellectual property leakage is a term you might hear. Data exfiltration is another one, constant exposure, the idea that you are exposing information to people who shouldn't be seeing it, and finally, information disclosure, or information breach. Any one of those things is usually associated with the idea of information leaving your enterprise.

Just as a brief aside, I find it really humorous that there are many people out there who are talking about the deperimeterization of the enterprise, for example, the group called the Jericho Forum, and I've blogged about them in the past. They believe if you remember the story of Jericho, the walls of Jericho came tumbling down and they believe that's what should be happening, or is happening with the enterprise. I would agree that that's definitely the case, but unless you, as your organization, own the whole Internet, there is a point where you own something and someone else owns another piece of equipment.

That by very definition defines a perimeter of some kind. I think it's funny that there are groups who are pushing this idea of deperimeterization in an age when we are struggling and scrambling to put products in place at our perimeters so that they will watch as this information leaves. You either got to accept that there is a perimeter and there is a place where you could potentially control this flow of information, or you should just completely abandon the idea of a perimeter. Don't even worry about these products because, if you don't believe there's a perimeter, there's no difference between you and anyone else on the Internet and you need to look at some other means of controlling your information. Perhaps with Enterprise Rice Management, which is a subject that Information Security Magazine covered very well several months ago.

To sort of conclude with this thought, what I recommend that you do is, the very first thing is to recognize, what are the business operations that your organization follows? Why are you in business? Secondly, just figure out what is the data that is required to support those operations? Then figure out what the systems are that process that information. By system, I don't mean necessarily a specific computer, I mean the financial system, the engineering system, whatever. Then once you have an idea what all those pieces are, and what matters to whom, then you define what an incident is. It's an incident when one of our sales people sends an entire list of customer contacts to a competitor. It's an incident when credit card numbers of people who buy our product is sent to a drop off site in Russia, and so forth. You can think of this as well as incident modeling or incident scenario development and it's a good idea to have all those sorts of ideas developed before you actually have the incident.

I'm going to talk now a little bit about defensible network architecture. This is a concept that I wrote about in my first book about network security monitoring, and I expanded upon it in Extrusion Detection. The idea behind defensible network architecture is there are certain steps that you can take that will give you the best opportunity to resist being compromised, that includes being compromised from the outside in or from the inside out. I really don't recommend people ever go out and buy any new equipment in any situation until you have an idea of what you have existing in the organization and whether or not you're using it to the full capability.

For example, any one that's got a Cisco router has a certain capability to do layer seven detection and restriction of malicious traffic. Ever since the CodeRed worm of 2001, this goes to talking about the ability to deny something like that using iOS, and network based access controls. When I say that, it's not Mac, as people talk about Network Admission Control these days. It's an older term, but it's a capability that's been in iOS for six years now. It's a good idea to figure out what you have, and figure out whether what you have right now could potentially address the problems that you have if you just reconfigured it properly or configured it in a different manner. The elements of defensible network architecture are these:           the first thing you should do is monitor what you can with what you've got right now.

Every good security implementation or every implementation that would be good, people don't always do this, but if you follow this, you will have a good implementation, begins with figuring out what's happening. If you're going about implementing solutions or adding products to an enterprise and you don't actually know what's happening, then you're doing something what I call soccer goal security. An apology to my Europeans, and everyone else in the world who says that soccer's football, but in terms of what I'm talking about here, soccer ball security is a case where you are defending a certain part of the goal, for example, the left side, and meanwhile the other team is scoring on the right side, because you're fixated on one part of the goal, you're not even seeing what's happening on the other part.

This happens all the time with enterprise networks. People are so concerned about a certain type of attack that they may have heard about in the news, that they read about in a magazine, and yet, something else completely differently could be happening within their own enterprise. The only reason they don't know about it is because they're not looking. I always recommend that the first thing you should do is monitor, and I've written an entire book on how to monitor, but there are ways you can do that.

The second thing I recommend that you do is you implement whatever methods of control you can, and by control I mean blocking, proxying and throttling. Blocking both inbound and outbound, essentially everything that you don't need, so you only allow in what you need and you deny everything else. Proxying, I recommend especially on the outbound. So if you're allowing outbound web traffic, I recommend that you run that through a proxy. If you're allowing outbound whatever else kind of traffic, if there's a proxy for it, I recommend that you apply it. Proxies are good for two reasons; the first reason is they tend to keep detailed logs so they can be configured to keep detailed logs. When you have those logs, the analysis of them is much more efficient than having a network based product try to figure out what's happening. I always recommend a proxy for that reason, or reasons.

Secondly, the proxy if it's application aware, it can imply a certain level of enforcement to the protocols it sees. If somebody tries a lame attempt to send a non HTTP protocol over port 80 outbound, the proxy will see it and deny it, and now you've got an indication that something bad might be happening.

Finally I recommend throttling, and the reason for that is if you do allow certain services outbound, if you don't implement some type of throttling, then it's possible for a perfectly legitimate service that's being used in an illegitimate way to consume some or all of your bandwidth, and that's just as bad as potentially disclosing in some other cases. For example, email. If you were to allow anybody to go out to port 25 to any server on the Internet, you could have an internal machine that becomes compromised, turned into a spam system and suddenly the thing is using up all your email, sending out spam. Throttling, while it's not something I would recommend as your only defensive measure, you should use it as a means to hold the fort until you figure out what's causing that internal problem.

Monitor, control, the third step of the defensible network architecture is to minimize by removing unnecessary client server applications. You can't have someone running a windows box, look at a malicious movie or malicious sound file, if they don't have the application to read those sorts of things. We do have web browsers who are getting more and more capable of, Adobe Flash can do just about everything these days, but if you don't have the application on the box, it's much more difficult to be compromised by it.

Finally, everything that you have that's left over that is something that should be allowed, keep it up to date so that you avoid the exploits for which there are already patches out there. To sum up for that, defense network architecture, it gives you the baseline you need to do the best you can resisting compromise, and reduces the  number of channels that you have to watch. If you allow absolutely anything inbound, absolutely anything outbound, and most people obviously don't allow everything inbound, but many people still do allow a lot outbound, if you're in that situation, it's basically impossible to see what doesn't belong and to stop what shouldn't be there. If you reduce those channels down to something that's manageable, you'll have much better success on the extrusion detection or prevention side.

Now I'd like to talk about three different approaches to this extrusion problem. The first one I call retention, and the first step is to record what is leaving the company. You might record what's entering the company but in a context of the extrusion problem we're talking about today, this talks about recording what's leaving the enterprise. You might say, "I don't see the value in that. I want to stop everything." The reality of the situation is there are people, and there are processes and there are tools that will absolutely evade any type of detection, or any type of prevention system that you could devise.

Any way up to the most advanced covert channel as an extreme example, but the idea is that if you are simply keeping track of that information, at some point in the future, when you realize that there has been an intrusion, there is some sort of a problem, you can go back to that retention product and say, "Hey. What can you tell me about this type of traffic?" This is exceptionally valuable and I think we're going to be seeing it more and more in the future. I've written some blog posts, one of which is, I called it "The Revolution Will Be Monitored," and the idea is that we are moving more and more to having everything about us recorded. This isn't a novel idea, but we're actually seeing it implemented technically on networks now.

Yesterday, as a matter of fact, I posted a story about CALEA which is the Communications Assistance for Law Enforcement Act, which basically mandates that all ISPs, both broadbands, wireless, anyone who provides internet access essentially that's not a company providing access to their own employees, is required to give law enforcement authority the ability to access traffic in and out of their network. This is mainly affecting ISPs and wireless ISPs. We're getting to the point where everything's going to be monitored, and while that's not very good from a privacy standpoint, if it's done appropriately, legally and in a sensitive manner that it best balances privacy against the needs to defend the enterprise, you can have a better idea of what's going on in an intrusion.

For example, just this morning I read about the testimony that was given at a house hearing on cyber security where there were representatives from the State Department, the Commerce Department, and the problem that was really evident in both of those statements, or at that hearing, was that it's difficult to know when an intrusion first began. So if you're never collecting information about what's happening, then, you can't figure out when the intrusion first started.

I list a couple of products there that are in the commercial space, that are what are called network forensics products, namely NetWitness, Infinistream from Network General, NetVCR from Niksun, NetIntercept from Sandstorm and GigaStor from Network Instruments. These are all boxes that you can buy; I should add Solara in there because they just came out with a CALEA plan. These are all boxes you can apply to the network that will collect traffic, a lot of it. It's important to realize that you don't necessarily have to capture every packet, there's a difference between collecting packets and collecting useful network information.

I break it down in my network security monitoring methodology by saying that you have alert data, you have session data, you have statistical data and you have full content. Most people simply think in terms of full content and the difference is there's a middle ground there where you can just collect session. Sessions are records of who talked to who, when, for how long. To me, session information is the real goal there. As far as stand alone session programs, Argus is probably the most famous of these. Argus is available at and there's no U in there, it's an intentional spelling.

Once we get beyond the issue of simply keeping track of what's happening, what's left leaving enterprise, and I recommend that as a baseline step for everyone. If you go into a vendor situation, say, "I need such a product," but you don't even really know what's happening on the network, then you're in a compromised situation. I recommend you at least figure out what's happening first so you have an idea of what you want to block. You want to try to inspect or block Word documents, or database traffic, and so forth.

The second approach that you'll find with the extrusion problem is that of detection. That is the idea of not just recording traffic, as it's going in and out of the network, but figuring out what is actually there. Is there a way to detect the release of information that you care about? There are two broad categories of techniques that you'll see. The first one is looking at the nature of the traffic. In other words, what sorts of protocols or services are observed, what's the general badness level of what you're seeing?

For example, are you seeing outbound ISP traffic that you don't expect? Are you seeing outbound encrypted traffic that you don't expect? Those are things that can be found without regard to content, in other words, we're not looking necessarily for a credit card number, we're looking for an encrypted channel that we didn't expect.

The second broad category of detection, involves looking for specific content and this is information like Social Security Numbers, personally identifiable information, credit cards, driver's licenses, spreadsheets, whatever you could imagine that might be important to you. This is why it's important to know what's your threat model? What is it you care about? Who's out there who could be trying to hurt you? Who would want to be stealing information from you? What form does that information take? If you don't know that and you simply go into a potential buying situation, you may find some vendors are really strong with certain forms of data, and other vendors are not.

All products that claim to do prevention should, honestly, as far as I understand, they do, provide detection as an option, because what you'll find is that, the detection issue is something that can be done passively or perhaps, at least, with a quarantine option, and if you jump directly to prevention, many people are leery of that because you don't want to be knocking down a business related traffic, necessarily.

The interesting thing about detection, by the way, several years ago I think it was in 2003, we may remember when Gartner said that intrusion detection systems were dead. They're definitely not dead, in fact all that's happened is they've been turned around so they're not watching what's coming in, they're watching what's leaving. I find that highly ironic, but the same technologies that were used to detect attacks against server side systems, has simply been turned around to inspect traffic as it's leaving your enterprise.

Going beyond keeping track of what's happening on the network, and then detecting what potentially could be traversing on the network, we finally come to what I call resistance. I don't call it prevention because prevention seems like it's an absolute term and in this realm, prevention is not 100%. I call it resistance; you do your best to avoid having these problems happen. What these sorts of products will do is they will see traffic, they will detect something that they don't like about it, either given a statement from you or something that they're built in to handle. Then they will interdict, terminate that traffic by a couple of different means.

If they're an offline product, they are not directly inline, meaning they're not between your border router and your firewall, they're sitting perhaps, on a tap or a span port or something like that. What these products will do is, when they see traffic that violates one of their policies, they will inject a spoof TCP reset, usually to both sides of the conversation to try to knock down a TCP session. If it's an EDP session or I should say an EDP conversation that's happening, they might try to send an ICMP destination unreachable to each side, to try to confuse the two end points. The other alternative is to use an inline product. Inline products will not allow the traffic that is seized, that violates a policy, to pass.

So inline products tend to be more effective. They do introduce a single point of failure unless you architect them properly, and what I recommend with inline products is you always deploy them with a bypass switch, and I have a net optics bypass switch shown there in the corner. I love bypass switches because you deploy the bypass switch inline, first of all, traffic will just go right straight through the bypass switch, period. You can then take an inline device, plug it into the bypass switch and the bypass switch will say, "Hey. I've got an inline device here. As long as that inline device is working, I will send traffic through it."

A good bypass switch will detect that traffic is no longer flowing through the inline device for some reason, for example, let's say it has some sort of operating system level problem or an application level problem, or maybe the power supply dies, for whatever reason, if the inline device fails, traffic will still keep passing through the bypass switch. I even recommend using external bypass switches over devices which shift with bypass switch capability in them. The reason is, if you ever need to swap out that box where you just want to remove it from the network that means you're going to have to move around cables, and you're going to have to schedule down time. Once you put a bypass switch into place, it can stay there forever, and you don't have to physically move it. So I always like having an independent stand alone bypass switch.

It's important to realize, as sort of a concluding thought with this resistance slide is that, any product that does enforcement blocking traffic, is going to suffer all the problems you have for the detection system. The only reason people will run a system inline actively, such that it's knocking down traffic, is if they have a high degree of confidence in that system. Usually that high degree of confidence only comes after testing it and potentially deploying it with a subset of the available features, or the available detection mechanisms that it has. I'd like to give you a few caveats about these products.

The first one, or I should say, the really important one is that every single one of these products can be evaded. Unless you have an incredibly strict and incredibly smart product that you tell, I don't care what the risk to knocking down business traffic is, I want you to kill such and such. Unless you deploy it in such a manner, you can get information out, and volumes of it. Even if you do approach such a product with that high level of restrictiveness, I don't care what product it is, you can still get information out of that enterprise. Like I said, with covert channels, you can devise a channel that will get information out of the enterprise, period.

The trade off that happens is that as you become stealthier with your covert channel, you tend to lose bandwidth. In other words, if I didn't want to be stealthy at all, I could easily transport gigabytes or terabytes of data out of the enterprise, but that's going to be pretty obvious. If I want to be exceptionally stealthy, the easiest thing for me is to vastly decrease the amount of bandwidth that I'm using, even to the point where I could be setting a certain bit in a DNS packet, for example, in order to communicate information to a remote site. You've traded extreme stealth for extremely low bandwidth.

This is just something you have to accept, it's the reason why I don't believe strictly in products that try to stop traffic or detect traffic, I always believe that you should have a forensic capability on one side and you should have a detection/prevention policy on the other side. Because at some point you may find, if you're dealing with an advanced intruder or simply a moderately advanced intruder but with an advanced tool set, you may find a case where something has happened and you didn't realize it until weeks or months later. If you've got some sort of forensic capability to go back, it gives you the idea that you can scope an incident.

Now we get to the slide on inside threats, and this to me, is probably the over hyped story of the last several years. What's funny about this is it happens in about five year waves. If you go back and look in the archives of Information Security magazine, you'll see that there were cover stories on insider threats in 1999 and 2000, and then it went away. Now we're coming back, the insider threat is such a big deal. To the degree that these extrusion products are good at keeping unauthorized material leaving your network, really doesn't matter whether they're dealing with an insider or they're dealing with an external party who has gotten a foothold on the inside of the organization and is now looking like an insider.

What I'd like to think is that if you have a dumb insider, the dumb insider is going to do something like, "Oh, I want to email this list of contacts so that when I leave this company, I can be a sales person at a competitor."' They're going to take that information, they're going to put it in an email attachment and they're going to send it off to their home email address, or they're going to send it to a Gmail address, or whatever the case may be. You're going to be able to stop that sort of activity with one of these extrusion products. But if you're dealing with a smart insider, they're going to completely avoid the network.

They're going to copy their sensitive data to a USB token or maybe even take screen shots with their digital camera and their phone and you're never going to see that kind of stuff. It's important to realize that, you've got to figure out who it is you care about in terms of the threats, and decide if your measure addresses that. I would argue that, while this is a technical presentation, in the sense that we're talking about capabilities, many times the non-technical means of dealing with insiders is best and I talked about that several months ago when I did a webcast for DNE Communications, and that's online, if you'd like to see that.

We've seen over the last two years, some specialization in the extrusion piece. I think that many of these capabilities are boutique capabilities that we're just going to see rolled into everybody else's products, so I don't necessarily think that you need to go out and buy a specific product branded for a specific problem, for example, extrusion against database communication, or instant messaging extrusion, or email, or web traffic, or whatever. Just in general, you're seeing a collapse of all network inspection and interdiction activities into what you might call security switch or secure router.

Richard Steaming calls this secure network fabric, other people have other names for it, or actually some people call them unified threat management devices, and I just blogged about that yesterday, as well. What you're seeing is that any device that takes action against network traffic or watches network traffic, all this functionality's being rolled into single boxes, and the firewall of today does a ton more than it did five years ago. The firewall of tomorrow is going to be doing all the things that we're talking about in this presentation, as well as, other capabilities that you see in stand alone products. I do see, however, a separate role for  network forensic equipment because it doesn't make a lot of sense to have a secure switch, or a security switch, security router with a huge set of hard drives in it that's trying to record all this network traffic.

I have been talking about products to a certain degree, and I wanted to talk about open source possibilities because it's not strictly necessary to go out and buy a product, if you figure out by doing your analysis that you have a specific problem in mind and you're trying to solve it with something that might be available for free, both in terms of source code availability and in terms of the price. I did mention earlier that an intrusion product is in many ways, just an extrusion product that's looking in another direction. You could potentially use something like Snort if you operate it, you could operate it as a detection module or if you wanted to run it inline, you could operate it as an extrusion product that is knocking down traffic actively.

Another option you might have is if you don't want to have something that's inline, but you do want to have something that interacts with your access control devices, you could run a project called SnortSAM. SnortSAM will talk to your routers and your firewall and will reconfigure their access control lists in real time. I should say, as fast as it can interpret what's happening from Snort. Another option that's listed there is layer seven filter, it also has ITT2P, and then finally Mike Rash's firewall Snort, which will build a  net filter rule set that mimics certain Snort rules.

I wanted to bring up another open source possibility, simply because it gives you a different idea of the sorts of things you can do when you're inspecting network traffic, and that's Flop. Flop is a passive layer seven fingerprinting tool that Polish researcher Michael Zaluski wrote. What Flop does, is it doesn't look at the content of traffic, but it examines the sequence of bytes that are transferred by a server and a client. It doesn't care about port numbers either. What it does, it is looking at the sequences of traffic that's sent, it can identify all sorts of interesting activity.

One of the great examples I like to use with Flop is that many times people might be running unauthorized Secure Shell servers. Secure Shell's probably the easiest way to create a completely arbitrary tunnel that can carry just about anything you want within it, and while there are pretty good signatures within Snort, for example, for detecting Secure Shell traffic on arbitrary ports, Flop offers another way that you can potentially address this problem.

What I did was, just to test Flop, I ran a Secure Shell, and I tried connecting to various systems, for example, in the first scenario that I have listed there, it was the first time that this system had tried to connect to this other system and it said, "Do you accept this key?" I said, "No." In the second test, it said, "Do you accept this key?" Then I said, "Yes." Then in a third test, I repeatedly entered the wrong password. With these tests, Flop was inspecting the traffic that was passing by and what it did was, it didn't understand Secure Shell, it didn't understand that this is an encrypted protocol; it wasn't looking at the contents of the packets. All it was doing was counting the sequence of bytes that were transferring back and forth, and because it had already seen Secure Shell traffic under various conditions, such as those that I just ran, but I hadn't programmed Flop, the developer had programmed Flop, it was able to identify each one of those cases and so, even though someone may have set up Secure Shell with a custom banner so they couldn't be detected with a signature, and they may be running it on a port that's not recognized as being a Secure Shell port, Flop was able to identify, "This to me looks like Secure Shell in these various scenarios."

Of course, once an intruder knows about this, they can simply modify Secure Shell to send different information, but the idea is that, looking for traffic using non-content means, could be valuable in an extrusion scenario. I thought I would give you just a little flavor of how Flop does this. Essentially, what it does is, it counts bytes of traffic sent by the server, sent by the clients. And with certain maximum transmission unit sizes and certain delay. The details are there, if you attend my TCP/IP Weapons school class that I'll be teaching at Black Hat this summer, I go into detail about how that works and we demonstrate it in class.

As we get to the end of the presentation, I wanted to mention that there are many, many players in this extrusion space. I mentioned several: Fidelis, Reconyx, Versup, Tablis, Oakley, Vertices, McAfee, Provel, there's been several others that are out there as well. Everyone who has a product that inspects network traffic has figured out that it's not too difficult, that if that product was looking on the inbound, turn the thing around and watch the outbound.

What I recommend that you do is follow the steps that were first in the presentation. Figure out what it is that you care about, what is the information that your network carries, what are the scenarios you could envision where if something bad happens, you want to know about it or potentially stop it, and then see how those scenarios run up against these vendor products. Give the vendor products a test.

With that, that concludes the formal presentation that I had. I hope that, in the course of the presentation, you got a better sense of what you might want to think about when you're looking at this extrusion piece. This is definitely a problem, I'm actually very gratified to hear that so many people are out there who are trying to figure out what's happening on their networks, and they're trying to do something about what's leaving their companies. What I would caution you though, is that you've got to come up with a realistic idea of what is happening, simply by passively watching, for example, and then see what you have that's existing that might be able to deal with some of these problems, and then if you find that there are still gaps in your detection profile, talk to a vendor, see if they can apply one of their solutions to your problem set.

Eric Parizo: All right. Great presentation, Richard. Thanks so much. I have a few questions for you. First off, going back to the beginning of the presentation, I was fascinated by your emphasis on policy over products, just because the way you presented it seemed so simple and logical. One thing I was wondering though, to that end, does that kind of policy based strategy often lead to enhanced or new policy management products?

Richard Bejtlich: That tends to be the case when you've got very large companies, in fact, it would be very nice if there was some sort of a product that said, "Here is the security policy," and then tried to explicitly map those policy elements to the defensive elements, or the inspection elements in the network, to see if you could actually do everything that you're trying to accomplish. Here's the deal. There are people out here right now that have policies on their network, whether they're written or not, it's encoded in things like their firewall access control lists, the signatures that they've got enabled on their intrusion detection system, or their intrusion prevention system, so everyone has a policy. The question is whether or not it's actually the policy that you want to enforce. It would be nice, there may be products that do this, I'm just not aware of them, that explicitly map a policy to the enforcement mechanism.

Eric Parizo: Do you think that's eventually where the policy management market will wind up?

Richard Bejtlich: I would hope so. I think I could think of several clients who, if they had something like that, it would make them pretty happy.

Eric Parizo: Moving on, I know when you spoke about data retention and the importance of keeping and using session data. In your experience, is it a significant shift in strategy for organizations to do that? For those folks listening, who haven't really thought of data retention in those terms before, is it difficult to make that kind of shift?

Richard Bejtlich: I do find that many people approach the network as a black box. In other words, they say, "As long as the things are working, I really don't care what's happening."' What I found is that, that is just a recipe for disaster because if you don't know your network, an intruder's going to know your network before you do. When that happens, you end up in a world of hurt. I like the session approach, simply because it tends to be a low impact yet high value proposition that can be done, even in the most dire circumstances, you can do it with re-purposed hardware and open source software.

You can get something working that would give me, for example, as an investigator and incident responder, some very high quality data that I could actually do something with, as opposed to having me show up and have to start collecting data on my own. I am seeing enterprises realizing they have to do more in terms of figure out what's happening on the network, but of course, they're trying to balance that against privacy issues. I think session data is a good place to start because it doesn't show you what someone is saying, what someone is posting and so forth, but it does tell you where the systems are going. At least you have that level of information.

I should say that session data, if it is implemented, is not sufficient. In other words, if you see an outbound connection on port 9000, through a box in Toronto, it might be nice to know what's in that port 9000 connection, so just having the ability to, even in on demand scenario, be able to watch arbitrary traffic, the full content of it and, of course, in accordance with your wire tap laws and such, because that tends to be considered a wire tap, the ability to collect that full content information when you see a session that looks interesting, pairing those things up is really powerful.

Eric Parizo: Richard, I wanted to touch on exfiltration, for a minute, as well, because it's certainly an interesting topic worth exploring. Can you explain just how intruders conduct exfiltration?

Richard Bejtlich: It all depends on the skill level of who you're dealing with. If you're dealing with a person who's inside the company and they're non-technical, they know how to use email and they're going to send something out using email, and to the extent that email's not encrypted, it's trivial for someone who is inspecting that traffic to say, "Hey. Why does this email have an attachment that says called Customer Contacts 2007?," and potentially knock that thing down. Once you start dealing with more sophisticated intruders, you're going to encounter encryption, you're going to encounter, potentially, channels that try to stay under the radar in terms of the amount of data they're sending out at various points in time.

You'll run across anything from encrypted channels using standard protocols like Secure Shell or HTTPS or I should say SSL, I guess, but over HTTPS, to perhaps custom protocols that are developed. There've been various malware pieces that speak IP but they don't speak GCP and EDP, ICMP, they speak IC protocol 11, as was found by the Honeynet Project, several years ago. One of the interesting points about that, though, is that, for me, things that don't look like normal traffic, to the extent that they're different, that's easy to find. The most difficult thing for me would be to find someone with an outbound HTTPS traffic, that could be anything. It could be someone just visiting a site that requires HTTPS, but it could be also carrying something that I don't want to leave the company.

Eric Parizo: Richard, can a security professional use the products they have as part of their infrastructures today to prevent exfiltration, or does the problem require a new type of product?

Richard Bejtlich: I definitely recommend that, if you have bought any new gear, actually any gear you have probably has some capability to address this problem. Especially if you've bought new gear in the last two years, there are so many features being equipped into just standard things you would take for granted. For example, two years ago I read a book called "Cisco Router Firewall Handbook," and that isn't a mis-statement. The book, I think the author's name is Richard Deal, but the book essentially it's a Cisco press book, and it talks about, look at all the things that you can do with IOS on your Cisco router. It was really fascinating to see all of the capabilities that are built into iOS.

If you were to enable all these things, you could potentially kill your router, because given the amount of RAM that it has and so forth, it might not be up to speed to be doing all this different activity, but if it's sufficiently equipped, you can do some pretty amazing stuff. People tend to think in terms of access control list technology that they learned in 1996. There's so much more that's been added to iOS since then that it's just waiting for us to use. You may find that there's lots of capability if you just pull out your iOS manual or whatever your router or switch center might offer you.

Eric Parizo: Richard, here's our final question for today. Do you think the exfiltration problem can ever truly be solved?

Richard Bejtlich: I would say no. There is no ultimate solution to this. My background is as an Air Force intelligence officer, it's the same issue that has been confounding intelligence agencies since there were such things. The idea that you're trying to stop someone potentially, at least as far as the insider goes, you're trying to prevent someone on the inside from getting information out, there is always a way to do it. The ultimate case of getting information out with potentially no one knowing about it is sending someone into an organization that has an exceptionally phenomenal ability to remember information. They've got a photographic memory. There is no way to prevent that level of exfiltration.

Honestly, many times the best way to prevent these sorts of problems, if you're dealing with insiders, is through non-technical means. The idea that a product can solve all your problems is not necessarily accurate, many times it's the, I like to say in some cases, human resources is your best weapon against the exfiltration of the extrusion problem. Of course, if you're dealing with a Romanian who has broken into your company, is now pulling data out, that's a completely different problem set and the two ways I approach that is, one, we need to encourage our law enforcement and military and give them the tools that they need to go find, apprehend, prosecute, incarcerate those sorts of people, and then, two, do what we can on the vulnerability side to make it more difficult for those people to get into our networks and then to control their ability to maneuver within the network, once they are there.

Eric Parizo: All right, very good. Richard, thanks so much for your insights.

Richard Bejtlich: Take care. Appreciate the invitation to speak.

Eric Parizo: Once again, we'd like to thank Richard Bejtlich of Tao Security for joining us today. For more information, read Richard's exclusive companion tips on conducting a network forensics exam, and be sure to check out more of our comprehensive resources in our Data Protection Security School at And thanks to all our listeners for joining us. I'm Eric Parizo. Stay safe out there.

View All Videos

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.