Manage Learn to apply best practices and optimize your operations.

IT patch management best practices: Overcoming the challenges

With targeted attacks and zero-day vulnerabilities shrinking the window of time between vulnerability disclosure and exploit availability, it’s becoming more incumbent on security managers to understand the assets in their IT environment and the patch levels of those machines.

In this presentation on vulnerability management and IT patch management best practices, application security expert Diana Kelley explains how to improve your asset discovery processes, determine the patch level of the machines in your environment, and improve testing and deployment processes to keep pace with patch and vulnerability management.

In this presentation, Kelley discusses:

  • Patching and remediation as a component of the vulnerability management lifecycle.
  • Implementing a vulnerability management program including scanning and prioritization.
  • Remediation: When and what to patch:  
    • Testing
    • Deployment
    • Validation
  • Remediation alternatives.
  • Keys to successful configuration and patch management lifecycle.

About the presenter: 
Diana Kelley is a partner with Amherst, N.H.-based consulting firm SecurityCurve. She formerly served as vice president and service director with research firm Burton Group. She has extensive experience creating secure network architectures and business solutions for large corporations and delivering strategic, competitive knowledge to security software vendors.

This presentation was originally recorded Oct. 14, 2010.

Read the full text transcript from this video below. Please note the full transcript is for reference only and may include limited inaccuracies. To suggest a transcript correction, contact [email protected].  

IT patch management best practices: Overcoming the challenges

Diana Kelley: Hello and welcome to the presentation on patching and
vulnerability in configuration management. My name is Diana Kelley. I'm a
partner at Consultancy SecurityCurve and a 20-year veteran of the IT
industry. Thanks for listening in. So to get started, as you noted, this is
going to be patching and vulnerability in configuration management, but
what does that really mean?

Well, looking at patching overall, it's actually just a component of the
total vulnerability management life cycle; because patching is one piece of
the process, but it isn't the entire process and if you just think of it as
an atomic unrelated piece you're not going to be able to do it as
effectively and you're going to possibly actually add overhead to the
entire process, so you may actually lose some pieces that you need to patch
or be aware of or to control, as well as add additional time. We're going
to talk about making it a component rather than an atomic piece of your
program. Then obviously implementing that program itself. Remediation, what
you do, you found a problem, now what are you going to do to actually fix
it, because it's not always patch. It would be nice if it was always patch,
but there are a number of reasons why patch may not always be the best fit.

So what do you do if you can't patch right away or sometimes you can't
patch at all and that will be the alternative. Then finally, we'll wrap up
with some of the keys to success, how you can actually build a program, if
you've got a program in place -- how you can strengthen it, or if you're still
developing a program, you can use the pointers on how to actually make that
a real program for you. So overall, as I had mentioned, patching is really
part of the bigger picture. It's something that is a process, but not just
a one-time event. It can be easy to think when we say patch, ‘oh apply the
patch;’ it's kind of almost like a band-aid in a lot of our heads. Yeah, it's
broken, I patched it, I band-aided it, I glued it together, it's better.
It's more of a story and the reason is that vulnerabilities don't occur
just individually, in terms of how they relate to the rest of the ecosystem
or the rest of your network. So you may have a vulnerability, but it could have
multiple impacts or repercussions around the organization.

Also, your systems are not working in a vacuum generally themselves either,
so we've got a platform. Think about your Windows platform for example,
Windows 7 or if you're still on XP, that's the operating system. That might
need a patch. Maybe you could add an application on top of that. Say it's
to use the Google Chrome browser for example. We'll use that as the
application. That's one application, that browser may need a patch, but
then there could be an application that you use within that application,
either a plug-in for example, like an Adobe Acrobat Reader, we needed a lot
of patches for Adobe recently, or you could be using a Web-based
application and that Web application may be patched.

Again, although we think of a patch as a single thing and very often it can
be one fix, they're really working within an ecosystem, so we need to think
about how all the different pieces interact, as well as how your
organization interacts with the process to actually affect it. Because we
don't always have the arms of Vishnu and we can't always do everything at
once, sometimes we have to step back and say, “All right, with the resources
and the time that we have, what can we do right now? What can wait and we
can do later?’ That's prioritization. I'm not saying that things are
necessarily not important. You may have a lot of things that are important,
but sometimes just for resources, we need to prioritize. And then finally
that big piece, which is really what do you do for an alternative if that
patch isn't available? So I had mentioned that this is a process. This is
really, in some ways, a life cycle. Please don't be scared. I know sometimes
when I say life cycle people go "Ah!" Yeah, it's going to be so heavy and
there's going to be so much to do. It's a life cycle, so many steps.

That is true, a life cycle does introduce steps, but it really does help to
organize how you’re responding and to create some normalized and repeatable
processes, which in the long run as you get better at doing them are going
to actually make the process more efficient. So, although it looks like a
lot of work upfront, please don't be scared of it or think, here's just one
more life cycle, because really these steps when they're done as a whole
and they're thought of as an intertwined process you will find that they
actually help to make things a little bit faster. So, in this process for
this remediation life cycle, I set out that you start with scanning. That
includes scanning for vulnerabilities and asset discovery, looking at the
patching, and the patching is really about, where is that vulnerability? How
would somebody get there from outside or from inside, because again, as I
had mentioned, when we've got these layers of operating system,
application, application -- we also have patches within a network too. So you
have multiple kinds of patching and that really can impact whether or not
that vulnerability is going to be exploitable or whether or not the
prioritization is high or low. So you need to understand that. Compliance,
it's a big one. Everybody mentions it, I know it's a buzzword, but really
we do have to go through compliance checks.

A lot of us have internal auditors as well as external auditors that we
need to respect and deal with and we need to be prepared to have proofing
on what we've done to prevent vulnerabilities, to do our due diligence, as
well as whatever specificity is within that regulation for compliance.
Business risk assessment, which is really understanding what the true risk
is related to this vulnerability; then the prioritization that we've
already discussed. Alternatives, because it may not always be the patch. It
often is, but it may not; and then the actual engaging in the remediation
process whatever the decision is after you've finished all those pieces and
then of course it is a life cycle. So we go back to scanning again. Again,
it's a process. It's about continuous work, but also continuous
improvement, so hopefully making that work more efficient.

Why would you implement the program? Again, I know that it seems a little
bit heavy. “Hey, there's a patch, let's just put it on.” When you see this
going on in most organizations, especially larger organizations,
organizations with multiple business units. What you start to see is a sort
of chaos the area begins to affect in that organization. We thought we had
this system patched, we didn't. We thought this data center had received
patch, but they didn't update it. Other things, even with phishing, because
the phishers can be pretty smart, so one form of spear phishing is actually
writing into a company and saying, "Hey it's time to update your patches"
and people are saying "OK, I'll go to that site and I'll download it," and rather
than it being a legitimate patch, it's a Trojan.

A lot of different things can fall out when you don't have a program both
from the actual system is not getting covered, as well as from your users
possibly engaging in activities that are dangerous because they don't
understand how the process works. So a program is really about normalizing
this, writing it down and getting a repeatable process that you can put
into place and understanding how you do the steps within that process so
that you can make them more efficient, like business risk, as we'll get to.
It's a really key part of the process. The more often you do it, the better
you're going to be able to tune it, and the faster it's going to go. We all
have to start with scanning. That's where it all begins and scanning is a
couple of different components. The first one is really understanding what
it is you're going to scan, so the asset discovery piece.

How do you know what might be vulnerable or what you need to patch without
understanding what's in your organization? If you've got a smaller network
you may feel like, “Come on, I know exactly what's here.” But when we get to
very large networks and even medium-sized networks, what you find is the
complexity that we have, the number of systems we have, we can get a lot of
surprises or gotchas if we aren't actually doing the asset discovery. It
really makes sense to go ahead and do that before you even start scanning
those assets for whether or not they're vulnerable. Once you have actually
gotten that information, then you can go out and you can say, "OK, do I want
to patch this or do I not? How would I go about re-mediating on this?"

You may or may not already have some asset information in your
organization. If you do go ahead and reuse it. It may not be perfect, so
I'm not recommending that you take an asset list from, say your operations
department or your finance or your purchasing department and say, “All right
we know that this is the one true list because we've gotten it from another
group.” As the security or the audit group you're probably going to need to
go out and do your own scanning and your own validation of that list, but
at least that's going to be a really good starting point and then you
validate and verify from that. Check around, as I had mentioned, you can
get it. Possibly purchasing might have something. Operations might have
something. It may be in the form of an asset database. It may be in an
Excel spreadsheet, and there may actually be some other asset discovery
type solutions that's being used by your network group, for example. So
start there, but then go ahead and do your own validation and verification
and add on to whatever data you found to make sure that you've got the
right scanning and asset coverage for your security work.

Once you know what you're going to be taking a look at, now you can actually
go out and scan those devices for whether or not they've got
vulnerabilities. One of the easiest or simplest ways is doing the patch of
OS Fingerprinting and the versioning, so you can do a very light scan
against that device and just find out what the type of operating system and
what the patch level is currently on it. Sometimes that will tell you if
you're in or you're out of compliance, if you're in or out or patch needs.
Go way back until you've got XP and you have service pack 1 on it. Then
that's an example of that would be an easy one. “Oh OK, we know that we're
back there on the patch.” You can go a little bit deeper, however, with
vulnerability scanning where you can start actually trying to exploit
vulnerabilities on the system, log into the system, check for some other
things that are related to security and compliance. For example, passwords.
We talked about configuration management, so if you've configured a device
to have an 8-digit passwords and they have to have a mixture of characters
and letters and alphanumeric. You could actually with the vulnerability
scan, going a little deeper than just a passive OS fingerprint, you can
look at whether or not the passwords are at the right level.

Try logging in with a lower level password, creating a lower level
password, or using some other kind of identification on that machine to see
if it's within compliance. That could also be part of what you're checking
for with the vulnerabilities. You can look internally or externally. It
really does matter which way you're looking at those vulnerabilities,
because an external scan is coming from the outside. You can do it
yourself, you could also have a third party do that scan for you, but
looking at your organization and your vulnerabilities from what an outsider
could see; how far they could get into the system. You also want to have a
good idea of what's going on with your overall vulnerability picture from
internal as well. Because we do have threats from internal folks at our
organizations. Some people are the insider threat, do try to exploit
vulnerability, so you're not necessarily "safe" if you've got an unpatched
vulnerability or an unmitigated vulnerability that's within the

Most companies will have third parties and other entities that come into
their network that may not be as trusted as their own insider, so
outsourcers, consultants, business partners. So again, you want to take a
look at *both*. That's why I've asterisked both. I really strongly believe
you should do both of these. Then you want to ask yourself what the
frequency is
. There are some recommendations and some regulations that say
quarterly is something that makes sense, there's also annual. I do think
annual is really stretching out, so you're getting a snapshot once a year.
There are a lot of changes. On the other hand, if you're doing it monthly
for example, then you're initiating a lot of scans. One thing to think
about is that you do have different kinds of scanning. So there's the more
invasive types of scanning, there's less invasive types of scanning, sort
of low and slow. So what you might want to do is look at doing something
that's low and slow all the time, but then do a heavier, more robust scan
once a month, once a quarter, once a year. Again, it's going to really
depend on what you need. Good starting point, what kinds of regulatory
compliance do you need to adhere to? So PCI DSS, for example, says quarterly
for the external scan. You know it has to be at least that. But what if
you're an energy company? We'll look to NERC for any recommendations there.

If you're financial services, not looking at PCI, but you look to the FFIEC
for any guidance. So, start there and then match whatever compliance you
have. Take a look and then match that against what you have to do for your
own needs for your own company.

When you're looking at the actual scan itself, the depth of the scan really
does matter, so consider if you've got an IPS or you've got a firewall
running and you're doing an external scan and that IPS or that firewall
they're doing their job and they're blocking everything. That's actually
good. On the one hand you know if you are an external attacker, wow you've
really got that blocked.

On the other hand, it may not tell the whole story. So one example of how
that whole story might not be told is that broad scan may not have actually
tried to exploit an application that is allowed through the firewall. So
you may have an application behind the firewall, port 80 is open, scanned
the firewall, it didn't let anything outside of port 80. Port 80, yet you
saw you could get to a web service or a web server you know is supposed to
be locked. OK fine. But there could be, for example, a sequel injection fault
within the application running on that Web server, so you'd want to get a
little bit deeper to actually exploit that specific vulnerability to see if
you can get farther along into the organization.

Another place where just that external scan where you've got the firewall
doing for the most part its job, also isn't telling you all about the other
patchs that may go into your network that may not be directly to the
outside. So on that I'm talking about the business partners outsourcers;
let’s say you've got an application that you had somebody write for you, they
have remote administrative access to that application. Well that's another
way that they can get in, not going through that firewall. It's great to
scan to see that the firewall, the IPS at the perimeter or the edges are
doing what they need to do. But do keep in mind that you want to get a
little bit deeper to make sure that you've really looked at all the
vulnerabilities rather than just looked at one component of it.

The other thing is mitigating control. So you may actually have protections
that mean that you don't need the patch, or the vulnerability may have been
taken care of in another way. An example here would be that you do in fact
have a sequel injection running on that application, but you find out that
if you try and exploit it, you've got a Web application firewall that has a
very specific rule that was written to prevent exploit of that
vulnerability. So also checking on that to make sure if that's there
because then that could mean, OK we don't need the patch, or if it's a
custom written application we don't have to actually go back and rewrite
it. Once you've done this, you've got your asset discovery, you've got your
scanning and now you're starting to say "OK what are we going to do now?
How do we prioritize what gets fixed or patched or re-mediated first?" You
really need to understand the number of things, but the most important
thing is to be very realistic about what happens if you don't patch. What's
the quantitative or the value associated with not patching? It sounds
really easy when you say that. I feel like it sounds easy. But when you
start to take it down and understand specifically how you're going to
assess that for each of your devices, it can get pretty complicated, because
you have to look at all these different moving parts about which business
might go down, what kind of business might go down, what kind of business
might go down, is this a theoretical exploit, is this an exploit that's
being actively used over the Internet or over the world?

You really do have to step it down and look at what the true cost is, look
at the damage that can be caused and then look at that business risk
analysis, which is where the complexity of trying to break that down comes
in, and I'm going to be doing that on the next slide, so I'm not ignoring
it. We'll just take that on the next slide. The other consideration and I
had mentioned this before. But compliance really can give you at least a
baseline, not necessarily where you want to be that's right for your
company, but what compliance does tell you is where you have to be in order
to be compliant with that regulation. So if you look at that as a starting
point and then say, “OK from there I'm going to figure out what next, what
my next scan, what my next move is.”

So that business risk analysis that I had talked about; the first thing to
do is take a look at what kind of data is at risk. So, in that example where
our firewall was working, but we had a sequel injection vulnerability on
the Web application, and that meant that the database in the back end was
vulnerable. Alright. Well, what was the data on that database in the back
end? It could have been company’s intellectual properties, it's got very
high value. It could be patient healthcare records. If you're in healthcare
at all, you know, or if you actually go to a healthcare professional, so
pretty much all of us. Part of what's happening with high tech is that
we've got EHR, so electronic health records are becoming mandated within
health entities. Most of our healthcare information is already, or being
converted to be stored within digital means, very often on a database.
Healthcare information generally considered very highly sensitive. So we
have due diligence to protect around that. So what if that was in the
database? What if it was Social Security numbers? Again, something very

On the other hand, what if this was a database of my dog's pictures that I
like to post on the Internet? Important to me, but I have a backup and
really honestly if somebody stole them it's not a big deal. I'm using my
dogs and I realize that's not a business case, but I don't want to take any
liberties and assume that you have data in your company that's at that
level of sensitivity. The point being, we've got different levels of
sensitivity. We need to first understand how important that information is
and if it's not as highly sensitive, that may be a lower priority. An
example in business would be for example, public data. So if you've got
information, you've got data sheets about your products, you've got
published customer case studies, these are available on the database for
searching, but you've also got backups. Again, this is public information,
so losing it isn't really actually a big problem the way that losing
sensitive or classified data is. Then you want to understand, so once you
understand well, what if they get this data? What's the loss to the
organization? Now figure out, ‘how easy is it for them to get to this data?’
So one thing is do they want it? If it's valuable they probably are going
to want it, especially if it can be resold.

An attacker really enjoys finding something that has value. That's why
credit card numbers are so attractive a target. But then also how easily
could this vulnerability be executed? Because as we know, as we've seen,
some are easier to execute than others. So understanding, is this a very
high complexity problem to execute? Is there something out there that's
already written that means that it could be easier? Unfortunately, the
reality is that attackers are now making money by selling exploit kits. So
an exploit is discovered, it may be difficult to exploit it from scratch,
but you can buy a kit and some of these even have customer support. Oh you
can't use our attack tool, here let me explain to you how to use it. So, is
it easy to exploit? If it's not easy to exploit, has it actually become
commonly easy to exploit and you can look at some good references to find
out what's going on with that exploit in the wild and to get keys and
insight into how easy it is to actually do it. Then also the threat actor;
so the threat itself, who is this attracted to? Is this something that
could be exploited by an insider only, or is it something that could be
exploited from outside? That's going to give you potentially a different
weight depending on how you treat insiders and also the level of
protections that you have for inside versus outside.

Another thing with outside is that you've got a much larger, broader threat
surface, because you've got every attacker outside in the world. Whereas if
it's an insider, you've got your employees and whoever else you've allowed
to come into your network. So going a little bit deeper into what's the
actual cost for losing or exposing this data, I've got some points here to
think about. You may not need to think about all of these, and also you may
have some that I haven't thought of, but overall it's a really good
starting point. First, think about the regulatory and the compliance fees.
In the past, we've seen some different numbers come out. People will violate
PCI DSS, what they get charged with for their cost, and we had some HIPAA
violations come out, so you can look at what's been charged in the past.

You can also ask if there are any known fees that will be levied against
you, but what happens if you don't do this? Think also about the loss of
the revenue. So what happens if this is going to cause some actual
downtime? So back to our application, we had talked about the database and
what was the sensitivity of the data in the database. But let's flip this
and let's say that this is an airline and the information in the database
was the timetables related to people and the list of the passenger
manifest, so being able to know if you could book online or not. Well let's
say an attacker manipulated that data, closed that database down, then the
problem here is not just that the data was gone, but also that there's
actual downtime now because the airline itself can't take online
reservations, so for that window, they're losing to their competitors who
are still able to take the reservations. So what would happen with that
kind of a loss is downtime? Loss of customers, especially in a reputational
kind of environment. So if you're a bank and there's been a big loss of
money from your company people may feel, “Oh I don't trust this bank. I want
to go to a bank that's more trustworthy.” So they may actually take their
assets and move them elsewhere. This one I would say take a grain of salt,
though, because how many of us know somebody who said, "I'm never shopping
at Marshall's or TJ Maxx or HomeGoods ever again," after the TJX companies
were hacked and everyone that said that to me, I know they've gone back and
started shopping.

Take that loss of customers, understand it is a possibility, but in the
long run it may have a lower impact that you may initially think. Customer
exposure though, that can definitely be a high impact. So if your customer
feels that you somehow harmed them, there's been a problem related to you
losing their sensitive data, actually coming after you and saying,

“OK this is something that I want to sue on, there's actually been damage,”
whether it's their health information or whether it's the loss of their
credit card or in the case of the TJX, we had the issuing banks who were the
people that create the credit saying that they had damages because they had
to pay to actually reissue cards to their customers, so it was a financial
damage there. So what's the exposure? Is there something that you could be
sued over and that would actually be. Lawyers are generally very expensive.
Usually you get into lawsuits, we're not talking small bucks. Your public
perception, that goes back to if you're supposed to be trustworthy and then
it's seems that you've lost customer data. Is there going to be a perception
within the media that there's something wrong with your organization or
that your somehow not trustworthy or not worthy of the customer being with

Then another one that sort of doesn't get thought of as often, or a lot of
people don't think of it as often is the further exploitation; so could
this be the beginning of something else? We're starting to see a lot more
very complex kinds of attacks, so well thought out rather than, “Oops, here
I've got a Trojan, get it on a drive-by download and here we go.” It's
more you plant something onto a user’s computer and then later on you may
turn it on or off. You may use it as part of a bigger attack or a more
composite attack months later. So we're starting to see that kind of a
development, so you also want to think about what was that actual attack
and is it possible that somebody was actually setting up rather than an
exploit, at that moment was setting up a composite or readied attack for
further exploitation later.

On these next couple of slides, what I've got here is, I've taken that point
of let's say that we don't have sensitive data on the back end. So we've
classified this as not highly sensitive data. We scanned, we got that there
was in fact a sequel injection problem that had a CVS severity score of
7.5. The company weighted it out at the same thing, they did the test to
validate and found out that despite the sequel injection, no sensitive data
was actually exposed. The company gave it a weight of zero. However, from a
compliance, so the company said, "Hey you know what, yeah there was a
problem but there was no sensitive data." But looking at the compliance
piece, the PCI DSS, their report on compliance gave them a failure because
there was a sequel injection even though it didn't lead to getting
sensitive data, it was within the card data environment ecosystem.

So that then became a 10 for the company because they don't want to fail
their PCI DSS ROC obviously. So then there's a final weight here on the
business risk with that no sensitive data. In this example we've got a
total weight of 19.5, that would probably be a low rank, despite the
failure of the PCI DSS, it was balanced out by the other considerations.
Here we've got sensitive data and we've got a different setup. So we've got
the same exact problem with that sequel severity level and the company
weighed on it, but as it was tested and validated as you moved down that
patch, it was in fact sensitive data. So the company weight rather than
being a zero, is really amping up because it's a 9.

We've still got the problem with the DSS failure, but when we come down to
the overall business risk we've greatly increased the total weight of this,
which is now a 35 and the difference here being that it was sensitive
versus nonsensitive data. Everything else staying the same. And again, this
is an example. You can plug in your own numbers. You may actually have some
frameworks that you've been using, but if you set up these frameworks and
you continue to use and reuse them, what you'll find is that over time you
get much better at creating and understanding the weights. Things like my
nonsensitive data, it could be there was a dependency that gets discovered
that always impacts that particular kind of a flaw. So you might then go
back and say you know what, it looked like it was lower severity than it
is, but it's higher. Alternatively, the sensitive one might get lower,
although, I doubt that. The point being, as you go and you redo this you're
going to get better at it and also understand where if any place you hadn't
had it really tuned highly and most effectively.

So now that you've got some sort of an idea of the prioritization, what are
we going to patch? Because we need to get to this because it's got a higher
business weight for impact to us. At that point, now you're ready to
actually go ahead and perform the remediation. In general, from those
numbers, the higher the number is going to be, the faster the patch. Your
mileage may vary based on your risk analysis. Do document it. Again, this
is to really avoid that chaos and it's really not to add extra pages to the
digital folder, it's to get a process into place. So you want to document
where you're going to perform the activity and also what the activity is
for the different types of targets. Although a patch is, OK you put the
patch on, you may have very different needs and plans for the kind of
target. For example, if it's a server, this may be a server that needs to
be up 24 hours a day. Well what do you do if you need to patch a server
that can't be offline? Well if it's in a high availability environment, you
may patch one server that's not online at that moment, put all the traffic
over to that server, take the other one down and patch that for example.
But that's probably different than a desktop, which you may just do a wake
on LAN overnight and do the patching.

Mobile, how are you going to corral these systems back into the
organization or when do you have the opportunity when they come back
online? Then cloud and remote, because we've moved a lot into the cloud. So
the question is, are you working with your cloud provider to manage their
patch process on your servers, on the devices that you're using for your
organization? Do you actually do some level of maintenance on servers that
you've got at the cloud? Because you still retain some of the management
even though the location, it's located someplace else. So again, different
kinds of processes for the different targets themselves, and you do want to
look as you start putting things into place, what's working and what's not
so that you can hone that process over time.

The next thing, after you've actually got the patch ready to go and you
know who you're going to deploy it to, you want to do a little bit of
testing prior to sending that patch out there. There are a number of
different reasons for this. A big one is that, very often the patch may not
actually fix the problem that you think it's patching. It would be nice if
every patch we got from a vendor or provider did exactly what it was
supposed to do, but that's not always the case. You want to first make
sure you apply the patch in a lab environment and see if it's actually
fixed the problem. Also take a look if that patch is having any kind of
dependency or any kind of change to a device or to an application within
the system. One of my favorite patch stories was that there was a big
patch that was very critical for an operating system, but for a particular
vendor of identity management, and it was an agent-based identity
management, for this particular vendor, what that patch did, it didn't like
that agent, I'm anthropomorphizing. For some reason, the way the patch was
written, that agent was identified as not supposed to be in the box and was
deleted. The patch was applied, but now the entire identity management
system had been erased on all of the targets. Generally, it's not that bad,
but we do see examples where the patch does something unexpected. So you
also want to make sure that it hasn't introduced a new issue, changed how
any device or service or application is working with another one.

Because you are in a large ecosystem, you may not be able to test every
single bit, but certainly on servers on key or critical applications, do put
the patch on in a lab environment, and at least do a little bit of testing
to make sure it's doing what it should and it's not interfering with what
it shouldn't. One way to make this easier overall is if you're using
golden images. Now on servers, there may be some level of golden image, but
there's often quite a bit of tweaking, because servers are very purpose
built. On your desktop, you may have a much better chance with this because
most of our desktops, laptops, especially at larger organizations, we
flatten them down to golden images and even been able with good robust
backups to be able to take it back down to the golden image as needed.
What you'd do is you would test the patch against the golden image. And I
would very much recommend for companies if you haven't done this already to
get a nice straight baseline level of what your system looks like for most
of your users and then put in robust backup ability on the sensitive and
changing data like documents, Excel, Word and the messaging, because these
often are the key bits of information that you want to have backed up. And
that allows you to, if you need to in case of a vulnerability, to go back and
be able to know that you've got a nice, clean, standardized baseline for
testing and for deployment.

Alright. Speaking of deployment, how do you actually get the patch out
there? Well there are a number of different models for doing this, a number
of different solution types that you may or may not have within your
organization already. Network operations very often, in the large network
operations management solution, they do have a way to deploy out patches to
targets like servers and to laptops and to desktops. So, that's one
possibility. There are some patch and vulnerability specific solutions if
you don't have one, but you're looking into one and you want to have this
capability, absolutely put that into the RFP. There are some platform-specific, Microsoft has some for their operating system; it's certainly one
way to go, but the issue with the platform specific or least the caveat
there is that very often they only work with their own platform.

So if you've got a heterogeneous environment, which many of us do, you may
need something that's going to play well with others. Another thing to
really think about here, especially as you're thinking about patching, is
we're now more and more getting these very, very remote mobile devices in,
so Netbooks which are very often Unix-based, although more and more we are
seeing Windows based. Unix-based, which may not have been in your patch
management, but now are suddenly in your corporate ecosystem and need to be
patched. The tablets, they're coming. Again, if it's a known platform that
we're already supporting OK, but if it's something new that we haven't had
before. For example an iOS on an iPad, then you're going to have to have
something in place for that if it's going to touch the corporate network
and then also the smartphones and the other smart devices or mobile phones.

Whatever you're going to pick for your platform, however you're going to
deploy this, and I've talked about a couple of different options here, but
whatever you do pick I would recommend that you look at some sort of
centralized reporting. Because you want to be able to know that you've done
that patching and that testing across the entire environment and if you've
got multiple pockets of patch management, which you're going to have to do
to get that big picture report, is you're going to have to aggregate and
normalize is somehow. It's going to be easier for you if you have a
solution that can do the bulk. Maybe it can't do everything you want, but
that can centralize and report back on the bulk of the targets that you're
looking at for vulnerabilities and patching and configuration management.
Another thing is it will help you with the normalizing policy, because
that's a big one. If you've ever looked at a syslog, for example, to see what
happens on an application or a device, you know that things that you think
are going to be very normalized aren't necessarily normalized. You may not
even be able to normalize across your entire organization. For example,
passwords, we've got some legacy devices that won't support more than more
than 4-character passwords, actually some even newer devices that won't
support more than 4 characters. So you may have a policy that says 8
characters, but you can only support 4.

Alright. So you can't normalize and implement the policy across all the
devices, but at least being able to have that policy normalized for
knowledge and then you could write an exception saying, "But on this one
device, unfortunately until we upgrade the device, we're not actually going
to be able to keep that policy the same." But being able to understand that
policy across all the devices as well as across the patching, because
again, that sort of that loop and that chaos theory that if you haven't
actually done everything as a coherent whole, you may have some blind spots
where you don't patch. Another big benefit to doing some level of
centralization is that you're going to be able to see where you haven't
patched, coming back into one main console; rather than if you console A,
B, and C and console C has two unpatched devices, is that necessarily going
to get back-rolled up to the MetaConsole?

I talked a little bit about exception handling. This is critical. It's not
a perfect world. It would probably be boring if it was a perfect world, but
it certainly makes our lives as security and audit professionals a little
bit more difficult. Because we have to be very clear and write down when we
can't do, what is within the policy or required for the regulation, that
we've created an exception and explained why this can't be done the way
that we had expected to. Then integration with the other types of consoles,
the other MetaConsoles that you're using, so the configuration management
and the scanning you may find a solution that does different components of
this. It does the asset discovery, it does the scanning of the systems and
always does the patching, or you may have a patch system that's stand-
alone. If you do, that may be the right thing for you. But then absolutely
make sure that it's going to integrate with these other components because,
back to that asset, you can't scan what you don't know, but it's also going
to be hard to patch what you don't know. So you definitely want an asset to
scan to link into deployment.

The next big one is validation. Because just because that patch is out
there, you tested in a lab that it prevented the problem that you wanted to
prevent, but you've got to test it again actually in deployment and
production, to make sure that it's doing what it was supposed to. So, was it
actually applied? It sounds funny, but it really can be sort of like a tree
falling in the woods. You say, "Hey go out and apply this patch." You get
back success, well sometimes unless you check you don't know if it's a

A way to actually go through and make sure that it's been applied and to
also validate that fix along the path and make sure that that fix has not
just been applied, and we know that it's on that application or that
server. But it's also actually preventing the exploit that we wanted it to

Some alternatives, because you can't always necessarily patch. What happens
if you get into that kind of a situation? Well, there are some other ways
to limit or control the damage and it may be that that other way could be
more cost effective or more beneficial or preferable to the organization in
the short or the long term. So what are some other ways that you could
limit and exploit a vulnerability? The IPS systems, either on the host or
on the network itself. I mentioned Web application firewalls already.
They're certainly one option for your Web app, specifically. You could add
a firewall rule, generally on a perimeter firewall. They're not going to be
granular enough, but it is possible that a firewall rule or even an ACLon
a router could prevent that vulnerability. Just understand what the
vulnerability is and then look at these alternatives. Another thing is
actually just going ahead and rewriting the code. Obviously it's going to
be really hard for us to rewrite Oracle code unless we're Oracle, but if
it's a custom application that we've written for our own organization or
we've had outsource, we could go back to the developers and say "Hey, we
found this now we'd actually like it fixed."

Keep in mind however that while this is an alternative to patching, it may
be an expensive or time-consuming one. So how do you make the right
decision? What should I do? Who gets patched first? Do I fix it? Do I find
it? Do I find an alternative? It's going to be up to your company and your
individual decisions for that vulnerability. However, number one, take a
look at whether or not there's a patch. Because if there isn't a patch,
you're going to have to look at the alternative, and that does happen. First
start off, is there a patch available? If you're looking at a zero-day
exploit, and these get a lot of press, when you see how they're actually
used. Very often it may not be as bad as a lot of the press that we hear. I
mean, we have a lot of non-zero day exploits that are still being exploited
that we've had patches for, so we really have to think of this as a big,
big picture. But thinking about a zero-day you may not have a patch ready
or you may have to wait until the next Patch Tuesday or the next patch
cycle, so without that you're going to have to look at least in the short
term, at an alternative. If there is a patch then you can ask them how long
is it take to apply and what would be the risk with patching or without
patching, because there are some risks associated with patching. It didn't
work. It broke something else. Another potential risk related to patching
is if you happen to have a device that is certified at a particular level.

So, say you're 6140-2 certified, well for that exact version of the
software, you change that software, it's going to throw it out of
certification. Healthcare devices are another great example where if you
patch it's not a healthcare device or a voting system, it's not a voting
system. It was certified at a very specific snapshot in time. So patching
could actually throw it out of where it was, so that would be another risk
to consider. How do you look at the alternatives and whether you want to
use them or not? What's the time to implement? What would be the cost to
actually put that in? How good is the solution; the alternative? Is it
better than the patch? Is it worse than the patch? This is again, if you
can patch? And then if you're going to mitigate, what's your long-term plan?
Because very often a mitigation is a short-term window until we can get to
the more robust version of the fix, so can you put a plan into place?

Alright. So winding up with some keys to success. Patching is really part
of a greater life cycle. It's part of the big whole. So if you do it
atomically, you're probably going to miss some devices, you're not going to
have a robust solution and you may actually leave your company open to
further attacks. So really make it part of the life cycle and again, it
doesn't have to really be this long terrible process. It really can be
something that, as you do it again and again, think about the routines that
we have in business or in our life. The more practice we have it the easier
it is for us to do and the better we get at it, so this is really about
getting to that kind of a process, where it's repeatable, sustainable and
you're honing it with continuous improvements. Do prioritize and test
before the patch. It's easy to go "Ahh, there's a patch!” you know, but do
think about which systems need to have that patch first if you can't patch
all at once, and also don't forget to do a little bit of testing. You may
need to do mitigating controls, you may need to look at alternative. In the
case of a zero-day, there probably isn't a patch yet, so what are you going
to do in the meantime to protect that? Or if you have to delay patching for
other reasons or if it's a custom application. So what are your
alternatives? These are valid choices. You can put in mitigating controls.
It isn't, throw the baby out with the bath water and say we don't have a
patch so we shouldn't do anything, but do keep in mind that if you're
putting in mitigating controls assess if they're going to be long or short
term, because it could be short term as you actually look for a better fix

Do continue to validate and refine your processes. As your maturity
increases, you really will find that you're going to get better at doing
this, that you're going to find that you can tune your systems a little bit
better, you'll have your processes written down. So, it really in the long
run is going to overall improve and become a more effective process if you
keep at it, I promise.

So, some resources and references on the next couple of slides. I have some
lists here. NIST, always a really great resource, so they actually have
some very good advice on how you can go ahead and create this kind of a
vulnerability program. I have some listings here from the specific
regulations that you may be under, so PCI, NERC, HIPAA, all of them, the
specifics around these. All these references are specifically pointing you
to where you can find information on the patching. The Unified Compliance
Framework is a great resource that normalizes regulations and controls
across the organization and then also at the end. I had mentioned CVSS and
CVE earlier; these are common vulnerability enumeration and scoring
systems that can help you as you're creating your own business risk
analysis to understand is this is easy to exploit? What's the actual
exposure related to this vulnerability? These are resources that can help
you with that specifically. And then SANS there with their top security
risks index, which is what I think they renamed their original top ten.

So, I hope that this was helpful for you as you're thinking about creating a
vulnerability and patching life cycle and program rather than just patching
it as the patches come out, because they really will in the long run help
your organization to run more smoothly and to more effectively prevent
exploits of the vulnerabilities, whether they be zero-day or longer term.
So I want to thank you very much for listening in to this presentation and
watching. For more information on configuration, patch management and
vulnerability management, please visit

View All Videos