Most cybersecurity reports tend to gather information from the same sources -- IT professionals and vendors --...
but a new report attempted to highlight an underserved perspective: hackers and pen testers.
For The Black Report, Nuix performed a hacker survey at Black Hat and DEFCON 2016 in Las Vegas and Chris Pogue, CISO at Nuix, wrote in the cybersecurity report that the "research was quite contrary to the conventional understanding of cybersecurity."
"Some countermeasures that you think will stop an attacker won't even slow them down. Other defensive techniques that you think are totally arbitrary actually have a tremendous impact on your defensive posture," Pogue wrote. "We found that unequivocally, perception and reality are in desperate need of realignment."
The Black Report targeted both pen testers and hackers for the survey because, as one respondent noted, "The only difference between me and a terrorist is a piece of paper [a statement of work] making what I do legal. The attacks, the tools, the methodology; it's all the same."
Director of assessments at Rhino Security Labs and former black hat hacker Hector Monsegur told SearchSecurity he was a big fan of research projects and hacker surveys like this, because "it is extremely important that people have access to information that will empower them, as well as their organizations."
"Hackers and pen testers alike deal with the reality of security problems on a day-to-day basis, even becoming victim to attacks themselves. Exploring their thoughts and experiences dealing with cybersecurity is a very good first step," Monsegur said. "Although conferences like DEFCON, Black Hat, HackInTheBox, etc., provide an outlet for these researchers to disclose vulnerability information, methodologies and techniques, the truth is that most people simply do not know how to access this content or know it even exists."
Joseph Blankenship, senior analyst at Forrester, agreed and said the report "provides a useful view of security."
"The report offers a perspective I haven't seen in other cybersecurity reports. I don't believe many security decision-makers have gotten this perspective either," Blankenship told SearchSecurity. "Getting an assessor's view of security is helpful and may help to guide some decision-making."
Not all of the information in the report clashed with conventional wisdom, of course. The hacker survey found that 72% of respondents used social engineering to gather information before launching an attack, and 86% said they used vulnerability scanning to find the weaknesses of a target. Additionally, the two most popular attack methods were phishing (40%) and direct server attack (43%).
Pogue told SearchSecurity "it's important to understand how the adversary thinks and acts. The more defenders know, the better they can prepare themselves."
The cybersecurity report also found that the tools being used in an attack may not be what cybersecurity professionals expect, given the focus on exploit kits. The report found 60% of respondents used open-source tools and 21% used custom made tools.
Pogue noted that only 10% of responders used commercial tools, which was something of a surprise because "being employed by an organization usually means a budget to purchase tools. But, according to the research, even when the budget is available, the respondents opted for the free or self-created solutions."
Jake Williams, founder and CEO of Rendition Infosec LLC based in Augusta, Ga., said these findings could imply a sampling bias due to respondents all being attendees of a major hacking conference.
"This seems to imply that most attackers don't use exploit kits and commercial tools. Very few people will admit to using illegal exploit kits, the vast majority of which are rented panels that would be unsuitable for commercial client engagements," Williams told SearchSecurity by email. "At DEFCON (somewhat less commercial) I would be surprised to find many people admitting that they use Core Impact or Cobalt Strike."
However, Monsegur said the low use of commercial tool and exploit kits (just 3% of respondents) could be because "exploit kits that are sold on the dark web are usually buggy, highly specific and more often than not, backdoored," so more experienced hackers wouldn't trust them.
"I think the percentage leaning toward open source tools is mainly because some of the best security tools are in fact open-sourced. It is because these tools are open and freely available, and accessible, is why most security researchers use them predominantly," Monsegur said. "These same security researchers could make their own exploit kits, write custom tools, or setup environments similar to Kali for their engagements."
Blankenship said the perception that assessors use more commercial tools may be "thanks to vendor marketing efforts and a misunderstanding of the pen testing process."
"Proposals for security testing often include a list of tools to be used by the assessors. Most of these include a list of commercially available software tools, open source tools, and tools developed by the assessor," Blankenship said. "Skilled assessors will use a variety of tools that mimic those used by adversaries which will likely be a combination of open source and self-built tools."
The cybersecurity report also found that half of respondents said they change tactics with every engagement with 38% changing tactics within every six months, and 56% said they changed just to keep learning new tricks.
Williams attributed the changing tactics to "security products evolving to cover the old techniques."
"This is an unfortunate reality, but security countermeasures will always be playing catch up. It's in the name, that's why they are called countermeasures," Williams said. "But for the 56% who say they do it just to learn new techniques, they are doing it for more than the desire of knowledge. If they don't continue to learn new techniques, they will be stuck with only techniques that don't work."
Monsegur said tactics have to change out of necessity.
"Each engagement is different. Different scopes, targets and environments," Monsegur said. "Most researchers I know employ the use of a methodology, and it usually involves a streamlined process for any pen testing engagement, but eventually they begin to shift or change strategy along the way as they begin to find new vulnerabilities, configurations and software."
Cybersecurity report demographics
The cybersecurity report from Nuix found that 53% of respondents didn't place themselves in one specific group, but identified as "a combination of hackers, professional pen testers and students of technology." And, as far as motivation, 66% said they simply liked the challenge of hacking or pen testing.
Of those hackers surveyed, there was a mix of education levels between high school graduates (21%), college graduates (37%) and those with a postgraduate degree (28%), but Nuix also found a sizeable contingent (14%) who felt that "formal education is for suckers." The cybersecurity report also found many didn't believe in the value of technical certifications beyond looking good on a resume, as 66% of respondents had fewer than three certifications, and 76% didn't believe certifications were a good indicator of ability.
Pogue told SearchSecurity this is "a common theme in the security community" and warned hiring professionals from putting too much stock in certifications.
"There are just as many security rock stars that have multiple certifications as there are security duds. Likewise there are some amazing security pros that have no certifications. They are useful, and in some instances necessary, but employers need to be careful that they don't assign an arbitrary value to the certifications," Pogue said. "The biggest advantage a pen tester has is how they think; training classes are hard pressed to convey this sort of skill in a meaningful way."
Pogue wrote in the Nuix cybersecurity report that these findings "indicate that if your defensive countermeasures are less flexible than the people trying to get around them, they have little to no chance of being effective."
"You will be protecting against an attack pattern that is no longer relevant. This underscores the importance of incorporating realistic, goal-oriented penetration testing into your security program," Pogue wrote. "Only by continuously evaluating and enhancing your security countermeasures can you follow constantly shifting attack strategies."
Unfortunately, the hacker survey results in terms of effective countermeasures left experts underwhelmed. Respondents said endpoint security (36%) presented the most challenge during a pen test, followed by intrusion detection or intrusion prevention systems (29%) and firewalls (10%); although 22% said no countermeasure could stop a successful attack given enough time.
But, experts noted that these findings contradicted the respondent suggestions for the most effective place to spend a security budget, which was led by IDS/IPS (37%), pen testing (25%) and data hygiene or information governance (21%).
"Endpoint security is not mentioned among the effective countermeasures, although 36% of respondents said that it represents the biggest challenge for them," Blankenship noted. "Overall, however, the results demonstrate that having controls at the endpoint in addition to robust security monitoring provide more effective protection, which is not at all surprising."
Williams said the survey results about penetration testing could be more proof of sampling bias.
"IDS/IPS are becoming less and less effective with the ubiquitous use of encryption on the network," Williams told SearchSecurity. "I'm really surprised that more respondents didn't recommend spending more on endpoint detection since on the same page respondents note that it is their biggest challenge while penetration testing."
Monsegur said he wasn't surprised about the lack of consensus in terms of security countermeasures in the hacker survey "because, quite honestly, security is hard."
"When you have one environment with various operating systems, devices, users, configurations, settings, remote employees, open source and proprietary software then the attack surface expands to a degree that there is not one fix," Monsegur said. "There is no single solution to the security problem that most companies and people are prone to. There are various steps and areas to focus your budget on, before you reach a state of security enlightenment, if that is such a thing."
The hackers surveyed seem to agree that such "security enlightenment" may be far off, because 64% said their greatest frustration as an attacker was that people and enterprise don't fix things they know are broken and 75% said the most common response by an organization after an engagement was to perform "some remediation, usually focused on high and critical vulnerabilities."
Pogue wrote in the report that while this "appears to be a logical approach to remediation, it misrepresents the true nature of vulnerabilities and provides a false sense of security for decision-makers."
"If you only address specific vulnerabilities that you have chosen arbitrarily and devoid of context, it's the cybersecurity equivalent of taking an aspirin for a brain tumor; you are addressing a symptom as opposed to the root cause," Pogue wrote. "Simple remediation of specific vulnerabilities fails to take into account why that deficiency exists in the first place; it discounts strategic shortcomings such as poor or missing patch management policies, lack of a vulnerability management program, or an untrained security staff. This approach also fails to recognize the complexities of multistaged attack vectors."
Blankenship said this was "a fair assessment of the reality in many enterprises."
"Too many companies see pen testing as a compliance requirement that doesn't produce actionable results," Blankenship said. "Others may get the results of an assessment, see the need for changes, but don't have the resources (time, people or budget) to implement the changes."
Williams said the divide between the data found in the cybersecurity report with traditional cybersecurity wisdom could be due to perspective and because "the penetration testers are almost always lacking some data."
"What a penetration tester (or their tools) labels critical may not actually be critical. Organizations may mitigate some vulnerabilities with defense in depth, opting to create layered defenses around a vulnerability rather than completely mitigating it," Williams said. "In other cases, organizations don't remediate vulnerabilities because the penetration testers fail to communicate the risks effectively. Effective communication and report writing are key penetration tester skills that are commonly overlooked by security firms."
Monsegur agreed that "communication and education [are] key to ensuring companies fix their problems."
"Otherwise by sending off reports and disclosures to companies [that] are swamped out of time, budgets or manpower, you end up doing the opposite of what it was you set out to do," Monsegur said. "It is important to discuss the security problem and maintain a flow of communication with the right people and hope [that] by making them aware of their issues, risks and potential for compromise, they move on your remediation suggestions."
Learn why pen testing must be followed by action.
Find out what pen testers need to consider when cloud pen testing.
Get info on four pen testing tools for improving midmarket security.