Sergey Nivens - Fotolia

Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Does mass scanning of the internet do more harm than good?

Mass scanning of the internet can reveal how pervasive a vulnerability is. Expert Michael Cobb explains how these scans work and what the arguments for and against them are.

A blog post from security researcher Robert Graham tackled the issue of mass scanning of the internet. According to Graham, the Department of Defense has warned him against conducting these kinds of scans. How are these scans conducted, and why are these scans viewed as potentially harmful by the government and other enterprises?

There are always two sides to every argument, and certain aspects of IT security are definitely dividing opinions at the moment. Should Apple have given the FBI access to a locked iPhone in New York? Should security researchers sell their zero-day vulnerabilities to governments, or work with the vendor to fix it? Should researchers scan the internet to establish the potential level of exposure to particular vulnerabilities? Researchers who undertake mass scanning of the internet are viewed as white hats, black hats or grey hats, depending on which side of the debate someone stands, or whether they are affected by these scans.

Most mass internet scans are done using Shodan, a search engine for internet-connected devices. According to the Shodan website, 56% of Fortune 100 companies and over a thousand universities use it to "analyze the internet in seconds." It can be used to find answers to questions such as which countries are building the most wind farms, or what companies are still affected by Heartbleed. It also provides a public API that allows other tools to access all of Shodan's data.

Robert Graham uses mass scanning as part of his research --  a recent scan of his shows between 250,000 and 300,000 devices are still vulnerable to Heartbleed, despite patches having been available for over two years. According to Graham, the Department of Defense (DoD) has complained stridently about its IP addresses being included in these scans. So why is mass scanning viewed as important by those conducting them, and as potentially harmful by the government and other enterprises?

Graham believes that reporting how many DoD systems still remain vulnerable to Heartbleed will put pressure on the DoD to make patching government systems a higher priority. A cryptic response to Graham's blog indicates that the DoD feels the scans are an irritating distraction tying up valuable resources, as it is required to investigate the alerts that the scans trigger. The DoD gets probed and attacked millions of times a day; it can't stop the Chinese, Russians or any cybercriminal from scanning its networks, and many researchers feel attempting to stop them from conducting nonmalicious internet scanning is unreasonable. On the other hand, having someone tell you what you already know and are trying to fix must be very irritating. Some experts believe that the number of supposed vulnerable servers reported by these scans is skewed by load balancers sitting in front of actual servers that may well be set to accept all requests, even though the server may reject these same requests and therefore is not actually vulnerable.

Interestingly, the Pentagon invited responsible hackers to try to breach its public facing websites in a program called Hack the Pentagon, where 138 security flaws were uncovered. It's the first time the federal government has undertaken such a program with outsiders; pen testing or red team attacks are commonly used by enterprises to test their defenses, provide a realistic picture of their security readiness and find and fix security vulnerabilities.

Much like the debate about vulnerability disclosure and backdoors in encryption, mass scanning will continue to be viewed as a source of irritation for those being scanned, or as a source of information by those who feel it can be used to make the internet a safer place.

Ask the Expert: Want to ask Michael Cobb a question about application security? Submit your questions now via email. (All questions are anonymous.)

Next Steps

Find out if bug bounty programs beget higher-quality submissions

Read about the mass scan that found millions of unpatched JBoss versions

Learn about the pros and cons with bug bounty programs

This was last published in July 2016

Dig Deeper on Penetration testing, ethical hacking and vulnerability assessments