Tools that do operating system fingerprinting are a hacker's dream. They make it ridiculously simple to identify...
easy targets. Run Nmap against a target, learn what OS version it's running, and then look for a set of attack tools that can take out that particular release.
If you place a POF sensor near one of your outgoing network connections, you can quickly build a map of machines and their OSes.
Fortunately for us (the good guys), most fingerprinting scans leave distinctive patterns that are easily detected by a decent IDS. But aside from that, the good guys can also use a powerful OS fingerprinting technique called Passive Operating System Fingerprinting (POF). Several POF tools are available; the original is called "p0f" (with a zero), co-created by Michael Zalewski and Bill Stearns.
POF is invisible, silent and nonintrusive. Unlike active fingerprinting tools such as Nmap, POF operates only as a sniffer and generates no packets. This is extremely important, because that means it won't interfere with legitimate traffic, and it won't force you and your IDS to worry about which scans are legitimate and which are not. Since it's run on the target network, it's not particularly useful as a hacking tool--hackers will continue to prefer active scanning techniques such as Nmap.
In principle, POF works the same way as active fingerprinting. Nmap generates customized packets and sends them toward the target, attempting to elicit a response. The response packets are collected and compared against a knowledge base that contains OS-specific details about how various IP stacks interpret the recommendations of Internet RFCs.
POF works exactly the same way, but without first generating stimulus packets. Instead, POF watches ordinary traffic for telltale signs of popular OSes. For example, if it sees a TCP SYN packet with a total header length of 44 bytes and a time to live of 225, it can reliably conclude that the host is running Solaris, because only Solaris generates that specific combination of options. POF includes a knowledgebase with a representative number of POF-specific OS fingerprints. Additional fingerprints are added periodically.
Of course, some OSes have similar properties, making it difficult to distinguish between them without collecting lots of packets and determining if fields such as IP-id or sequence number appear to be randomized. POF looks deeply into every type of system traffic, from ICMP queries and responses to TCP SYNs and FINs to ACKs to keepalives.
POF can help improve your "situational awareness" of your network. If you place a POF sensor near one of your outgoing network connections, you can quickly build a map of machines and their OSes. This could be extremely valuable if, for example, you needed to track down what machines were affected by a worm or DoS attack. POF will also let you know if, suddenly, one of your Windows DHCP clients starts emitting packets like a Linux machine.
POF is such a useful technique that IDS and security information management vendors will probably start using it to map vulnerabilities to system types to improve the quality of their diagnosis. Already, The Honeynet Project uses POF to determine what kinds of machines are being used by hackers--another interesting data point in the attempt to understand the demographics of bad guys.
Tools like POF open the door to other exciting possibilities, such as passive application fingerprinting. Applications may show useful fingerprints between versions that would allow them to be mapped, as well. For example, such a tool could help compile a list of machines running a vulnerable version of Outlook.
Today's vulnerability scanners use a lot of logic to try to figure out what operating systems you're running. Prior to POF, the best approach was to run Nmap on your network and hope that the traffic didn't increase congestion or interfere with your hosts. Now, with POF, you can keep that information constantly available at your fingertips.
I don't think the days of packet-emitting vulnerability scanners are numbered, but I'll go so far as to predict that within a year or two, the better vulnerability scanners will shift over to active scanning only when they don't have all the information they need. Why scan a Windows machine for Solaris vulnerabilities? If you've POF'd a machine, you can automatically scan by OS type and avoid having to emit weird packets. That could save us a lot of IDS false positives and dramatically improve our situational awareness.
Marcus J. Ranum is an independent security consultant and author. He is the founder of NFR Security and built the first commercial firewall product, DEC SEAL.