Vulnerability Scanning with Multiple Products

Should you rely on just one solution to identify all of your vulnerabilities? Most of us rely upon just one anti-virus scanner, right? Every vulnerability scanner claims to be better than its competitors, but how could this be? Where is the Consumer Reports on this subject? I think there is a mix of reasons why this subject has not been picked up by the likes of Gartner or Forrester—it’s quite technical and hard to understand, and the audience may be too small. I have inquired of two independent security test labs recently as to whether or not vulnerability scanning products were ever tested and compared against one another, with the results then published. The short answer is no. Products are often benchmarked against standard criteria, and results are privately reported according to whether or not they meet the minimum criteria. There have been some rogue studies on the subject, and I have conducted extensive testing myself. I can confirm that certain products are better than their competitors, but not in all areas. Because there are not well-defined standards or readily available test results, security practitioners are left using a vulnerability scanner that performs like a piano with many keys out of tune. In our own testing we have seen variations of up to 60% among leading products. In addition, their comprehensiveness and accuracy depend on what operating systems, applications, and configuration settings you have and whether or not your scanner vendor agrees that a particular vulnerability is important enough to test for. In a decade-old product space, we have not seen complete maturity of either the space or the products themselves. During this time there have been a number of acquisitions of product vendors, and some of those acquired products no longer exist. At the same time, new and exciting products and vendors continue to emerge. The requirements of a scanner have evolved from OS level service checks to include web application vulnerabilities, authenticated configuration testing, and zero day attacks. Within the typical server environment, there are so many vulnerabilities identified time and time again, that many organizations find it difficult to embrace the idea that there may be actually more vulnerabilities out there that go undetected. If your security team is a capable one, I encourage you to incorporate both commercial and open source tools, and even consider the introduction of more than one commercial product. If you outsource this service, ask your vendor what products it tests with and whether or not it can consolidate all findings from all vendors into one comprehensive report. In lieu of product comparison benchmarks, this approach may be your best option to ensure you are not leaving large areas of vulnerabilities undiscovered. Keep in mind, if you hire a product vendor to perform your assessment, its professional services team may not be able to use a different vendor’s product within its own solution. For those of you concerned with the thought of too many vulnerabilities, check back in a couple weeks, as I plan to discuss some techniques for vulnerability prioritization and remediation.