Penetration Testing – Deception through Vocabulary

For those of you who have followed the NetSPI blog, you will (hopefully) have noticed that we do try to make our posts useful and informative.  We’ve kept the rants to a minimum and the speculation non-sensational.  Many of our posts are technical and focused on detailed descriptions of testing techniques.  Some of our posts are less technical and focused on industry trends and advice. This post is not of the technical nature (I’m the wrong guy) nor is it really about industry trends (maybe a little).  I want to use this post to focus on some industry-specific vocabulary.  While there have been those in the security industry that have knowingly mis-used terminology to deceive clients, it seems that the trend is growing and we’d like to take the time to help those of you who read this blog or stumble over this post really understand what a few related, but very different terms mean. Specifically I want to focus on the term ‘Penetration Testing’ and its derivative services. Please note that I’m writing this for both the people outside the security community that are buying penetration testing or penetration testing tools as well as consultants and technical assessors within our industry.  I think that there are many on both sides that are either ignorant or willfully misusing language. Here’s how NetSPI (and our clients) define the term:

Penetration Test – An assessment of an environment or application (or both) that utilizes a combination of automated tools and manual processes to 1) enumerate vulnerabilities, 2) verify the existence of the vulnerabilities, and 3) safely exploit those vulnerabilities to better understand the risk that those vulnerabilities pose to the environment.


Please note that this is a three-part process.  If you only enumerate vulnerabilities it is NOT a penetration test (this is sometimes called a ‘health check’ or is referred to as a ‘scan’ as it is primarily an automated exercise).  If you only enumerate vulnerabilities and verify that they exist it is NOT a penetration test (this is what NetSPI calls a ‘Vulnerability Assessment’). Note also the phrase ‘a combination of automated tools and manual processes’.  If you are only using automated tools and are not manually testing, verifying, and penetrating, you might be able to call what you are doing a ‘Penetration Test’, but it’s a very, very poor one. There are a lot of information security companies out there right now that provide ‘Penetration Tests’ that stop at enumeration.  There are also a lot of companies out there selling ‘Penetration Tests’ that might verify some or all of the vulnerabilities they enumerate actually exist.  Both of these situations are misleading and we constantly have to educate organizations on what the term ‘Penetration Test’ really means.  It has ‘penetration’ in the name, for goodness sake; if there is no penetration how can it be called a ‘Penetration Test’?

Level of ServiceAppropriate Nomenclature
Vulnerability Enumeration through Automated Scanning / Reporting“Scan”, “Health Check”
Vulnerability Enumeration and Verification Through Automated Scanning and Manual Processes“Vulnerability Assessment”
Vulnerability Enumeration, Verification, and Safe Exploitation through a Combination of Automated Tools and Manual Testing“Penetration Test”

I realize that for many of you (most of you, hopefully) this post is nothing new. If so, I’m certainly sorry for wasting your time, but every time I think we as an industry are past this issue it pops up again. I’ve also discovered that non-security executives often seem to think that a pen test is a pen test is a pen test and while this certainly isn’t the case (there is real skill involved in effective penetration testing, as well as the need for a solid process), what’s really frustrating is that it’s often the situation that what people call a pen test is actually a vulnerability assessment or a scan and that drives me nuts. In any case, please let me know if you have any feedback or thoughts on this topic. This is a big one for us – NetSPI focuses very heavily on penetration testing (as well as vulnerability assessment) and, in my opinion, we are the best in the business. Even if you’re in the industry and want to argue with me on that (btw – you’re wrong), I hope that you are doing your part to help educate clients as to the differences between these terms and the levels of service associated with each. Alex Crittenden