Back
The Power of Instrumentation to Automate Components of Vulnerability Testing – from the Creator of IAST
In a recent episode of Agent of Influence, I talked with Jeff Williams, a celebrity in the cyber security space. Jeff is the co-founder and Chief Technology Officer of Contrast Security where he developed IAST. He was also previously the co-founder and CEO of Aspects Security and was a founder and major contributor to OWASP where he created the OWASP Top 10 – among other things.
I wanted to share some of his insights in a blog post, but you can also listen to our interview here, on Spotify, Apple Music or wherever you listen to podcasts.
Critical Responsibilities of Cyber Security Consultants
Jeff believes that critical to a career in cyber security is the knowledge of security defenses and security vulnerabilities. You actually have to learn how defenses are supposed to work. Your job as a security consultant in a lot of respects is to make sure those defenses are in place and that they’re working the way they’re supposed to. Therefore, understanding how they’re supposed to work is critical. Jeff sees a lot of consultants who read a document about how a security defense is supposed to work, and they assume that that’s how it does work. But that’s often not the case.
The other piece you really have to understand is how vulnerabilities work. And not just in theory – you actually have to work through them, exploit them, and learn how they work. Jeff believes that if you know about these vulnerabilities only in theory, you know nothing about them. You really have to dig in and make sure they work.
You can use things like WebGoat, which Jeff created, to start to understand them, but you should go back and recreate them. And it’s not going to work the first time, you’re going to have to experiment around and figure out how to make it work – which is part of the job.
Effectively Communicating Vulnerability Findings
Learning how to write up vulnerabilities is almost as important as being able to find them. It’s really important to be able to communicate your findings and get people to take action. Jeff said he’s read a number of vulnerability write ups that are so bad because they’re too technical and don’t describe the risk, especially the risk from a business context.
Ultimately your work goes to waste if you can’t effectively communicate with others what you found and the importance of what you found.
You can read more in this article Jeff wrote on LinkedIn.
The Necessity and Benefits of IAST (Interactive Application Security Testing)
Jeff was having trouble getting their customers to succeed in their application security programs. They were getting some results and fixing some vulnerabilities, but it was a lot of work to get there. He wrote a paper a number of years ago called Enterprise Java Rootkits, and the question was: what could a malicious developer do inside a major financial enterprise? Everything in that paper is still valid today – and it’s terrifying. One of the techniques that he looked at was instrumentation, and dynamically instrumenting an app from within that same app.
This paper got Jeff thinking about instrumentation and if it could be used for good. It struck him that this was a way of getting inside the running application and watching it run. He realized he could watch a SQL injection vulnerability from soup to nuts. He could see the data come in, track that data through the application, see it get combined into a SQL query, see that query then get sent to the database, and check back on that path to see if the data went through the right defenses.
If you see that path, if you see data come in, and go into a query without being escaped or parameterized, then that’s pretty good evidence that you’ve got a SQL injection vulnerability, so he started playing around with it and tested it on WebGoat.
He shared about the first time he found the SQL injection in WebGoat without doing anything other than adding the agent and using the application normally. He watched the log spitting up, and then saw this thing that said: SQL injection detected. That magic has stayed with him to this day.
It’s amazing to watch instrumentation work. It’s like finding all this fantastic information out of your application without any extra work.
IAST is also seamless from a development perspective. It happens in the background and in real time. You don’t have to have a security background or security awareness to be able to do this.
Noisy Cricket: Strengths and Weaknesses of Static Analysis and IAST
Jeff shared that OWASP decided they didn’t really know what static tools were good at and bad at, so they wanted to measure it. To do that, they created this huge test suite, almost 3,000 test cases, half of which are false positives and half of which are true positives. Then they run a static tool against it, get the report from the static tool, feed it into the benchmark, and it will score the report and create charts to show the strengths and weaknesses.
It’s a low bar; none of these tests are particularly difficult. But what is surprising is how poorly the static tools do on things like data flow problems and all the injections (including command injection, SQL injection, XSS, LDAP, etc.). In response to that, the static vendors started changing their products to do better against the benchmark, which was one of the intentions of the benchmark project: set a bar so that products could get better.
Jeff noted, however, that the strategy the static tools chose was to not miss any true vulnerabilities, but basically not care about false positives. As a result, the static tools increased their identification of true positives, but at the same time, added false positives.
In response to this, Jeff wrote a tool called Noisy Cricket that finds all the true positives without caring about false positives. Basically, it says any place you use SQL, that’s SQL injection, any place you use encryption, that’s a weak encryption. It reports all the results. And when you look at the results of Noisy Cricket, they’re not that different from what the static vendors are producing. It was kind of a joke, but also demonstrates that finding all the true positives without caring about false positives provides zero value. The only value happens when you find true positives with low false positives. That’s how you measure the value and that’s how the benchmark project scores tools.
Jeff believes there has to be a balance and static tools have never been able to improve in that direction. They can only bias their findings towards finding true positives or towards only reporting true vulnerabilities.
IAST is a great solution because of the nature of the analysis and the fact that it produces results that are not very noisy with low rates of false positives. Meanwhile, static analysis by nature, out of the box is extremely noisy and shows a lot of false positives, but there is the opportunity to fine tune and customize your static analysis capability. It can be a lot of work to get it to a point where you reduce the false positives to an acceptable rate, but there is value in both. The low false positive rate is one of the reasons though that IAST really shines against static analysis techniques.
However, for certain use cases, static analysis is exactly the right tool. For example, if you’re a security researcher, and you’re tasked with finding new and interesting kinds of vulnerabilities, static can be a real powerful tool. In addition, if you get good at writing custom static rules, you can search your code for things that are custom to your code.
To listen to the full podcast, click here, or you can find Agent of Influence on Spotify, Apple Music, or wherever you listen to podcasts.
