Code Review – is automated testing enough?

We have worked with many companies that are following the letter of the law. The law being the PCI Council’s requirement (6.3.2) that all code must be reviewed prior to release. It states:

6.3.2 Review of custom code prior to release to production or customers in order to identify any potential coding vulnerability. Note: This requirement for code reviews applies to all custom code (both internal and public-facing), as part of the system development life cycle. Code reviews can be conducted by knowledgeable internal personnel or third parties. Web applications are also subject to additional controls, if they are public facing, to address ongoing threats and vulnerabilities after implementation, as defined at PCI DSS Requirement 6.6.

NetSPI has reviewed and used a number of automated scanning tools. These tools include HP’s Fortify SCA, Ounce Labs (now part of IBM’s Appscan toolset), Veracode, and Checkmarx. These tools do a fine job for what they were built for, performing an automated scan of the source code. All of these tools meet the 6.3.2 requirements, but they simply are not enough. The tools are missing many of the problems that the manual review finds, such as authentication and authorization vulnerabilities, among others. In addition, many companies are providing software as a service (SAAS) solutions for code reviews. By using this service, a company meets the requirement. These services make it easy to do the code reviews; you upload the binaries and in a few days, you get a report with many findings. So now what do you do with this report? Many organizations throw the report back at the developers and say “fix it”. The developers look at the overwhelming number of findings and start applying resources to fix them. What many companies have experienced is that these reports contain so many false positives that the developers just give up. Are you really meeting the requirement? Of course you are, but how many vulnerabilities are you missing? Based on what NetSPI has experienced, maybe half of them.