Manual vs. Automated Testing

I’ve always been a firm believer in incorporating manual testing as part of any security assessment; after all, a human is the best judge of evaluating the contents of application output, and best able to truly understand how an application is supposed to function. But after attending Darren Challey’s (GE) presentation at the 2009 OWASP AppSec conference, I was encouraged that someone actually measured the value of manual testing – and justified my belief! According to Darren, no single application assessment or code review product could find more than about 35% of the total vulnerabilities GE could find with a manual process. That alone should encourage anyone serious about eradicating vulnerabilities within their applications to step it up a notch! I would not want to be the person certifying an application for public consumption with only 30% of security issues fixed!

To understand why manual testing is so critical, let’s break down some of the reasons why assessment tools have limitations. For network scanners, vulnerabilities are largely based upon remote OS and application footprints; accuracy will decrease if that footprint is inaccurate or masqueraded. Application scanners must try to interpret application output; if an application uses custom messaging, what’s the scanner supposed to think? Code review products are never going to be able to accurately interpret code comments, identify customized backdoors, or follow application functionality that might appear orphaned. One must also keep in mind that an assessment product will only report on something if the vendor has written a check or signature for it; think about how many vulnerability signature authors exist compared to the number of hackers identifying new exploits.

Automated testing has a very important role in security assessments: these security tools help us identify a large swath of mainstream issues in an efficient manner. And manual testing can be expensive and time consuming. However, the cost of fixing vulnerabilities after an application or system is in production is also very costly and time consuming. According to the Systems Sciences Institute at IBM, a production or maintenance bug fix costs 100x more than a design bug fix. Furthermore, the cost of a breach increases annually as well. Adding comprehensive manual testing to your assessment criteria does have an ROI, and more importantly, could improve your detection accuracy by 60% or more!


HITRUST Part 4 Looking Forward

In this conclusion of the HITRUST blog series, I would like to discuss some definite opportunities and challenges that HITRUST is likely to face.

Putting together a single prescriptive framework for the healthcare industry is truly an ambitious effort. However, cross-referencing this framework with different regulatory requirements and then proposing a mechanism by which companies can be certified against this framework takes any such ambitions to a whole new level. The good news is that many of the healthcare industry’s biggest organizations have gotten onboard and made significant contributions to this effort. Additionally, with the way HIPAA is written, there seems to be a lot of need for a framework such as this, which can enable companies to better defend their interpretations of HIPAA requirements. Therefore, I think the future of HITRUST is going to be defined within the following considerations:

  • Quality of the Framework – In order for the framework to gain traction, it must be of good quality, and it should achieve its stated objectives of being risk-based and prescriptive. Even though the framework is a product of multiple organizations collaborating, HITRUST does not necessarily govern by community and will make the final decision about CSF content. Another important aspect of the framework will be the approval process of alternative or compensating controls, and ensuring that the process of approvals or denials is transparent. Nothing will de-value the framework faster than perception of its being driven by the agenda of any specific company rather than the industry as a whole.
  • Maturity of the Certification Process – Having gone through the assessor training, I feel this is perhaps the weakest HITRUST point so far. In starting a certification program from scratch, mistakes are easy to make and are common (just ask the PCI Council). However, PCI DSS was not a voluntary program; compliance was mandatory. Requirements such as submitting complete gap analysis reports to HITRUST (including all found vulnerabilities spelled out in detail) are clearly not going to last, since I can’t imagine any company willing to submit a comprehensive set of their dirty laundry (including all areas where they are not compliant with regulatory requirements) to a for-profit company for their assessment and evaluation. However, I feel that once they begin to get this kind of feedback from HITRUST practitioners, they will make the necessary changes in their approach.
  • Certification Quality Assurance – Not all consulting firms are equal; in fact they differ greatly in the quality of their work. Therefore, HITRUST needs to establish a better-defined QA program, to govern the certification process. Protecting the integrity of the HITRUST certification will be essential for internal auditors to begin considering it in place of alternative third-party audits.
  • First Breach / Legal Challenge – In spite of the fact that HITRUST does not make any representations that regulatory compliance is synonymous with HITRUST certification, the first time a HITRUST-certified company suffers a breach or is a subject to regulatory inquiry, we will see the first official test of the framework. One of the big selling points of the framework is that their interpretations of HIPAA are valid and substantiated by the whole healthcare industry. However, if a judge disagrees with any of their interpretations, this may be very damaging to HITRUST acceptance.

I really want HITRUST to succeed. I think it’s a great initiative that has a lot of promise for the whole industry. However, I think it has a long way to go before it is widely accepted, and the certification process is sufficiently mature to inspire confidence on all sides. My recommendation for all healthcare providers and vendors is to begin looking at HITRUST and seeing how their security controls compare with those specified within the CSF. For those companies that do not have a security program in place and are looking to undergo a HIPAA gap assessment for the first time, I would recommend adopting the CSF. After all, the risks are fairly small, since the framework is based on current standards and not anything new. As to the expense of undergoing a full certification, I would recommend putting that on hold until the framework is more widely accepted, or in cases of service providers, until your customers begin to ask you for it.


What's Happening in the Application Security Arena?

Application security attacks are increasing

According to Gartner, 75% of the attacks are coming though web applications and not through the network. This means greater emphasis needs to be placed on application security. However, this does not appear to be happening.

Application security vulnerabilities are increasing

For the first half of 2009, Cenzic identified about 3,100 total vulnerabilities, which is an increase of over 10 percent from the second half of 2008.  (  Another revealing piece of data: WhiteHat Security has stated that in 83% of the 1,300 websites they scan have had at least one serious vulnerability ( Of the projects NetSPI has done in the application security area, 83% of these projects also had serious findings (serious vulnerabilities are those of HIGH, CRITICAL, or URGENT severity as defined by PCI DSS naming conventions).

What can happen if you do not fix the problems?

The first real risk is the theft of your data or your customers’ data. If applications are not done right, SQL Injection can allow a person (or persons) access to your database. Think TJX and all of the problems they had. Another risk is to your company’s reputation. Given the right situation, a user could be redirected to a site that is not under your control. It could be a porn site or even a site that looks like yours; it just exists to steal your users’ credentials. Your reputation will take years to repair, and the cost to your company may be insurmountable.

What can you do?

Many of the problems can be fixed by training. These do not have to be external training courses; they could just be brown bag lunches that cover specific secure coding techniques. A good place to start is the OWASP web site ( ). This site gives good information on detecting and preventing these vulnerabilities.

Perform code reviews and application vulnerability assessments on a regular basis. Code reviews need to be performed every time the code changes. Application vulnerability assessments need to be done at least annually.

By doing code reviews, and vulnerability assessments, you are helping both your company and your customers.

Discover how NetSPI ASM solution helps organizations identify, inventory, and reduce risk to both known and unknown assets.