This winter, we decided to create our own dedicated GPU cracking solution to use for our assessments. It was quite the process, but we now have a fully functional hash cracking machine that tears through NTLMs at roughly 25 billion hashes per second (See below). While attempting to build this, we learned a lot about pushing the limits of consumer-grade hardware.
We set out to build a cracking rig with four high end video cards (AMD Radeon HD 7950) to run oclHashcat. We also wanted this solution to be rack mountable, so that it would be easy to store in our data center. As it turns out, there are not a ton of video card friendly server cases. We were only able to find a few GPU cracking friendly cases, but most of them cost more than the rest of our cracking hardware combined. If you have the money to spend, we would recommend going with the special case to save yourself from other issues, but this isn’t really an option for everyone. The reason why we recommend this is that the cards themselves do not take well to being lined up all together on a standard ATX motherboard. The fans tend to stick out further than they should and end up hitting the next card in the row. On top of that, the cramped conditions lead to overheating cards and cracking jobs stopping. The specialized cases have enough space to avoid these issues, making it easier to set up a box.
We opted for an “open air” configuration for our cracking box. This was primarily driven by trying to mimic the setups of bitcoin mining rigs that we had seen online. I will say that this is not the prettiest option for housing all of these cards. However, it is one of the most efficient ways to space the cards out for cooling. With the “open air” setup, we’re able to connect riser cables to two of the cards and keep the other two cards down on the board. These riser cables can have their own problems. We ended up opting for one (16x to 1x) riser cable and a different (16x to 16x) riser cable that has some modifications for voltage. The 16x to 16x cable has a 12 volt molex adapter soldered to the 12 volt pins on the riser slot.
While this looks a little hackish, it actually works quite well. We had to do this to supplement the voltage from the motherboard, as it was unable to pull proper voltage for all four cards (with two riser cables). I should also mention that there is some crafty engineering taking place to suspend the two cards above the board. This was accomplished with several zip ties and a modified piece of wire-mesh shelving.
I should also note that this whole rig is tied down (with stand-offs) to an old rack mount shelf. All in all, this setup works quite well. We can have all four cards running at full speed and the the hottest card will top out at 85° Celsius. We’re very aware of the fact that this looks insane. It’s hopefully a temporary solution. Eventually, we’re looking at securing a single rail to the rack to screw the cards into.
As for performance, here’s our current averages for hash cracking (OCL in Brute-Force mode):
MD5 – ~16000.0M/s
NTLM – ~25500.0 M/s
SHA1 – ~7900.0M/s
5 Tips for Building Your Own
So if you’re planning on putting together your own GPU cracking rig, here’s some steps that you may want to take to make it easier.
PTaaS is NetSPI’s delivery model for penetration testing. It enables customers to simplify the scoping of new engagements, view their testing results in real time, orchestrate faster remediation, perform always-on continuous testing, and more - all through the Resolve™ vulnerability management and orchestration platform.
We help organizations defend against adversaries by being the best at simulating real-world, sophisticated adversaries with the products, services, and training we provide. We know how attackers think and operate, allowing us to help our customers better defend against the threats they face daily.
At NetSPI, we believe that there is simply no replacement for human-led manual deep dive testing. Our Resolve platform delivers automation to ensure our people spend time looking for the critical vulnerabilities that tools miss. We provide automated and manual testing of all aspects of an organization’s entire attack surface, including external and internal network, application, cloud, and physical security.
Our proven methodology ensures that the client experience and our findings aren’t only as good as the latest tester assigned to your project. That consistency gives our customers assurance that if vulnerabilities exist, we will find them.