This winter, we decided to create our own dedicated GPU cracking solution to use for our assessments. It was quite the process, but we now have a fully functional hash cracking machine that tears through NTLMs at roughly 25 billion hashes per second (See below). While attempting to build this, we learned a lot about pushing the limits of consumer-grade hardware.
We set out to build a cracking rig with four high end video cards (AMD Radeon HD 7950) to run oclHashcat. We also wanted this solution to be rack mountable, so that it would be easy to store in our data center. As it turns out, there are not a ton of video card friendly server cases. We were only able to find a few GPU cracking friendly cases, but most of them cost more than the rest of our cracking hardware combined. If you have the money to spend, we would recommend going with the special case to save yourself from other issues, but this isn’t really an option for everyone. The reason why we recommend this is that the cards themselves do not take well to being lined up all together on a standard ATX motherboard. The fans tend to stick out further than they should and end up hitting the next card in the row. On top of that, the cramped conditions lead to overheating cards and cracking jobs stopping. The specialized cases have enough space to avoid these issues, making it easier to set up a box.
We opted for an “open air” configuration for our cracking box. This was primarily driven by trying to mimic the setups of bitcoin mining rigs that we had seen online. I will say that this is not the prettiest option for housing all of these cards. However, it is one of the most efficient ways to space the cards out for cooling. With the “open air” setup, we’re able to connect riser cables to two of the cards and keep the other two cards down on the board. These riser cables can have their own problems. We ended up opting for one (16x to 1x) riser cable and a different (16x to 16x) riser cable that has some modifications for voltage. The 16x to 16x cable has a 12 volt molex adapter soldered to the 12 volt pins on the riser slot.
While this looks a little hackish, it actually works quite well. We had to do this to supplement the voltage from the motherboard, as it was unable to pull proper voltage for all four cards (with two riser cables). I should also mention that there is some crafty engineering taking place to suspend the two cards above the board. This was accomplished with several zip ties and a modified piece of wire-mesh shelving.
I should also note that this whole rig is tied down (with stand-offs) to an old rack mount shelf. All in all, this setup works quite well. We can have all four cards running at full speed and the the hottest card will top out at 85° Celsius. We’re very aware of the fact that this looks insane. It’s hopefully a temporary solution. Eventually, we’re looking at securing a single rail to the rack to screw the cards into.
As for performance, here’s our current averages for hash cracking (OCL in Brute-Force mode):
MD5 – ~16000.0M/s
NTLM – ~25500.0 M/s
SHA1 – ~7900.0M/s
5 Tips for Building Your Own
So if you’re planning on putting together your own GPU cracking rig, here’s some steps that you may want to take to make it easier.
Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.
YouTube session cookie.
Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.
Analytics cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.
Preference cookies enable a website to remember information that changes the way the website behaves or looks, like your preferred language or the region that you are in.
Unclassified cookies are cookies that we are in the process of classifying, together with the providers of individual cookies.
Cookies are small text files that can be used by websites to make a user's experience more efficient. The law states that we can store cookies on your device if they are strictly necessary for the operation of this site. For all other types of cookies we need your permission. This site uses different types of cookies. Some cookies are placed by third party services that appear on our pages.
Discover why security operations teams choose NetSPI.