Last week Karl Fosaaen described in his blog the various trials and tribulations we went through at a hardware level in building a dedicated GPU cracking server. This week I will be doing a complete walkthrough for installing all the software that we use on our box. This includes the operating system, video drivers, oclHashcat-plus, and John the Ripper. Because we have AMD video cards, the driver installation and compiling John the Ripper sections will be tailored for AMD, sorry Nvidia users.

Installing the OS:

For an operating system, Linux and Windows are going to be the way to go. For a headless server however, Linux is the best way to go. The only downside with Linux is that driver support among video cards, especially AMD, is somewhat lacking to its Windows counterpart. However, the good news is that both AMD and Nvidia have been increasing their support for Linux drivers in recent years.

Any Linux distribution will do, but for our server, we opted for Ubuntu 12.10 64-Bit server edition to do the most minimal setup. Much of the information for the next few sections is from the hashcat wiki.

To start off, download the Ubuntu 12.10 server edition ISO from Ubuntu. We don’t have a cd drive on our server, so we had to copy the ISO to a flash drive. YUMI and UNetbootin make this process painless on Windows and Linux, respectfully. Otherwise, the ISO can be burned to a disc.

Boot up the Ubuntu image, choose your language, and select Install Ubuntu Server.

Navigate through the installation options and select your preferences. For most people, the defaults should be sufficient. Then create your user when the dialog comes up. When the installation reaches the “Partition Disks” section, either manually set them up (if you know what you’re doing) or just use the “Guided – use entire disk” option. We choose not to use LVM on our box, but the option is up to you.

After you are done partitioning your hard drive, write the changes to the disc. If you have an HTTP proxy, enter the information when the dialog appears. If not, then just continue. Next, select if you would like to have automatic updates enabled. We opted not to, but it’s entirely up to you. When the software selection appears, select OpenSSH server by navigating to it with the arrow keys and pressing spacebar to select the option.

None of the other packages are required unless you need them. Press enter to install the software. When the installation is finished, install GRUB to the master boot record and reboot. You should now be booted into your new Ubuntu server!

Setting Up Ubuntu:

Before we install the video drivers, we have to setup our Ubuntu server with X11. This is because the AMD drivers require X11 to interact with video cards to obtain fan speeds and GPU temps, which are very important to know when cracking away.

To begin, ssh into your server and update Ubuntu with the following command:

sudo apt-get update && sudo apt-get upgrade

After Ubuntu has updated, we will need to install a minimal X11 environment that our user can automatically login to when the server is rebooted. This is to ensure that the xserver will always be running and in turn allow continuous cracking without any hiccups.

To keep it simple, a light weight window manager is recommend. Openbox, fluxbox, and blackbox are three simple light weight window managers that we can use. You are by no means restricted to a window manager. If you want gnome, xfce, or kde, those can be installed too. For this installation, we will install fluxbox with lightdm as the display manager. To install these, run the following command:

sudo apt-get install fluxbox lightdm-gtk-greeter x11-utils

This should install all the necessary packages for an X11 environment to run. Now that we have an X11 environment installed, we need to let applications from the console know which display we are using. To do this, we set the DISPLAY variable to our current display. The format for the DISPLAY variable is hostname:display. For a local instance, the hostname can be omitted. The default display is usually going to be 0. Run the command below to set your current display to 0.

export DISPLAY=:0

Add the above command to your bashrc to make it persistent whenever your user logs in. I have run into many issues because I did not have this set. So make sure your bashrc is setup with your correct display location.

Now that our X11 environment is setup, we can install the AMD drivers.

Installing AMD Drivers:

To begin installing the AMD drivers, we need to install some prerequisites. First install unzip with the following command

sudo apt-get install unzip

Next, we need to install the dependencies for fglrx, which is the proprietary Linux driver for AMD on Ubuntu’s repositories. The only difference between fglrx and AMD’s Catalyst drivers is that the latter is newer, but they both require the same dependencies. Run the following command to install the fglrx dependencies:

sudo apt-get build-dep fglrx

If the fglrx dependencies are not installed, the AMD driver installation will fail with this fglrx error:

Now we can grab the latest version of the AMD Catalyst drivers from the AMD Catalyst™ Proprietary Display Driver page.

The latest version at this time is 13.1. It should also be noted that oclHashCat-plus requires a specific version of Catalyst installed, which at this point is 13.1. So we’re good there.

Fetch the AMD driver with wget and then unzip it:

wget https://www2.ati.com/drivers/linux/amd-driver-installer-catalyst-13.1-linux-x86.x86_64.zip

unzip amd-driver-installer-catalyst-13.1-linux-x86.x86_64.zip

There should now be a .run file in the directory. Execute the file while running as root.

sudo sh amd-driver-installer-catalyst-13.1-linux-x86.x86_64.run

Select the default options on all the dialog boxes and let the driver install.

After it is done, create a new xorg.conf by running:

sudo amdconfig –adapter=all –initial -f

Then we are going to setup our user to automatically login to fluxbox when the server boots up.

Open the lightdm.conf file in /etc/lightdm/ as root and add the following lines:

[SeatDefaults]
greeter-session=lightdm-gtk-greeter
user-session=fluxbox
autologin-user=USER
autologin-user-timeout=0

Reboot the server and your user should be automatically logged into fluxbox.

When the server boots up run

amdconfig –list-adapters

and

amdconfig –adapter=all –odgt

to verify that all your cards and their temperatures can be seen.

Now that the AMD drivers are installed, we can install our cracking software.

Installing oclHashcat-plus

Download the latest oclHashcat-plus. We will just wget the latest version to our box.

wget https://hashcat.net/files/oclHashcat-plus-0.14.7z

oclHashcat-plus comes in a 7z format. So we need to install p7zip to extract it.

sudo apt-get install p7zip

Run p7zip with the –d flag to extract a 7z file.

p7zip -d oclHashcat-plus-0.14.7z

Navigate to the newly extracted ocl directory and run one of the Example.sh scripts to test run the cracking process.

If all goes well you should see your cards loading up and the hash getting cracked! If you do not see all your cards being recognized, make sure that your xorg.conf was created properly. Try running the amdconfig command above again to regenerate an xorg.conf.

Next we will install John the Ripper with OpenCL support

Installing John the Ripper

Like oclHashcat-plus, John also supports cracking hashes on GPUs, but it must be compiled with the options to do so. Much of the information here is taken from the john GPU wiki (https://openwall.info/wiki/john/GPU).

First download the Accelerated Parallel Processing SDK from AMD. 32 bit and 64 bit are supported, so make sure you download the correct one for your architecture.

Copy the file to your server with scp or if you’re on Windows, WinSCP.

Next, extract the APP SDK file.

tar xvf AMD-APP-SDK-v2.8-lnx64.tgz

Then run the Install-AMD-APP.sh as root.

sudo ./Install-AMD-APP.sh

Reboot the server.

After the APP SDK has been installed, download the latest version of John. We will be using the jumbo version.

wget https://www.openwall.com/john/g/john-1.7.9-jumbo-7.tar.gz

Extract john with the following command:

tar xvf john-1.7.9-jumbo-7.tar.gz

Next, install the libssl-dev package from apt-get so that John compiles correctly.

sudo apt-get install libssl-dev

Navigate to the john src directory. Compile john with OpenCL for either 32 bit or 64 bit with

make linux-x86-opencl

and

make linux-x86-64-opencl

respectfully. John can also be compiled with CUDA support if you have Nvidia cards. The information on how to do that is located on their wiki.

If you get openssl headers not found during compilation, install the libssl-dev package.

Navigate back to the run directory and your newly compiled john binary should be there. You can test that John can use your GPUs by running a test command.

Conclusion

This is guide details one of many possible setups for a GPU cracking server. When all is done, our cracking server built with these specifications works very well. In Karl’s blog here, he describes common ways to obtain hashes to crack on Windows, Linux, and web applications.

Eric Gruber

Intro

This winter, we decided to create our own dedicated GPU cracking solution to use for our assessments. It was quite the process, but we now have a fully functional hash cracking machine that tears through NTLMs at roughly 25 billion hashes per second (See below). While attempting to build this, we learned a lot about pushing the limits of consumer-grade hardware.

We’ve recently updated this blog with more recent info – https://blog.netspi.com/gpu-cracking-rebuilding-box/

Goals

We set out to build a cracking rig with four high end video cards (AMD Radeon HD 7950) to run oclHashcat. We also wanted this solution to be rack mountable, so that it would be easy to store in our data center. As it turns out, there are not a ton of video card friendly server cases. We were only able to find a few GPU cracking friendly cases, but most of them cost more than the rest of our cracking hardware combined. If you have the money to spend, we would recommend going with the special case to save yourself from other issues, but this isn’t really an option for everyone. The reason why we recommend this is that the cards themselves do not take well to being lined up all together on a standard ATX motherboard. The fans tend to stick out further than they should and end up hitting the next card in the row. On top of that, the cramped conditions lead to overheating cards and cracking jobs stopping. The specialized cases have enough space to avoid these issues, making it easier to set up a box.

We opted for an “open air” configuration for our cracking box. This was primarily driven by trying to mimic the setups of bitcoin mining rigs that we had seen online. I will say that this is not the prettiest option for housing all of these cards. However, it is one of the most efficient ways to space the cards out for cooling. With the “open air” setup, we’re able to connect riser cables to two of the cards and keep the other two cards down on the board. These riser cables can have their own problems. We ended up opting for one (16x to 1x) riser cable and a different (16x to 16x) riser cable that has some modifications for voltage. The 16x to 16x cable has a 12 volt molex adapter soldered to the 12 volt pins on the riser slot.

While this looks a little hackish, it actually works quite well. We had to do this to supplement the voltage from the motherboard, as it was unable to pull proper voltage for all four cards (with two riser cables). I should also mention that there is some crafty engineering taking place to suspend the two cards above the board. This was accomplished with several zip ties and a modified piece of wire-mesh shelving.

I should also note that this whole rig is tied down (with stand-offs) to an old rack mount shelf. All in all, this setup works quite well. We can have all four cards running at full speed and the the hottest card will top out at 85° Celsius. We’re very aware of the fact that this looks insane. It’s hopefully a temporary solution. Eventually, we’re looking at securing a single rail to the rack to screw the cards into.

As for performance, here’s our current averages for hash cracking (OCL in Brute-Force mode):

MD5 – ~16000.0M/s
NTLM – ~25500.0 M/s
SHA1 – ~7900.0M/s

5 Tips for Building Your Own

So if you’re planning on putting together your own GPU cracking rig, here’s some steps that you may want to take to make it easier.

1. Look into a nice GPU server case and motherboard combo like this one https://www.newegg.com/Product/Product.aspx?Item=N82E16816152125
   1. These will be spendy (~$3,500+ for the combo, cards not included) but they are meant for this kind of setup.
2. Look at what the bitcoin miners are doing.
   1. Our “open-air” setup is actually pretty similar to most mining rigs that I can find.
   2. Replicate their parts list for your setup, if it works for them, it “should” work for you.
3. Plan everything out as best you can.
   1. From components and case layout to power and cooling requirements.
   2. Measure twice and cut once to avoid returns, repairs, and rebuying parts.
4. Devote a resource to the project
   1. Intern not busy enough? Have them build the cracking machine.
   2. Find the person that plays more PC games than you.
      1. They may know more about the cards and multi-GPU setups.
5. Don’t get discouraged if your set up isn’t working.
   1. We didn’t get it right on the first try, but we eventually got there.

Check out GPU Cracking: Setting up the Server by Eric Gruber on how to configure your cracking box to see all of the cards and run the cracking software.

Karl Fosaaen