Last week Karl Fosaaen described in his blog the various trials and tribulations we went through at a hardware level in building a dedicated GPU cracking server. This week I will be doing a complete walkthrough for installing all the software that we use on our box. This includes the operating system, video drivers, oclHashcat-plus, and John the Ripper. Because we have AMD video cards, the driver installation and compiling John the Ripper sections will be tailored for AMD, sorry Nvidia users.
Installing the OS:
For an operating system, Linux and Windows are going to be the way to go. For a headless server however, Linux is the best way to go. The only downside with Linux is that driver support among video cards, especially AMD, is somewhat lacking to its Windows counterpart. However, the good news is that both AMD and Nvidia have been increasing their support for Linux drivers in recent years.
Any Linux distribution will do, but for our server, we opted for Ubuntu 12.10 64-Bit server edition to do the most minimal setup. Much of the information for the next few sections is from the hashcat wiki.
To start off, download the Ubuntu 12.10 server edition ISO from Ubuntu. We don’t have a cd drive on our server, so we had to copy the ISO to a flash drive. YUMI and UNetbootin make this process painless on Windows and Linux, respectfully. Otherwise, the ISO can be burned to a disc.
Boot up the Ubuntu image, choose your language, and select Install Ubuntu Server.
Navigate through the installation options and select your preferences. For most people, the defaults should be sufficient. Then create your user when the dialog comes up. When the installation reaches the “Partition Disks” section, either manually set them up (if you know what you’re doing) or just use the “Guided – use entire disk” option. We choose not to use LVM on our box, but the option is up to you.
After you are done partitioning your hard drive, write the changes to the disc. If you have an HTTP proxy, enter the information when the dialog appears. If not, then just continue. Next, select if you would like to have automatic updates enabled. We opted not to, but it’s entirely up to you. When the software selection appears, select OpenSSH server by navigating to it with the arrow keys and pressing spacebar to select the option.
None of the other packages are required unless you need them. Press enter to install the software. When the installation is finished, install GRUB to the master boot record and reboot. You should now be booted into your new Ubuntu server!
Setting Up Ubuntu:
Before we install the video drivers, we have to setup our Ubuntu server with X11. This is because the AMD drivers require X11 to interact with video cards to obtain fan speeds and GPU temps, which are very important to know when cracking away.
To begin, ssh into your server and update Ubuntu with the following command:
sudo apt-get update && sudo apt-get upgrade
After Ubuntu has updated, we will need to install a minimal X11 environment that our user can automatically login to when the server is rebooted. This is to ensure that the xserver will always be running and in turn allow continuous cracking without any hiccups.
To keep it simple, a light weight window manager is recommend. Openbox, fluxbox, and blackbox are three simple light weight window managers that we can use. You are by no means restricted to a window manager. If you want gnome, xfce, or kde, those can be installed too. For this installation, we will install fluxbox with lightdm as the display manager. To install these, run the following command:
This should install all the necessary packages for an X11 environment to run. Now that we have an X11 environment installed, we need to let applications from the console know which display we are using. To do this, we set the DISPLAY variable to our current display. The format for the DISPLAY variable is hostname:display. For a local instance, the hostname can be omitted. The default display is usually going to be 0. Run the command below to set your current display to 0.
Add the above command to your bashrc to make it persistent whenever your user logs in. I have run into many issues because I did not have this set. So make sure your bashrc is setup with your correct display location.
Now that our X11 environment is setup, we can install the AMD drivers.
Installing AMD Drivers:
To begin installing the AMD drivers, we need to install some prerequisites. First install unzip with the following command
sudo apt-get install unzip
Next, we need to install the dependencies for fglrx, which is the proprietary Linux driver for AMD on Ubuntu’s repositories. The only difference between fglrx and AMD’s Catalyst drivers is that the latter is newer, but they both require the same dependencies. Run the following command to install the fglrx dependencies:
sudo apt-get build-dep fglrx
If the fglrx dependencies are not installed, the AMD driver installation will fail with this fglrx error:
oclHashcat-plus comes in a 7z format. So we need to install p7zip to extract it.
sudo apt-get install p7zip
Run p7zip with the –d flag to extract a 7z file.
p7zip -d oclHashcat-plus-0.14.7z
Navigate to the newly extracted ocl directory and run one of the Example.sh scripts to test run the cracking process.
If all goes well you should see your cards loading up and the hash getting cracked! If you do not see all your cards being recognized, make sure that your xorg.conf was created properly. Try running the amdconfig command above again to regenerate an xorg.conf.
Next we will install John the Ripper with OpenCL support
Installing John the Ripper
Like oclHashcat-plus, John also supports cracking hashes on GPUs, but it must be compiled with the options to do so. Much of the information here is taken from the john GPU wiki (http://openwall.info/wiki/john/GPU).
Next, install the libssl-dev package from apt-get so that John compiles correctly.
sudo apt-get install libssl-dev
Navigate to the john src directory. Compile john with OpenCL for either 32 bit or 64 bit with
respectfully. John can also be compiled with CUDA support if you have Nvidia cards. The information on how to do that is located on their wiki.
If you get openssl headers not found during compilation, install the libssl-dev package.
Navigate back to the run directory and your newly compiled john binary should be there. You can test that John can use your GPUs by running a test command.
This is guide details one of many possible setups for a GPU cracking server. When all is done, our cracking server built with these specifications works very well. In Karl’s blog here, he describes common ways to obtain hashes to crack on Windows, Linux, and web applications.
PTaaS is NetSPI’s delivery model for penetration testing. It enables customers to simplify the scoping of new engagements, view their testing results in real time, orchestrate faster remediation, perform always-on continuous testing, and more - all through the Resolve™ vulnerability management and orchestration platform.
We help organizations defend against adversaries by being the best at simulating real-world, sophisticated adversaries with the products, services, and training we provide. We know how attackers think and operate, allowing us to help our customers better defend against the threats they face daily.
At NetSPI, we believe that there is simply no replacement for human-led manual deep dive testing. Our Resolve platform delivers automation to ensure our people spend time looking for the critical vulnerabilities that tools miss. We provide automated and manual testing of all aspects of an organization’s entire attack surface, including external and internal network, application, cloud, and physical security.
Our proven methodology ensures that the client experience and our findings aren’t only as good as the latest tester assigned to your project. That consistency gives our customers assurance that if vulnerabilities exist, we will find them.
Is your organization prepared for a ransomware attack? Explore our Ransomware Attack Simulation service.