Back
Tearing Apart a Datto Device
Datto devices are becoming a popular backup solution for small to medium sized businesses. They are easy to use and well equipped out of the box. We recently found ourselves in an engagement where one of these devices was accessible via the LAN. Gaining access to backups is a bit of a goldmine during an assessment; unrestricted access to file shares, configuration information, extracting hashes from the NTDS.dit file, and a multitude of other things. Anyone familiar enough with Datto devices knows you have a few useful restore options like mounting backups over open SMB shares or even starting your own VM . Let’s review what we found.
Initial Access
These Datto devices have quite a few attack surfaces out of the box, a web control panel for management, SSH is configured, and VNC is enabled by default. The authentication password for VNC, “Northern”, is well documented by Datto, so that didn’t take long to find. This allowed us to connect to the device remotely and luckily, the user was already logged in! However, VNC access isn’t very stealthy, and we wanted SSH access to the device instead.
Via VNC access we noticed the “root” user is managed by Datto. By looking through logs, it appears these accounts have their passwords changed frequently (once/day), so they aren’t prime targets for attacks directly. However, they do appear to get used so Datto must be tracking these changes somewhere. Hmmm…
Moving on, “backup-admin” is an account created for the client use. The password to this account can be viewed in the partner portal (https://auth.dattobackup.com/simplesaml), where clients can also view device status, backup progress, and initiate a remote web connection to the device. This might be an interesting attack vector as well, but we’ll focus on more local attacks.
Another account, “aurorauser” looks interesting and potentially promising. This account is an unprivileged user account, used for automatic login on boot…providing the easy VNC access. Unfortunately, the SSH password didn’t match the VNC password for this user. However, with a bit of research the default password to the account was found, “NorthernLight$”. Throw it into SSH and we’re in!
Pulling out the root
The Datto device is really just an Ubuntu Linux install with LAMP and bash scripts providing a web console and dashboard intelligence. SSH tunnels and rsync are used to connect back to Datto for off-site duplication. Running ‘uname -ar’ shows the OS is Ubuntu 10.04, released February, 2012 with an EOL May, 2013 for desktops and April, 2015 for servers. Privilege escalation was trivial thanks to a CVE-2012-0056. The exploit used was Mempodipper, courtesy of Jason Donenfeld (https://blog.zx2c4.com/749). Thankfully, Datto was kind enough to provide GCC, so simply pop open vim on the box, throw in the source, compile, and we’re root!
Building a backdoor
Where can we go from here? Our first goal to access the backup data was to obtain access to the web console. Poking through some files revealed “/datto/config/local/webaccess”, which is a text file containing MD5 hashes in a “user:hash:” format. Easy enough…just add our username and MD5 password to the file and we’re in! It’s also worth noting that while the newly added user credentials worked, the username did not appear in the list of users within the web console management interface.
With our purpose being to restore data from a backup (SAM file, sensitive files, etc.), we hopped into the “Restore” page in the web console, selected the DC, chose the backup date, and hit the mount button. Uh oh! Datto devices offer backup encryption for agents added to the device. While not enabled by default, in this case the customer enabled it manually requiring a decryption password prior to mounting or accessing any backup data.
Finding the keys
Let’s dig a little deeper into how this encryption system works. Dan Fuhry, an engineer from Datto has provided a nice little “Datto vs. NSA” paper here, which provides some insight. Here is the relevant section:
“My design for Datto’s encryption system provides unique encryption keys per agent, so there is no single key that can decrypt every dataset we have. The key used to encrypt your actual data is the master key, and that master key is only ever stored in an encrypted fashion. It’s completely random – not derived from a passphrase – and no human ever sees it. When you enter your passphrase, your Datto device does some number crunching on that passphrase and some additional data to get a user key. That user key is used to decrypt an encrypted copy of the master key. This gives you the ability to change your passphrase without having to re-encrypt the entire dataset, and have multiple valid passwords per agent.
The important thing to realize here is how vital your passphrase is to decrypting your data. Without it, the number crunching required to find your data is impossibly immense, even for the NSA. Datto doesn’t keep your passphrase anywhere. Therefore, no court can compel us to hand over your unencrypted data, because we don’t have it and can’t get it.”
This all sounds very promising. To summarize, you need to access the decrypted user key (requires passphrase), to then decrypt the master key, to then decrypt the backup data.
But wait, let’s think about this. If the Datto is backing up agents continuously, and you don’t have to type in your passphrase every time, the Datto must be storing this master key somewhere. This is where “sealing” comes in (Datto’s term). Essentially, when you first reboot a Datto, you need to login to the web console and “un-seal” your agents. This means typing in your passphrase so that the Datto can begin encrypting the backups its taking.
Your first thought might be to just dump the live memory from the device. The master key must be in there somewhere! Let’s first learn a little more about what this “un-sealing” process. As mentioned earlier, pretty much all of the functions on the Datto are controlled through either PHP scripts, or bash scripts. The PHP scripts are obfuscated using ion-cube, so we can’t just throw them into a text-editor. Luckily for us, a neat little decoder tool exists (https://phpdecode.blogspot.com/) that will decode the files to a more readable format. It’s not perfect, but it works great for our purposes. Digging through the source, you’ll notice a function named “stashAgentKey()” in the encryptionFunctions.php file.
This snippet tells us that Datto is utilizing Linux shared memory at /dev/shm/ to store the master keys while the agent is unsealed. <3 u Datto. Here is an example dump of that file:
{"_stashSig":"6754d8e33202628d73abd270ca6d64d4acaa0ad4a7214aaa46c050faf5b86514","10.199.16.10":"PARTIALLYREDACTEDe332b4dec93617b98b820620e09285c29c02a4285fcff63e223bc767893e6305a812a1042046b08eba88d40de3e4e74944c5387100a0c779a990848d22","10.199.16.190":"PARTIALLYREDACTEDde2ef30759ffec62f7af649f5dca29c011526f0c3d70056440b46609a4e7ed600332af333454b4509b26a454d350fa5e0a7ca3f5ae7af39e998c05bab9",[...]}
Ahh, that’s much better. Edit: Now that we have the master keys for the agents, we should be able to decrypt the backups! However, that sounds like quite a bit of work. Based on Datto’s response, we learned we obtained the encrypted master key, not the actual master key. Here’s more detail taken from their response to this blog post.
“On a technical point, I will note that what your post concludes is a master key is in fact not, it’s an encrypted master key. At device boot we generate a completely random key and salt, that is then used to encrypt the master key while in memory. Granted, the boot key is in memory as well as the salt and encrypted master key. But now an adversary must know all three. It’s not insurmountable if you’re already in the device shell as root, but it is another hurdle.”
If you haven’t read their response, it’s definitely worth while to look it over. It’s not every day that an organization is proactive and receptive to potential security vulnerabilities regarding their product.
While perusing the source, another potentially interesting rabbit hole…or, “rat” hole was found.
Did someone say RAT?
All of the Datto functions are available through a command line application called “snapctl”. Here are the documented arguments:
add [hostname] -Registers a new agent' . ' removeAgent [hostname] -DESTROYS ALL DATA AND LOG FOR AN AGENT! DANGER! restartAgent [hostname] -Restarts the replay agent (Wait 30 seconds for agent to restart) renameAgent [old-hostname] [new-hostname] -Rename an agent, it\'s ZFS shares, and it\'s offsite shares makeStorage [hostname] -creates zpool for agent command [hostname] [command] [args] [dir] -Runs a physical command on the host system updateAgent [hostname] -Updates the SnapToVM agent to latest version setCompatibility [hostname] [on] -Set compatibility mode (1 = on 0 = off) setWriteCaching [hostname] [on] -Set write caching enabled/disabled (1 = on 0 = off) clearAllCompatibility -Disables share compatibility for all agents. list -Lists all agents protected by this core dirtyUpgrade [hostname] -This function updates the version of ShadowSnap regardless of previous version cleanupAppassure [hostname] -This function will clean up any left over Snap2VM files left in the agent directory restartSystem [hostname] -Reboots the system the agent is running on setEngine [hostname] [engine] -Allows you to set the backup engine start [hostname] -Starts a backup for hostname foregroundStart [hostname] -Starts a backup on the command line for hostname stop [hostname] -Stops the backup for hostname forceFull [hostname] -Forces the next backup to be full forceDiffMerge [hostname] -Forces the next backup to be diffmerge setInterval [hostname] [interval] -Sets the backup interval in minutes getInterval [hostname] -Prints the backup interval status [hostname] -Prints the status of the hostnames backup excludeVol [hostname] [mount] -Excludes a mountpoint from backup. Mountpoint must be a volume includeVol [hostname] [mount] -Includes a mountpoint from backup. Mountpoint must be a volume forceRemoveVol [hostname] [mount] -Removes an excluded mountpoint from the live dataset. Mountpoint must be a volume checkVol [hostname] [mount] -Prints the Exclude status for volume retention [hostname] -Runs ZFS retention operations dryRunRetention [hostname] -Runs a dry run of ZFS retention operations, which will return the points that will be removed offsiteRetention [hostname] -Runs ZFS retention operations on offsite servers setShareAccess [hostname] [username] -Sets all future shares for this hostname to be password protected with access for specified user. A blank user value will disable. createVM [hostname] (snap) (suffix) -Clones a snapshot of a host using an optional suffix. Defaults to the most recent snapshot and a suffix of -active. startVM [hostname] -Starts a virtual machine for specified host stopVM [hostname] -Stops a virtual machine for specified host destroyVM [hostname] (keepClone) (suffix) -Destroys virtual machine and optionally keeps clone setController [hostname] [controller] -Sets the VBox Controller type (IDE or SCSI) getController [hostname] -Prints the controller type updateVBox -Updates VirtualBox to the latest version mountPoint [hostname] (snap) (suffix) -Mounts a cloned snapshot\'s volumes for file restore. Optional snapshot defaults to the most recent. Optional suffix defaults to "[SNAPSHOT#]-file" unmountPoint [hostname] (suffix) -Unmounts a cloned snapshot\'s volumes after file restore. Optional suffix defaults to "[SNAPSHOT#]-file" shareVMDK [hostname] [snap] -Share RAW vmdk file unshareVMDK [hostname] [snap] -Unmounts and de-share raw VMDK shareVHD [hostname] [snap] -Share RAW vhd file unshareVHD [hostname] [snap] -Unmounts and de-share raw VHD convertVHD [path] -Converts a specified image or images in the specified directory to vhd hostOverride [hostname] [newHostname] -Overrides the hostname given to the agent for the path to the Samba share. Set blank value to disable. agentSetConfig [hostname] [key] [value] -Sets an agent configuration variable given the correct key and a valid value. Host must be running. agentGetConfig [hostname] -Returns the current agent configuration for a given host. Host must be running. screenshotOverride [hostname] [cpus] [ram] -Sets the resources to be used for the screenshot VM for this host. (RAM specified in MB) hir [hostname] [path] (cowMerge) -Performs HIR on a target location using the hostname as a source of agent information. Optionally will merge the HIR with the images. clearShares [hostname] -Removes all mapped network drives that point to the datto from the agent machine attachDMCrypt [path] -Attach an encrypted image to dm-crypt detachDMCrypt [path] -Detach a dm-crypt device decryptAgentKey [hostname] -Enter a passphrase to unlock an agent\'s images sealAgent [hostname] -Discard an agent\'s master key from memory addAgentPassphrase [hostname] -Add an additional passphrase that can decrypt an agent removeAgentPassphrase [hostname] -Remove an existing passphrase from an agent encryptAgent [hostname] -Set an agent up for encryption. Any existing snapshots/live dataset will be destroyed. uploadEncryption -Force manual upload of encryption salt/mkey information downloadEncryption -Fetch encryption salt/mkey information and merge with local stash screenshot [hostname] [delay] (snap) -screenshots the agent after delay in seconds using snap (optional) addScreenshot [hostname] [snap] -adds snap to the screenshot queue screenshotAll [hostname] -Screenshot all hostnames on this machine that have not had a screenshot in the last 24 hours killScreenshot [hostname] -kills the screenshot operation cleanupScreenshots (hostname) -Cleans up screenshot remnants. Optionally, specify an agent whose screenshot remnants you want cleaned up. debugScreenshot -Run the whole screenshot process in the foreground testFilesystems [hostname] (path) -Test all backed up filesystems for an agent to verify that they can be mounted. Optionally, specify a path containing a clone of that hostname. bmrInstall -Installs the latest network BMR environment. kzfs -Upgrade the device to use kernel based zfs r8169upgrade -Upgrades the realtek network drivers to version 6.015.00 largeFiles [directory] [minSize] -Show all files in the target directory that are larger than minSize kilobytes timestamp [timestamp] -Convert a linux timestamp to human readable date health (verbose) -Print a device health summary device-key show|hide -Show or hide the Encryption options section on the Admin page runTask -Run a task from the Admin page checkHost [hostname] -Prints connectivity information about host update [hostname] -Updates the agent info about the host info [hostname] -Prints info about the host viewLog [hostname] [lines] -Prints lines number of lines from the end of the log file printErrors [hostname] -Echos the replay log (Should be used with TAIL) getVMX [hostname] -Print the VMX Array data stored about the host getWriters [hostname] -Returns a list of installed VSS Writers clearError [hostname] -Clears last reported error from the system truncateLogs [hostname] -forces a truncate of the exchange and SQL logs setTruncate [hostname] [time] -Sets the log truncate time out in minutes updateOffSite -Updates which points are stored off-site sendWeekly -Send the weekly backup digest updateVols -Transfer sync/volume information offsite syncNetworkConfig -Updates /etc/network/interfaces with current state of physical interfaces totalFreeRam -Gets the true total free memory available. Includes ZFS cache. updateHardwareProfile -Update the model of this device after a field upgrade. updateVbox -Update VirtualBox to the latest supported version. suppressUpdateNotification [hostname] [hours] -Suppresses ShadowSnap update notification for given number of hours agentUpdateStatus [hostname] -Gets current agent update info as JSON data';
That’s a lot of functionality, and immediately you’ll notice some glaring issues. First off, the “command” option sends and executes an arbitrary command to any agent added to the Datto.
command [hostname] [command] [args] [dir] -Runs a physical command on the host system
Wow, remote command execution built into a backup device. It really is fully featured! Let’s run a quick test to try it out:
Perfect! We’re SYSTEM! Presumably this comes from the ShadowSnap agent that Datto requires you to install on agents backing up to the device. Based on the source code, the command doesn’t appear to be used that much. Moving past command execution, this utility has some other useful features. For one, the “screenshot” ability could be very useful for snooping.
screenshot [hostname] [delay] (snap) -screenshots the agent after delay in seconds using snap (optional)
Or how about restartSystem?
restartSystem [hostname] -Reboots the system the agent is running on
Or possibly removeAgent if you are feeling dangerous?
removeAgent [hostname] -DESTROYS ALL DATA AND LOG FOR AN AGENT! DANGER!
However, none of these are quite as interesting as two of the encryption commands.
addAgentPassphrase [hostname] -Add an additional passphrase that can decrypt an agent removeAgentPassphrase [hostname] -Remove an existing passphrase from an agent
So let’s wrap this up. Datto states the because a master key is separate from the user key, that you can have multiple passphrases per agent. The web console doesn’t let you configure this. It requires you to enter the current passphrase in order to change it. But we know that the Datto device already has access to the decrypted master key while an agent is unsealed. So theoretically…
Perfect! We were able to add a temporary passphrase to an existing un-sealed agent, mount a backup with it, demount, and remove the temporary passphrase all without knowing or disturbing the current passphrase.
Now of course this is only scratching the surface of these devices. Here are a few other things we didn’t have time to fully explore:
- Datto downloads scripts over HTTP periodically and executes them blindly, some DNS modifications or MiTM could make this very dangerous.
- Datto’s SSH syncing to the cloud might be vulnerable to different attacks including device traversal.
- How do user passwords get changed remotely?
- Can raw backup Datto be decrypted with the master keys obtained?
- How does Datto open remote HTTP tunnels from the partner site to enable remote login?
Hopefully this this post will be useful to anyone currently using or considering a Datto device.
Update: Datto wrote a great response to the findings reviewed above. They even straighten out some of our erroneous conclusions regarding identifying the encrypted master key (as opposed to the actual master key). The fixes are shown above. Their response can be found here.
