Back
Abusing Azure Hybrid Workers for Privilege Escalation – Part 1
On the NetSPI blog, we often focus on Azure Automation Accounts. They offer a fair amount of attack surface area during cloud penetration tests and are a great source for privilege escalation opportunities.
During one of our recent Azure penetration testing assessments, we ran into an environment that was using Automation Account Hybrid Workers to run automation runbooks on virtual machines. Hybrid Workers are an alternative to the traditional Azure Automation Account container environment for runbook execution. Outside of the “normal” runbook execution environment, automation runbooks need access to additional credentials to interact with Azure resources. This can lead to a potential privilege escalation scenario that we will cover in this blog.
TL;DR
Azure Hybrid Workers can be configured to use Automation Account “Run as” accounts, which can expose the credentials to anyone with local administrator access to the Hybrid Worker. Since “Run as” accounts are typically subscription contributors, this can lead to privilege escalation from multiple Azure Role-Based Access Control (RBAC) roles.
What are Azure Hybrid Workers?
For those that need more computing resources (CPU, RAM, Disk, Time) to run their Automation Account runbooks, there is an option to add Hybrid Workers to an Automation Account. These Hybrid Workers can be Azure Virtual Machines (VMs) or Arc-enabled servers, and they allow for additional computing flexibility over the normal limitations of the Automation Account hosted environment. Typically, I’ve seen Hybrid Workers as Windows-based Azure VMs, as that’s the easiest way to integrate with the Automation Account runbooks.
In this article, we’re going to focus on instances where the Hybrid Workers are Windows VMs in Azure. They’re the most common configuration that we run into, and the Linux VMs in Azure can’t be configured to use the “Run as” certificates, which are the target of this blog.
The easiest way to identify Automation Accounts that use Hybrid Workers is to look at the “Hybrid worker groups” section of an Automation Account in the portal. We will be focusing on the “User” groups, versus the “System” groups for this post.
Additionally, you can use the Az PowerShell cmdlets to identify the Hybrid Worker groups, or you can enumerate the VMs that have the “HybridWorkerExtension” VM extension installed. I’ve found this last method is the most reliable for finding potentially vulnerable VMs to attack.
Additional Azure Automation Accounts Research:
- Get-AzurePasswords: Exporting Azure RunAs Certificates for Persistence
- Using Azure Automation Accounts to Access Key Vaults
- Escalating Azure Privileges with the Log Analytics Contributor Role
- CVE-2021-42306 CredManifest: App Registration Certificates Stored in Azure Active Directory
Running Jobs on the Workers
To run jobs on the Hybrid Worker group, you can modify the “Run settings” in any of your runbook execution options (Schedules, Webhook, Test Pane) to “Run on” the Hybrid Worker group.
When the runbook code is executed on the Hybrid Worker, it is run as the “NT AUTHORITY\SYSTEM” account in Windows, or “root” in Linux. If an Azure AD user has a role (Automation Contributor) with Automation Account permissions, and no VM permissions, this could allow them to gain privileged access to VMs.
We will go over this in greater detail in part two of this blog, but Hybrid Workers utilize an undocumented internal API to poll for information about the Automation Account (Runbooks, Credentials, Jobs). As part of this, the Hybrid Workers are not supposed to have direct access to the certificates that are used as part of the traditional “Run As” process. As you will see in the following blog, this isn’t totally true.
To make up for the lack of immediate access to the “Run as” credentials, Microsoft recommends exporting the “Run as” certificate from the Automation Account and installing it on each Hybrid Worker in the group of workers. Once installed, the “Run as” credential can then be referenced by the runbook, to authenticate as the app registration.
If you have access to an Automation Account, keep an eye out for any lingering “Export-RunAsCertificateToHybridWorker” runbooks that may indicate the usage of the “Run as” certificates on the Hybrid Workers.
The issue with installing these “Run As” certificates on the Hybrid Workers is that anyone with local administrator access to the Hybrid Worker can extract the credential and use it to authenticate as the “Run as” account. Given that “Run as” accounts are typically configured with the Contributor role at the subscription scope, this could result in privilege escalation.
Extracting “Run As” Credentials from Hybrid Workers
We have two different ways of accessing Windows VMs in Azure, direct authentication (Local or Domain accounts) and platform level command execution (VM Run Command in Azure). Since there are a million different ways that someone could gain access to credentials with local administrator rights, we won’t be covering standard Windows authentication. Instead, we will briefly cover the multiple Azure RBAC roles that allow for various ways of command execution on Azure VMs.
Affected Roles:
- Virtual Machine Contributor
- Run Command Rights
- VM Extension Rights
- Virtual Machine Administrator Login
- “Log in to a virtual machine with Windows administrator or Linux root user privileges”
- Log Analytics Contributor
- VM Extension Rights
- Previously covered in this blog
- Virtual Machine User Login
- May have rights to login and access the “Run as” PFX file left over in C:\Windows\Temp
- Azure Connected Machine Onboarding
- Can add new ARC machines, which may get the “Run as”
certificate installed
- Can add new ARC machines, which may get the “Run as”
- Azure Connected Machine Resource Administrator
- Can add new ARC machines, which may get the “Run as”
certificate installed - Azure ARC Extension Rights
- Can add new ARC machines, which may get the “Run as”
Where noted above (VM Extension Rights), the VM Extension command execution method comes from the following NetSPI blog: Attacking Azure with Custom Script Extensions.
Since the above roles are not the full Contributor role on the subscription, it is possible for someone with one of the above roles to extract the “Run as” credentials from the VM (see below) to escalate to a subscription Contributor. This is a somewhat similar escalation path to the one that we previously called out for the Log Analytics Contributor role.
Exporting the Certificate from the Worker
As a local administrator on the Hybrid Worker VM, it’s fairly simple to export the certificate. With Remote Desktop Protocol (RDP) access, we can just manually go into the certificate manager (certmgr), find the “Run as” certificate, and export it to a pfx file.
At this point we can copy the file from the Hybrid Worker to use for authentication on another system. Since this is a bit tedious to do at scale, we’ve automated the whole process with a PowerShell script.
Automating the Process
The following script is in the MicroBurst repository under the “Az” folder:
https://github.com/NetSPI/MicroBurst/blob/master/Az/Invoke-AzHybridWorkerExtraction.ps1
This script will enumerate any running Windows virtual machines configured with the Hybrid Worker extension and will then run commands on the VMs (via Invoke-AzVMRunCommand) to export the available private certificates. Assuming the Hybrid Worker is only configured with one exportable private certificate, this will return the certificate as a Base64 string in the run command output.
PS C:\temp\hybrid> Invoke-AzHybridWorkerExtraction -Verbose VERBOSE: Logged In as kfosaaen@notarealdomain.com VERBOSE: Getting a list of Hybrid Worker VMs VERBOSE: Running extraction script on the HWTest virtual machine VERBOSE: Looking for the attached App Registration... This may take a while in larger environments VERBOSE: Writing the AuthAs script VERBOSE: Use the C:\temp\HybridWorkers\AuthAsNetSPI_tester_[REDACTED].ps1 script to authenticate as the NetSPI_sQ[REDACTED]g= App Registration VERBOSE: Script Execution on HWTest Completed VERBOSE: Run as Credential Dumping Activities Have Completed
The script will then write this Base64 certificate data to a file and use the resulting certificate thumbprint to match against App Registration credentials in Azure AD. This will allow the script to find the App Registration Client ID that is needed to authenticate with the exported certificate.
Finally, this will create an “AuthAs” script (noted in the output) that can be used to authenticate as the “Run as” account, with the exported private certificate.
PS C:\temp\hybrid> ls | select Name, Length Name Length ---- ------ AuthAsNetSPI_tester_[Redacted_Sub_ID].ps1 1018 NetSPI_tester_[Redacted_Sub_ID].pfx 2615
This script can be run with any RBAC role that has VM “Run Command” rights on the Hybrid Workers to extract out the “Run as” credentials.
Authenticating as the “Run As” Account
Now that we have the certificate, we can use the generated script to authenticate to the subscription as the “Run As” account. This is very similar to what we do with exporting credentials in the Get-AzPasswords function, so this may look familiar.
PS C:\temp\hybrid> .\AuthAsNetSPI_tester_[Redacted_Sub_ID].ps1
PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\My
Thumbprint Subject
---------- -------
BDD023EC342FE04CC1C0613499F9FF63111631BB DC=NetSPI_tester_[Redacted_Sub_ID]
Environments : {[AzureChinaCloud, AzureChinaCloud], [AzureCloud, AzureCloud], [AzureGermanCloud, AzureGermanCloud], [AzureUSGovernment, AzureUSGovernment]}
Context : Microsoft.Azure.Commands.Profile.Models.Core.PSAzureContext
PS C:\temp\hybrid> (Get-AzContext).Account
Id : 52[REDACTED]57
Type : ServicePrincipal
Tenants : {47[REDACTED]35}
Credential :
TenantMap : {}
CertificateThumbprint : BDD023EC342FE04CC1C0613499F9FF63111631BB
ExtendedProperties : {[Subscriptions, d4[REDACTED]b2], [Tenants, 47[REDACTED]35], [CertificateThumbprint, BDD023EC342FE04CC1C0613499F9FF63111631BB]}
Alternative Options
Finally, any user with the ability to run commands as “NT AUTHORITY\SYSTEM” on the Hybrid Workers is also able to assume the authenticated Azure context that results from authenticating (Connect-AzAccount) to Azure while running a job as a Hybrid Worker.
This would result in users being able to run Az PowerShell module functions as the “Run as” account via the Azure “Run command” and “Extension” features that are available to many of the roles listed above. Assuming the “Connect-AzAccount” function was previously used with a runbook, an attacker could just use the run command feature to run other Az module functions with the “Run as” context.
Additionally, since the certificate is installed on the VM, a user could just use the certificate to directly authenticate from the Hybrid Worker, if there was no active login context.
Summary
In conjunction with the issues outlined in part two of this blog, we submitted our findings to MSRC.
Since this issue ultimately relies on an Azure administrator giving a user access to specific VMs (the Hybrid Workers), it’s considered a user misconfiguration issue. Microsoft has updated their documentation to reflect the potential impact of installing the “Run as” certificate on the VMs. Additionally, you could also modify the certificate installation process to mark the certificates as “non-exportable” to help protect them.
We would recommend against using “Run as” accounts for Automation Accounts and instead switch to using managed identities on the Hybrid Worker VMs.
Stay tuned to the NetSPI technical blog for the second half of this series that will outline how we were able to use a Reader role account to extract credentials and certificates from Automation Accounts. In subscriptions where Run As accounts were in use, this resulted in a Reader to Contributor privilege escalation.
Prior Work
While we were working on these blogs, the Azsec blog put out the “Laterally move by abusing Log Analytics Agent and Automation Hybrid worker” post that outlines some similar techniques to what we’ve outlined above. Read the post to see how they make use of Log Analytics to gain access to the Hybrid Worker groups.
