Welcome to the CrowdStrike® Falcon CompleteTM team’s first “Tales from the Trenches” blog, where we describe a recent intrusion that shows how the Falcon Complete managed detection and response (MDR) service operates as an extension of the customer’s security team to quickly detect, investigate and contain an active attacker before they’re able to complete their goal. Once this intrusion was contained, the Falcon Complete team was able to clean and remediate the affected hosts without a need for reimaging or disrupting the customer’s business operations.
With the ongoing COVID-19 pandemic crisis, many organizations have had to act quickly to ensure their security operations are capable of operating 100% remotely. In many cases, this has led to an increase in misconfigured services and applications that are publicly exposed. A simple misconfiguration of a publicly accessible server/application can quickly lead to adversarial compromise.
Join us as we take you step-by-step through our methodologies — including the initial detection, triage and investigation that ultimately led to us thwarting the threat actor and bringing the customer back to a clean and operational state.
Table of Contents
The Initial Detection
The incident began when Falcon Complete received a high-severity machine learning (ML) detection for the unknown binary wmiic.exe on a host within the customer’s environment. The binary appeared to be attempting to masquerade as the legitimate Windows utility wmic.exe, which is a command-line utility used to interface with Windows Management Instrumentation (WMI). Reviewing the detection details further indicated that the unknown binary was quarantined on attempted execution, effectively neutralizing the threat.
For many organizations, the story would stop right there. However, the Falcon Complete team thoroughly reviewed the detection and its associated process explorer tree to identify any further activity. Reviewing this tree revealed that this binary had a parent process of cmd.exe, and a grandparent process of an unknown and suspicious binary, azure bak2.exe. Leveraging CrowdStrike’s internal malware analysis platform, we were able to determine that the grandparent process of azure bak2.exe was actually a self-extracting .rar file. On execution, it wrote numerous binaries to disk including an XMRig coinminer variant (this was the initial wmiic.exe binary), a remote administrative tool (AmmyyAdmin), as well as GMER, which is a security tool that can be used to tamper with the operation of other security products or rootkits.
Figure 1: WMIIC.EXE process execution tree
After this quick triage of the detection details and process execution tree, it became clear that a more thorough investigation was warranted, and we attempted to remotely connect to the affected host via Falcon’s Real Time Response (RTR) for additional investigation and remediation efforts. One of the key differentiators of Falcon Complete is the ability to remotely investigate and perform remediation or countermeasures on malicious activity as opposed to simply escalating to the customer for review and action. This capability could mean the difference between additional compromise and the successful containment of a bad actor.
While attempting to connect to the affected host, we discovered that the host was offline. Although an offline host is not always a suspicious or malicious sign in itself, we identified this host as a domain controller, and therefore the outage was likely anomalous. We immediately reached out to the customer to validate whether this server being offline was intentional or potentially related to the malicious activity.
Once the customer confirmed that the domain controller outage was not expected, the severity of the investigation increased, and we shifted into the scoping phase of the incident response to help identify additional malicious activity and have a better understanding of the extent of the compromise.
Investigation with Endpoint Detection and Response (EDR) Data
The CrowdStrike Endpoint Activity Monitoring (EAM) application gives the Falcon Complete team and Falcon customers the ability to gain real-time insight into attacks and to search the execution data collected by Falcon InsightTM EDR. This provides a powerful capability to quickly investigate and scope the extent of compromise in an intrusion. We leverage EAM to quickly search for known indicators of attack (IOAs) across all endpoints within a customer’s environment. Once additional hosts are identified, further investigation can be performed to potentially identify additional IOAs and continue scoping activities. The scoping phase is an iterative process we perform until we have a full understanding of the extent of compromise, at which point we shift to containment and remediation activities.
Quickly searching for known IOAs led to discovering numerous other hosts with the same files being written to disk around the same time frame.
Figure 2: EAM output for IOC query across the customer’s environment. Each line indicates a unique host that has been seen with the binary
With these new compromised hosts identified, we began to look for commonalities and discovered that the same account was being used to write these files to disk. Using this new indicator, we performed additional EAM queries to identify any hosts with a successful authentication via the compromised account.
When we investigated the lateral movement, we determined that the authentication to each host was via logon type 10, indicating access via Remote Desktop Protocol (RDP), and we obtained a list of remote IP addresses used as a source of the malicious activity.
This led to the identification of an unknown/unmanaged internal IP address, at which point we quickly worked with the customer to locate the unmanaged host and install Falcon for additional investigation. The customer also identified the account as having Windows Domain Administrator privileges and took action to disable it.
In traditional incident response investigations, many responders recommend against taking an aggressive whack-a-mole approach such as disabling accounts before full scoping is performed. However, with adequate deployment of the CrowdStrike Falcon® lightweight agent throughout the environment, and the real-time EDR visibility provided via Falcon Insight, taking a more aggressive approach is acceptable since any additional malicious activity can be quickly identified and contained.
Figure 3: EAM query based on logon type 10 events
The Rabbit Hole
To recap what we have covered so far: A single detection that after initial glance may have seemed less critical quickly led our investigator to an active, interactive attacker that had stealthily compromised numerous hosts.
We were able to quickly run specific EAM queries to uncover further affected hosts and additional IOAs that could be used for scoping the incident. Additionally, it was clear that a domain admin account was compromised and being used to move laterally throughout the network from an unmanaged host. Our next goal was to find the “patient zero” host used for initial access, contain it and then remediate the host in order to kick the bad actor out of the customer environment for good.
The customer was able to quickly locate the unmanaged host, which was also a domain controller. Based on the previous EAM query results, we identified this host as the main source of RDP connections to the other compromised systems. After working with the customer to install the Falcon agent, we connected to the host via RTR to perform an additional investigation, which did not result in any new IOAs.
We decided to dig further and investigate the security event logs of this domain controller, and quickly identified a type 10/RDP logon using the compromised domain administrator account from yet another unmanaged host in the customer environment. Based on the timestamp of the authentication, it appeared that this was now the earliest identified activity, and the priority became locating and installing a Falcon agent on this unmanaged endpoint. We once again worked with our customer to quickly locate the host in question and install the Falcon agent.
Patient Zero
As soon as we connected to this new Falcon-managed host, we immediately began investigating for some of the known binaries and malicious activity that were identified on other compromised hosts in the environment. This provided the investigative lead for us to pivot and assist with identifying additional malicious activity, including multiple unidentified binaries and reconnaissance tools that were not present on any other host in the environment.
Using RTR to inspect the network configuration via built-in commands, we determined that this host was externally facing, and had numerous established connections on port 3389 (RDP) coming from foreign IP addresses. An inspection of security event logs indicated that the system had been compromised via a brute-force RDP password-guessing attack. Based on this discovery, we were fairly confident that we had identified “patient zero.”
Following this, we initiated a network containment in order to sever the network connection from this host and the public internet. Once a host is network contained, all network connections are terminated with the exception of the CrowdStrike cloud, which allows the Falcon Complete team to continue performing investigation and remediation activities in a safe manner. We then informed the customer of the network containment.
Time to Act
Now that we had severed the threat actor’s connection to the environment, the final phase in our response was to perform remediation on the impacted endpoints and restore normal business operations. Stay tuned for Part 2 of this blog as we walk you through our remediation efforts, highlighting Falcon Complete’s ability to directly eject a threat actor and bring the customer’s environment back to a pristine operational state!
Leave a Reply