New CrowdStrike Research Challenges Container Predictability Assumptions

• General-purpose Linux utilities — often observed in living off the land post-exploitation activity — are frequently used in popular containerized applications.
• CrowdStrike researchers have identified significant variation in the usage patterns of general-purpose Linux utilities across these containerized applications.
• The findings challenge commonly accepted assumptions about the predictability and immutability of containerized workloads.
• Cybersecurity solutions that solely rely on the perceived predictability and immutability of containerized workloads are ineffective.

Cloud workloads — and containers in particular — are often seen as immutable entities with predictable behavior. But recent CrowdStrike research suggests that some cloud security solutions rely too much on this premise, leading to suboptimal detection outcomes.

CrowdStrike observes billions of container events each day. The data we collect gives us insights into real-world cloud workload behavior, which challenges these assumptions. In particular, our research demonstrates that Linux utilities often used as living off the land (LOTL) post-exploitation tools are also commonly seen in containers, including widely used containerized applications. In most cases, their usage is not a sign of malicious behavior. 

In this post, we’ll share the results of our research — as well as our interpretation and intuition behind it — and show why a detection approach that solely relies on the perceived predictability of containerized workloads is ineffective.

Are Containerized Applications as Predictable as We Expect Them to Be?

This is the question we set out to answer. To narrow the scope, we focused on well-known containerized applications, including various databases, message brokers, orchestrators, automation servers and web servers, among others. The wide adoption of these applications, along with the many reported vulnerabilities and exploits affecting them, makes them an appealing attacker target. 

Here is the premise we tested: If the core functionality, behavior, configuration options and use cases of an application running in a container are well understood and predictable, it should be possible to detect deviations from the expected behavior and use these detections to identify malicious activity in the container. To narrow the scope even further, we selected applications that aren’t heavily customized by users and therefore shouldn’t demonstrate vastly different behavior across the environments they run in. 

A typical containerized application that we looked at:

• Typically runs in a dedicated single-purpose container
• Typically has several instances running within a company’s environment
• Is widely used across the industry
• Limits user customization options

Using this approach, we selected the following applications for our in-depth analysis: 

• Elasticsearch
• Kafka
• MongoDB
• MySQL
• Nginx
• PostgreSQL
• RabbitMQ
• Redis
• ZooKeeper

Living Off the Land

The post-exploitation phases of an attack often leverage popular Linux utilities that are readily available in containers. These LOTL techniques are not new. We wanted to see how often these utilities are run within our set of selected applications and whether these occurrences can be used as a reliable indicator of a malicious activity in a container where an application is running. 

We selected the following set of common Linux utilities to focus on for this analysis:

Linux Utilities

apt, apt-get, crontab, curl, gpg, groups, groupadd, groupmod, gzip, hostname, id, ifconfig, ip, last, nc, nc.openbsd, nc.traditional, ncat, netcat, netstat, ping, printenv, route, rpm, ss, ssh, tar, uname, uptime, useradd, usermod, wget, whoami, zip

Data-Driven Insights

CrowdStrike regularly processes trillions of data points daily, enabling our data scientists to run large-scale analyses and uncover valuable information. Our analysis of billions of container events across a large set of diverse companies yielded a few insights.

First, Linux utilities are widely used in containers in general. For close to two-thirds of the companies analyzed, we identified at least one of these utilities running within a container in their environment. Further, we counted the number of different utilities seen running in containers, broken down by application container in focus for this analysis. Figure 1 shows the results, sorted from highest to lowest: