As a part of our ongoing work to secure cloud computing infrastructure, we delved into the inner workings of some popular Kubernetes add-ons. Our first subject of research was Istio, a popular service mesh add-on.
Istio is an open-source service mesh for Kubernetes that manages communication between microservices. It provides traffic management, security, and observability features without requiring code changes. Istio simplifies complex networking tasks, enhances security, and offers detailed insights into service interactions, improving overall application reliability and performance.
Our research focused on leveraging features of said add-ons to either gain additional access, escalate privileges in the cluster, or hide malicious privileged workloads “in the noise” of a cluster by abusing various features to hide in plain sight. While looking into Istio, we wanted to list all of the avenues an attacker could take advantage of in order to gain control of a cluster following a successful exploitation of a workload.
In this blog post, we cover the Istio feature that we chose to focus on in our research: the ProxyImage annotation. We share our research process, findings, possible ramifications, and the disclosure and remediation process.
The Impact of K8s Add-ons on Security
Kubernetes add-ons are components that extend the functionality, scalability, security, and management capabilities of a cluster. Popular add-ons include Prometheus, Grafana, Istio, Linkerd, and Argo. If the cluster is a managed cluster such as EKS, add-ons exist to better integrate the cluster with AWS and help to offer easy methods to manage persistent volumes. Similar add-ons exist for other cloud providers as well.
When add-ons are installed, they often extend the capabilities of a cluster via operator patterns processing resources specified with custom resource definitions (CRDs). Other add-ons may utilize admission webhooks to modify workloads to conform with a desired pattern. Istio, for example, can be deployed in a way that will modify workloads to include components necessary for Istio to function. Each of these add-ons increases configurability and usability of the cluster but also increases the attack surface. Each of these new configuration options is a potential avenue that could be abused to gain additional access to a cluster.
Istio Overview
Istio is an open-source, service mesh platform that helps organizations manage, secure, and observe microservices-based applications. A service mesh is a configurable infrastructure layer that sits between microservices, managing communication, security, and observability between them. It acts as a network of proxies that intercept and manage traffic between services.
Istio provides a uniform way to connect, manage, and secure microservices, allowing developers to focus on writing code without worrying about underlying infrastructure. According to the main Istio page, “Istio extends Kubernetes to establish a programmable, application-aware network.”
Istio Details
Istio’s modus operandi is rather complex. When configured to operate in sidecar mode, it injects a sidecar container that acts as a transparent proxy grabbing all traffic in and out of the workload container. In addition to the sidecar container, an initialization container is created to modify the traffic flow of the underlying node in order for the proxy container to receive all ingress and egress traffic associated with the workload. The topology shaping is being done by a series of intricate iptables configurations, which, in turn, creates the appearance of a “flat network.” Naturally, such “network tricks” can cause errors, and in order to debug such errors, a cluster admin needs to have access to either an environment or some kind of tool that enables the admin to debug the setup, run arbitrary commands, and use tools to debug issues in the service mesh. Since we are discussing a Kubernetes cluster, and not a single Linux machine, having the proper infrastructure to run these debugging processes is more complex than usual for multiple reasons such as containerization, permissions, namespaces, etc.
While exploring the Istio documentation, we found out there are various annotations that can be invoked by the user (a non-admin cluster user) in order to affect various resources inside the service mesh. There were two particular annotations that caught our eye.One is sidecar.istio.io/proxyimage, which allows us to specify which proxy image to use (with regard to a version of Envoy).
Leave a Reply