Lessons Learned From The Kubernetes Man-in-the-Middle Vulnerability
A semi-critical vulnerability was uncovered in the popular container orchestration platform Kubernetes last month: CVE 2020 8554.
I say “semi-critical” because it scores a paltry 6.3 on the Common Vulnerability Scoring System (CVSS). But two things make this vulnerability interesting and worth studying: first, it affects all versions of Kubernetes. Second, it cannot be patched. Whether you have Kubernetes in your wheelhouse or not, you do not want vulnerabilities that cannot be patched, particularly ones that affect all versions of an application.
In this article, I will explore CVE 2020 8554: how it happened, how it was found, and the lessons we can all learn from it.
How the Kubernetes MitM vulnerability happens:
It cannot be patched because it is not an implementation bug, meaning there were no mistakes made in code implementation. It happened because Kubernetes allows any tenant in a multi-tenant cluster, with certain control over their own routing, to reroute the traffic of any other tenants on that cluster.
Kubernetes provides a matryoshka nesting doll of abstraction layers. You can have one or many “clusters,” inside of which are one or many “nodes.” Each node maps to a computer (physical or virtual) running one or many “pods” of one or many “containers.” Inside each container live the software components that comprise your application. Each layer of abstraction has its own scope of policy, or its configuration. Additionally, Kubernetes has configuration for “namespaces” which cut across layers and are useful for providing isolation for the different tenants of your application.
A traditional network will have infrastructure services like Domain Name System (DNS), Address Resolution Protocol (ARP), or Network Address Translation (NAT) to ensure that client requests find their way to the servers it needs. With Kubernetes, client requests must find their way to the software running in the appropriate container (inside the pod, inside the node, inside the cluster). This process can be cumbersome. Kubernetes lets you manage these routes with configuration it calls “services.” You can set up load balancing services and external IP services which function as physical load balancers and the NAT translation that happens at your network’s edge.
The MitM vulnerability can exist here because these services are configured at the pod layer, but you can have pods with different tenants alongside one another.
No one is to blame for the vulnerability. It is a result of two decisions that unknowingly created a gap: 1) to let Kubernetes users configure services in a certain way and 2) To let clusters have multiple tenants. No one could foresee the security vulnerability that these two requirements would create when taken together.
Why not put a limit on the configurations?
Why not have single-tenant clusters? Or why not prevent tenants from altering these services? Many organizations do, but there are limits to the scalability they can achieve. And for those that do, there are third party solutions to help prevent and detect exploits of this kind of vulnerability.
How the Kubernetes MitM vulnerability was found:
“The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny…’” — Isaac Asimov
Architecture flaws are notoriously hard to identify. Finding them typically requires a deep understanding of the security implications of a lengthy set of decisions. This points to the importance of manual penetration testing. Gifted penetration testers or analysts who can perform manual protocol analysis or threat modeling are essential in finding vulnerabilities that tools cannot.
In this case, it was a builder that found the vulnerability. As they write in their blog post on the discovery, Etienne Champetier was deploying a Kubernetes cluster for a client when something that should’ve worked failed. A workaround that also should’ve worked also failed. Finally, something that never should’ve worked was successful. Etienne identified the security implications of the problems and reported them to the Kubernetes security team.
Lessons learned from the Kubernetes MitM vulnerability:
No form of static, dynamic, or interactive scanner could have found this flaw. I can’t help but reflect on our industry’s reliance on lightning–fast scanning of applications to keep defects from hitting production. Lightweight security testing via scanners is a valuable tool. A vital one, in fact. But it is not the whole toolbox. Only a skilled technologist who was willing to get elbows-deep in the technology could have discovered this flaw. Manual security testing of applications is critical to think like a real-world adversary.
Learn more about NetSPI’s approach to application penetration testing.