Exploring Container Security: Vulnerability management in open-source Kubernetes

When it comes to open source software (OSS) like Kubernetes, some people get nervous not knowing everyone who’s worked on the code in the project. “How can I trust something when anyone can contribute?” “If there’s a vulnerability, who’s paying attention?”

In fact, many OSS projects have robust security teams and rigorous vulnerability management processes, just like you’d expect to find in proprietary software. For Kubernetes, a dedicated Product Security Committee oversees the security response process. Learning about this process will give you a better understanding about the project, what to do if you suspect an issue, and your role in vulnerability response as a downstream Kubernetes user.

The Product Security Committee is a group of core maintainers, many with security-specific roles, nominated by other core maintainers and technical advisors within the community. The Committee’s role is to respond to any and all emails about a potential vulnerability, according to a documented response process. Here’s an overview.

1. Triage the disclosure
When the team receives a disclosure, it begins investigating whether the submission is a real issue or just a bug without security implications. If the committee confirms that it’s an issue, it then leads the development and release of a patch and notifies the community.

2. Assess the impact of the vulnerability
One key initial step is to determine a vulnerability’s potential impact. This is usually represented as a CVSS score and the documented severity criteria for Kubernetes vulnerabilities. This score looks at criteria like how easily the issue can be exploited, the consequences if it is exploited, and the privileges required to exploit it. A score under 4.0 is considered low; between 4.0 and 7.0 is medium; between 7.0 and 9.0 is high, and above 9.0 is critical. This CVSS score acts as a rough barometer for how issues should be prioritized and with what urgency, with the caveat that the vulnerability’s severity can vary depending on the specific Kubernetes deployment configuration and environment.

3. Generate a fix
Next, the Product Security Committee works with the relevant developers to generate a fix. If the vulnerability involves components from other open source projects, the team works with security groups within those projects.

There are times when a security fix can happen in the open, as part of a normal patch release. But if the vulnerability is severe enough, the patch will be developed and tested in private. At this point, only those who need to know should be aware of the vulnerability; you wouldn’t want someone with malicious intent to have early knowledge of an unpatched issue.

For fixes that follow the private release process (and some that don’t), the release of a security patch will include an explanatory announcement to kubernetes-security-announce. And when necessary, a retrospective or postmortem is also released to the Kubernetes community, spelling out the steps taken, response timeline, and any other relevant details.

4. Roll out the fix
The next step is to roll out notifications to certified Kubernetes distributors. Because distributors are responsible for what end users receive, it’s important that they have the opportunity to patch early (if a patch is needed) and prepare their own notifications. Like many open source projects, Kubernetes has a private list of distributors who get embargoed security notifications; anyone who meets the criteria can sign up for these messages.

So what does all this mean for you as a Kubernetes user
First of all, make sure your team knows about the Product Security Committee’s process–it should help anyone who’s unsure about open source security feel more confident in Kubernetes. The Product Security Committee is a robust and responsive group of Kubernetes and security experts working on your behalf, complete with an on-call rotation during working hours.

Second, if you’re deploying a container on Kubernetes and you notice something that makes you suspect a security issue, you should notify the Product Security Committee right away, safe in the knowledge that it’ll follow a prescribed process to get to the bottom of things. For more information about when and how to report a vulnerability, check out Kubernetes Security and Disclosure Information.

It’s also important to know your provider’s communication policies with regard to vulnerabilities. To that end, be sure you know the answer to the following questions:

• If there’s a problem, when will my provider contact me?

• Where are security bulletins posted for the services I’m consuming?

• How can I find out when patches will be available for those services?

• Where can I find information about vulnerabilities for what I run in my environment?

When appropriate, Google Cloud publishes security bulletins for Google Kubernetes Engine (GKE) and sends email notifications to affected users. If we can make patches on their behalf or the problem is one for which Google Cloud is responsible, we’ll take care of fixes. GKE users should defer to the GKE security bulletins as their source of truth; it’s updated whenever a new issue is severe enough to warrant attention.

In short, as a Kubernetes end user you should:

1. Share the Kubernetes response and reporting processes with your team
2. Report any future suspected issues through the Kubernetes disclosure process
3. Learn about your provider’s disclosure and patching policies

By answering these questions and familiarizing yourself with the Kubernetes security response process, you can be an informed and confident user of open source software and ensure that your organization is taking an active role in your container security. For the latest in Kubernetes security news, be sure to subscribe to kubernetes-security-announce@.