Can Kubernetes Keep a Secret? It all depends what tool you’re using

At Soluto, we have super-devs who have full ownership: from writing code to deploying it to monitoring. When we made the shift to Kubernetes, we wanted to keep our devs independent and put a lot of effort into allowing them to create services rapidly. It all worked like a charm – until they had to handle credentials.

This challenge leads us to build Kamus – an open source, GitOps, zero trust, secrets solution for Kubernetes applications. Kamus allows you to seamlessly encrypt secret values and commit them to source control. But before diving into how Kamus works, let’s do a quick recap of Kubernetes native secrets solution, and why we even need Kamus.

As you may already know, Kubernetes has a built-in object for secret management, with the super surprising name “Secret”. A Kubernetes secret is a simple object that’s stored securely (e.g. encrypted at rest) by the orchestrator, and can contain arbitrary data in key-value format. Here’s an example of what a Kubernetes secret looks like:

apiVersion: v1
data:
 super-secret: aHR0cDovL2pvYnMuc29sdXRvLmNvbS9hcHBseS95MDVYSXEvUHJvZHVjdGlvbi1FbmdpbmVlci1EZXZPcHM=
kind: Secret
metadata:
 name: sensitive-data
 namespace: default
type: Opaque

As you can see, we have the type (Secret) and our data. The value is base64 encoded, so we can also store binary data like certificates. Kubernetes makes it easy to consume secrets by letting you simply mount them onto your container, either as env var – not recommended – or as a file.

What’s wrong with Kubernetes plain Secrets?

Why do we need another solution? First of all, Kubernetes secrets are base64 encoded, not encrypted. This means you cannot commit these files into source control as-is (and this is even specified in the docs). Anyone with access to the repository can grab the secrets and use the secret value (unless this is a threat you’re willing to accept, which can be valid under certain circumstances). So we still need a simple way to store this secret and upload it to production, ideally without any manual operation.

Which leads to my second point: usability. Secrets are critical for normal execution of a service, and a change in one of them can lead to production issues. You need to have good visibility into these secrets – who changed them, when, and what changed – so you can investigate production issues. This is why I’m so much in favor of GitOps, where everything is audited via git. Yes, Kubernetes has audit mechanism, but it’s not as straightforward as git. So when it comes to usability, we need a way to modify secrets that will be transparent and easy to follow.

Thirdly, consider how your service consumes secrets. In Kubernetes, you can consume a secret in one of two ways: mount the secret as an environment variable or as a volume. Mounting the secret as environment variable is the simplest method, but not everyone feels comfortable with storing sensitive data in environment variables (check out this twitter thread to see some opinions).

For this reason, some people (including myself) prefer to mount secret as volumes. This allows you to access the secret content as a normal file in the container. Kubernetes creates one file per key in the secret’s data, and puts the decoded value as the file content. This might work when have one or two keys, but the more keys you have, the harder it gets – you need to read all these files from the application’s code. This is why we usually end up creating a configuration file, base64 encode it, and store it as a single secret value. However, this make it even harder to understand what changed.

Lastly, Kubernetes has great RBAC support, but it has its disadvantages. The existing permissions for secrets are get and set (as well as others that are less relevant to this discussion). Secrets are encrypted at rest, but there is no permission saying “allow the user to get the secret encrypted”. Once a user is allowed to get a secret, she will get it decrypted. Ideally, we’d like to have a zero-trust system where once you encrypt a secret, there’s no easy way to decrypt it. Currently there’s no easy way to achieve that with Kubernetes.

Alternatives to Kubernetes Secrets

There are a few good alternatives out there, like Sealed Secrets and Helm Secrets. Both solutions let you create encrypted secrets that can be committed to source control, and decryption to regular Kubernetes secrets occurs later on (either on the CD or in the cluster). But these solutions aren’t perfect, either. There’s no support for one-time encryption because in order to modify a value in the secret, you must decrypt its content. And all the usability issues we had before still exist.

Introducing Kamus

One day while working on Glue, I used Travis secrets encryption solution. If you’re unfamiliar with how Travis handles secrets, the server creates an RSA key pair per repository, and then exposes the public key. Devs can use the public key to encrypt values that later on can only be decrypted by Travis. This made me think: what if I create a similar solution for Kubernetes? What if I can have a specific RSA key-pair per deployment so I can leverage the same mechanism? And the result… was Kamus.

Let’s take a look at how we can use Kamus, and then deep-dive into how it works and the security of the solution. Kamus encrypts secrets for a specific service account, which is later used by a pod to authenticate Kamus and decrypt them. So the first thing we need to do is to create a service account:

kubectl create sa dummy

Now we can use kamus-cli to encrypt our super secret value for this service account:

kamus-cli encrypt --secret super-secret --service-account some-sa --namespace some-ns --kamus-url https://kamus.my-company.com

The returned value will be the encrypted value. We can now safely commit it to git, and in runtime, use Kamus API and the token of the pod’s service account to decrypt this value. If a different pod with a different service account tries to decrypt the value, the operation will fail.

Decrypting a secret is simple – just use the Kamus init container. The init container will read all the encrypted secrets, use the service account token to decrypt the values with Kamus, and write the decrypted values in a temporary location. To see this in action, try running the example app.

Ready to give it a try?

We at Soluto have been using Kamus in production for the last six months. This has significantly improved the developer experience, and now using secrets is a fun and simple task. This is why we decided to open-source Kamus. Getting started with it is simple – just install it with Helm on a Kubernetes cluster, and you’re done:

helm repo add soluto https://charts.soluto.io

helm install kamus soluto/kamus

You can check out our installation guide for a production-grade deployment.

Security is important to us, and is why we built Kamus. Since we understand how sensitive a project like Kamus can be, as part of the open source release, we also released a threat model. This is where you’ll find all the different threats we detected and how Kamus mitigates these threats. If you find a security issue, we’d like to hear about it – please check out the security.md doc to learn how to report security issues to us.

Keeping secrets secure is critical, but not always simple. Kamus solve this problem by providing a secure GitOps solution to the problem. It lets you tell a secret to Kubernetes, and makes sure that no one else knows this secret. The rest is up to you – will you give Kamus a try?

Related Post

Previous

React Class features vs. Hooks equivalents

Next

Overwhelmed by logs? Kubernetes to the rescue!

7 Comments

  1. Prince

    Why have a solution that works for GCP and Azure and then leave out AWS. What’s your beef?

    • We decided to wait with the implementation for the first user who needs it. It’s because (a) I’m not an AWS export and would like to discuss the implementation and (b) we don’t want to spend too much effort on something that might not be in use. Feel free to respond on the issue if you need AWS support – I’ll be happy to work this out with you.

  2. The link to the theat model part is giving 404.

    It is currently pointing to https://github.com/Soluto/kamus/tree/master/docs/features

    I guess the correct link is https://kamus.soluto.io/docs/threatmodeling/threats_controls/

    • Thanks! This was broken after releasing the new website, I updated the link so it will point to the right place – as you already find out 🙂

  3. dhruva

    Hi,

    We are Rancher with kubernetes, please provide me detailed steps for installing

Leave a Reply

Your email address will not be published. Required fields are marked *

Powered by WordPress & Theme by Anders Norén