In this article, you will learn how to manage secrets securely on Kubernetes in the GitOps approach using Sealed Secrets, ArgoCD, and Terraform. We will use Terraform for setting up both Sealed Secrets and ArgoCD on the Kubernetes cluster. ArgoCD will realize the GitOps model by synchronizing encrypted secrets from the Git repository to the cluster. Sealed Secrets decrypts the data and create a standard Kubernetes Secret object instead of the encrypted SealedSecret CRD from the Git repository.

How it works

Let’s discuss our architecture in greater detail. In the first step, we are installing ArgoCD and Sealed Secrets on Kubernetes with Terraform. In order to install both these tools, we will leverage Terraform support for Helm charts. During ArgoCD installation we will also create the Application that refers to the Git repository with configuration (1). This repository will contain YAML manifests including an encrypted version of our Kubernetes Secret. When Terraform installs Sealed Secrets it sets the private key for secrets decryption and the public key for encryption (2).

Once we successfully install Sealed Secrets we can interact with its controller running on the cluster with kubeseal CLI. With the kubeseal command we can get an encrypted version of the input Kubernetes Secret (3). Then we place an encrypted secret inside the repository with the app deployment manifests (4). Argo CD will automatically apply the latest configuration to the Kubernetes cluster (5). Once the new encrypted secret appears Sealed Secrets detects it and tries to decrypt using a previously set private key (6). As a result, a new Secret object is created and then injected into our sample app (7). That’s the last step of our exercise. We will test the result using the HTTP endpoint exposed by the app.

Prerequisites

To proceed with the exercise you need to have a running instance of Kubernetes. It can be a local or a cloud instance – it doesn’t matter. Additionally, you also need to install two CLI tools on your laptop:

1. kubeseal – the client-side part of Sealed Secrets. You will installation instructions here.
2. terraform – to run Terraform HCL scripts you need to have CLI. You will installation instructions here.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that you need to clone my GitHub repository. This repository contains sample Terraform scripts for initializing Kubernetes. You should go to the sealedsecrets directory. Then just follow my instructions.

Install ArgoCD and Sealed Secrets with Terraform

Assuming you already have the terraform CLI installed the first thing you need to do is to define the Helm provider with the path to your Kube config and the name of the Kube context. Here’s our providers.tf file:

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

Since I’m using Kubernetes on the Docker Desktop the name of my context is docker-desktop. Here’s the variables.tf file:

variable "cluster-context" {
  type    = string
  default = "docker-desktop"
}

Here’s the Terraform script for installing ArgoCD and Sealed Secrets. For Sealed Secrets, we need to set keys for encryption and decryption. By default, the Sealed Secrets chart detects an existing TLS secret with the name sealed-secrets-key inside the target namespace. If it does not exist the chart creates a new one containing generated keys. In order to define the secret with the predefined TLS keys, we first need to create the namespace (1). Then we create the Secret sealed-secrets-key that contains our tls.crt and tls.key (2). After that we may install Sealed Secrets in the sealed-secrets namespace using Helm chart (3).

At the same time, we are installing ArgoCD in the argocd namespace also using Helm chart (4). The chart automatically creates the namespace thanks to the create_namespace parameter. Once we install ArgoCD we can create the Application object responsible for synchronization between the Git repository and Kubernetes cluster. We can also do it using the same Terraform script thanks to the argocd-apps Helm chart (5). It allows us to define a list of ArgoCD Applications inside the Helm values file (6).

# Sealed Secrets Installation

# (1)
resource "kubernetes_namespace" "sealed-secrets-ns" {
  metadata {
    name = "sealed-secrets"
  }
}

# (2)
resource "kubernetes_secret" "sealed-secrets-key" {
  depends_on = [kubernetes_namespace.sealed-secrets-ns]
  metadata {
    name      = "sealed-secrets-key"
    namespace = "sealed-secrets"
  }
  data = {
    "tls.crt" = file("keys/tls.crt")
    "tls.key" = file("keys/tls.key")
  }
  type = "kubernetes.io/tls"
}

# (3)
resource "helm_release" "sealed-secrets" {
  depends_on = [kubernetes_secret.sealed-secrets-key]
  chart      = "sealed-secrets"
  name       = "sealed-secrets"
  namespace  = "sealed-secrets"
  repository = "https://bitnami-labs.github.io/sealed-secrets"
}

# ArgoCD Installation

# (4)
resource "helm_release" "argocd" {
  chart            = "argo-cd"
  name             = "argocd"
  namespace        = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  create_namespace = true
}

# (5)
resource "helm_release" "argocd-apps" {
  depends_on = [helm_release.argocd]
  chart      = "argocd-apps"
  name       = "argocd-apps"
  namespace  = "argocd"
  repository = "https://argoproj.github.io/argo-helm"

  # (6)
  values = [
    file("argocd/applications.yaml")
  ]
}

We store Helm values inside the argocd/applications.yml file. In fact, we are going to apply the same set of YAML manifests into two different namespaces: demo-1 and demo-2. The namespace is automatically created during the synchronization.

applications:
 - name: sample-app-1
   namespace: argocd
   project: default
   source:
     repoURL: https://github.com/piomin/openshift-cluster-config.git
     targetRevision: HEAD
     path: apps/simple
   destination:
     server: https://kubernetes.default.svc
     namespace: demo-1
   syncPolicy:
     automated:
       prune: false
       selfHeal: false
     syncOptions:
      - CreateNamespace=true
 - name: sample-app-2
   namespace: argocd
   project: default
   source:
     repoURL: https://github.com/piomin/openshift-cluster-config.git
     targetRevision: HEAD
     path: apps/simple
   destination:
     server: https://kubernetes.default.svc
     namespace: demo-2
   syncPolicy:
     automated:
       prune: false
       selfHeal: false
     syncOptions:
       - CreateNamespace=true

Now, the only thing is to apply the configuration to the Kubernetes cluster. Before we do that we need to initialize Terraform working directory with the following command:

$ cd sealed-secrets
$ terraform init

Finally, we can apply the configuration:

$ terraform apply

Here’s the output of the terraform apply command:

Encrypt Secret with Kubeseal

Assuming you have already installed Sealed Secrets with Terraform on your Kubernetes cluster and kubeseal CLI on your laptop you can encrypt your secret for the first time. Here’s our Kubernetes Secret. It contains just the single field password with the base64-encoded value 123456. We are going to create the SealedSecret object from that Secret using the kubeseal command.

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
data:
  password: MTIzNDU2

By default, kubeseal tries to find the Sealed Secrets controller under the sealed-secrets-controller name inside the kube-system namespace. As you see we have already installed it in the sealed-secrets namespace under the sealed-secrets name.

We need to override both the controller name and namespace in the kubeseal command with the --controller-name and --controller-namespace parameters. Here’s our command:

$ kubeseal -f sample-secret.yaml -w sample-sealed-secret.yaml \
   --controller-name sealed-secrets \
   --controller-namespace sealed-secrets

The result may be quite surprising. Sealed Secrets doesn’t allow encrypting secrets without a namespace set in the YAML manifest. That’s because, by default, it uses a strict scope. With that scope, the secret must be sealed with exactly the same name and namespace. These attributes become part of the encrypted data. For me, it adds the default namespace as shown below.

Therefore it won’t be possible to decrypt the secret in a different namespace than the namespace set for the input Kubernetes Secret. On the other hand, we want to apply the same configuration in two different namespaces demo-1 and demo-2. In that case, we have to change the default scope to cluster-wide. With that kubeseal parameter the secret can be unsealed in any namespace and can be given any name. Here’s the command we should use to generate the SealedSecret object:

$ kubeseal -f sample-secret.yaml -w sample-sealed-secret.yaml \
   --controller-name sealed-secrets \
   --controller-namespace sealed-secrets \
   --scope cluster-wide

The output file of the command visible above contains the encrypted secret inside the SealedSecret object. Now, we should just add that YAML manifest to our Git repository.

Apply Sealed Secret with ArgoCD

Our sample Git repository with configuration for ArgoCD is available here. You should go to the apps/simple-with-secret directory. You will find there a Deployment, Service and SealedSecret objects. What’s important they don’t have any namespace set. Here’s our SealedSecret object:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
  creationTimestamp: null
  name: sample-secret
spec:
  encryptedData:
    password: AgCW2Nf1gZzn42QQai/zr0VAtb5ZFyOjxMC8ghYcp5bu4EiYmJupX726zTx4XHQThrPgi/jHvzJoymToYJMIYuMegfKmZGcyMMZxJavYFTtlF9CIegPCkD3kjrJMCWcOadyDkBNIIfFAO6ljPwMD+stpsoBZ6WT8fGokxSwE/poKPpWFozC5RImf7HsYjGYVd8onxCySmcJZFYERi2G0qSWBlFDUsJ/ao5vyxIeiS25DBV1Bn475Lgyv6uTfvY6mesvrxw7OWjJmve2xRD/hS87Wp7cdBE264M/NMk1z24VysQr/ezSSI6S14NgzbcWo/hsKwWLmy6u259o8Xot5nVYpo2EhKFm/r62rko0eC2XMkjXhntMLKLpML3mTdadIFK50OauJvyVZS21sgeTlIMeSq6A6trekYyZvBtQaVixIthGHa/ymJXlIBZVJRL7/SJXquaX+J75AXUzPD3Hag8Kt5R5F6TVY2ox8RkMCpAVVAsiztMbyfRgzel6cAfDyj6l5f8GWI2T7gu5uHXgZFwVeyESn3aTO8qqws6NpLlwrtnjLwoCiXXC1Qo39wXaSJoH7fdJwihvOyiwbfaHkjhQwavNHpBoMEbKYQTV6DXSOTN8eeT1ZPoTN8AM+DtMdS2IpvMxZRsgaanh3O7gf5L02nGEq2WyP75s5sLoa7F8dQ27ZUeznqxIrNzrLqNM4dJuqZTbL4AM=
  template:
    metadata:
      annotations:
        sealedsecrets.bitnami.com/cluster-wide: "true"
      creationTimestamp: null
      name: sample-secret
    type: Opaque

Once it will be applied to the cluster, the Sealed Secrets controller will decrypt it and create the Kubernetes Secret object. Our sample app just takes the Kubernetes Secret and prints the value of the password key.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: quay.io/pminkows/sample-kotlin-spring:1.4.30
        ports:
        - containerPort: 8080
          name: http
        env:
          - name: PASS
            valueFrom:
              secretKeyRef:
                key: password
                name: sample-secret

We will test the app’s HTTP endpoint through the Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  type: ClusterIP
  selector:
    app: sample-app
  ports:
  - port: 8080
    name: http

Let’s verify what happened. Go to the ArgoCD dashboard. As you see there are two applications created by the Terraform script during installation. Both of them automatically applied the configuration from the Git repository to the cluster.

Let’s display details of the selected ArgoCD Application. As you see the sample-secret Secret object has already been created from the sample-secret SealedSecret object.

Now, let’s enable port-forward for the Kubernetes Service on port 8080:

$ kubectl port-forward svc/sample-app 8080:8080 -n demo-1

The app is able to display a list of environment variables. We can also display just a selected variable by calling the following endpoint:

$ curl http://localhost:8080/actuator/env/PASS

Final Thoughts

In general, there are two popular approaches to managing secrets on Kubernetes in GitOps style. In the first of them, we store the encrypted value of the secret in the Git repository. Then the software running on the cluster decrypts the value and creates Kubernetes Secret. That solution is represented by the Sealed Secrets and has been described today. In the second of them, we store just a reference to the secret, not the value of the secret in the Git repository. The value of the secret is stored in the third-party tool. Based on the key software running on the cluster retrieves the value.

The most popular example of such a third-party tool is HashiCorp Vault. You can read more about managing secrets with Vault and ArgoCD in the following article. There is also another very promising project in that area – External Secrets. You can expect my article about it soon 🙂