This is part 2 of my series on “GitOps with Flux v2”.

If you’re not familiar with what Flux is and how it helps you build GitOps workflows on Kubernetes, feel free to read part 1 here: “Introduction to GitOps on Kubernetes with Flux v2”.

In today’s guide we will look at Mozilla SOPS and learn how to incorporate it with Flux v2 to store encrypted secrets in our GitOps repositories and have Flux decrypt them automatically during deployments.

To follow along, you will need access to a Flux-enabled Kubernetes cluster.

Introduction to SOPS

If SOPS is old news to you, you can skip ahead to “Enabling SOPS in Flux v2” ⏩

Suppose you want to store a database password in a Kubernetes yaml and check that in to source control.

You could base64-encode the password and upload it to a private git repository that only your team has access to… If your repository is compromised, the attacker could easily read the password without any further measures in place to protect it. That’s where SOPS comes in.

SOPS stands for “Secrets OPerationS”. SOPS allows us to encrypt and decrypt certain parts of our Kubernetes yamls with PGP. This means that we can even make our yaml declarations containing secret data public and not worry about unauthorized parties peeking into the contents of the secret values because they would require the respective PGP private keys to decrypt the data.

ℹ️ Note that SOPS can also handle different file formats (JSON, ENV, INI, etc). In the Kubernetes context, we mainly use it with yaml files as that’s what we work with most of the time.

SOPS basics

In this section we’ll learn how to use SOPS on your local machine.

First you’ll have to create a PGP key with OpenGPG. The following command will guide you through the creation process.

gpg --full-generate-key

⚠️ Make sure not to specify a passphrase in the prompt if you’re going to use this key with Flux. You can use the default selections for most other parameters and just supply your ID information (name + email).

Verify that the PGP key pair was created and note the secret ID:

gpg --list-secret-keys ${your_email_address}
out:sec   rsa4096 2021-02-04 [SC]
out:CFF53C2B937EAFD676F75C48F70573E9355BF63B
out:uid           [ultimate] Leonid Koftun <leonid.koftun@gmail.com>
out:ssb   rsa4096 2021-02-04 [E]

Let’s create a simple Kubernetes secret yaml to demonstrate how to use SOPS with the new GPG key.

cat <<EOF > secret.yaml
out:apiVersion: v1
out:kind: Secret
out:metadata:
out:    name: my-database-secret
out:    namespace: awesome-namespace
out:stringData:
out:    database-password: Password123
out:EOF

Now we’ll create a SOPS config file in our working directory.

cat <<EOF > .sops.yaml
out:---
out:creation_rules:
out:- encrypted_regex: '^(data|stringData)$'
out:  pgp: >-
out:    CFF53C2B937EAFD676F75C48F70573E9355BF63B
out:EOF

This configuration tells SOPS to only encrypt values under the ‘data’ and ‘stringData’ keys inside of yaml files and to use our previously generated PGP key for the actual encryption.

We can now run sops:

# You can encrypt the file in-place
sops --encrypt --in-place secret.yaml
# Or write to a new file
sops --encrypt secret.yaml > encrypted-secret.yaml

The result of the encrypted yaml will look like this:

apiVersion: v1
kind: Secret
metadata:
    name: my-database-secret
    namespace: awesome-namespace
stringData:
    database-password: ENC[AES256_GCM,data:k8GGkwr4AE/CdlM=,iv:tecWFmg0INNY1vRfpdGLsDc+APd6UmKk6AS//U0OjI4=,tag:EEm7DHO7muNgpLOwUZh1Lw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2021-02-22T21:45:13Z'
    mac: ENC[AES256_GCM,data:jhzW3o+XcFZgkvGzMb05GpM3hu1dmhRE74woFIeYQOtOy1jCXCp9WgyHar3XDp+1TkOOaN0myfRMe6uz/WmyyKXMPZNf5i4MlB053UUeL2RFMjaGjFlEgq7kG+aoke7+JVN3vTLCiP9fMb4aV3wfPy3hMp5d10wSmPhcfG6/Fww=,iv:07ToHHvJL5tzI5RZLEkfFj+tqJ1y/3XOlADB9TAIuS0=,tag:xlZWGsK5Isrr6GEpBU7YyA==,type:str]
    pgp:
    -   created_at: '2021-02-22T21:45:13Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMAzkIo7JC/ReAAQ/+KV60yKpfRK/qiVaRLbHwu6iTNy59O23vS4+Qt5tv8B9n
            xW4IxTt4IiXeqZMDt2byJUh9KtncnNKectM5gdxDsFdh4QeChFgVYZsgl0CWG3bY
            JBq6Tc/X4udagIuKYIqE+kXiiSwH3+YlKO5VbvcKJPbKg+daeWQvQvXgfghzRZpL
            6auVpdw2E8K59gtADi5T+0JvCy8WXYRWu9JOP9TSQjMx1LRLXmiVITbDQFRokZmb
            7sGbweCVw6ixEDVDbNZx6rh/sH6MGV+yVVd/KMAUWWkKfvezZgbU4M4i7Bfp4sKT
            fp9I21UL1SERHV+6h+jsU379OYb6QdUQ3s+UU7PqejWzuojG4SaHxwPUCT05Doo7
            Bf7syjGta6BJcDgMW8CTfqO/58wpBCI5m1xqri9KYF1/E3T7qP4P1vYO4XLXKJK+
            Q6ee2pgb3W+y8qI0ev2QYh+lFjqH0mk7Z2L5E3JK/Jwsl+mG3jQBPo4fw/N4Iq1j
            WIel4uL9PgPPAgt13Z6UaaYjmYkfiaJIK0rVaY9ArQwU+ShkqHC9nFKB4wxRrGRA
            jJU/6pDLCGOfiHOrGC873qGpOnhnpEZlwxQEQClzFo+uug0bL9fMmD+UBgxaoT1C
            rrXs0tVkgVlyeYDgIpjuwsZEflfxjy/vuH49VeZETn8iQ+4dpqAXFcQyoAAFNWzS
            XgF3Cl2zDs/SIoRMwvZw+GlaWAX9yBLGjxjJGyA5oXkx03i1sL9+5B154O/iPm5q
            QwRTqwmZ22XmtMycV4cJP4tMC/kPKvCHyv3JUO28FkXfhovh+VHCpghzAFySKxc=
            =Pf3r
            -----END PGP MESSAGE-----            
        fp: CFF53C2B937EAFD676F75C48F70573E9355BF63B
    encrypted_regex: ^(data|stringData)$
    version: 3.6.1

Note that the stringData.database-password is no longer readable and the added sops block. The latter contains metadata for SOPS to allow decrypting the secret back to plain-text with the correct private key.

The encrypted yaml file could now be checked into git and distributed. If we try to kubectl apply the encrypted yaml directly to our cluster, it will fail because the secret is not a valid Kubernetes secret in this form.

We have to decrypt it before deploying it like so:

sops --decrypt encrypted-secret.yaml | kubectl apply -f -

The added security of encryption adds at least two extra steps to our DevOps workflow:

  1. We need to make sure we only add encrypted secrets to source control.
  2. We need to decrypt secrets in our CI/CD scripts before we can deploy to a cluster.

The latter can be automated nicely with Flux v2. Let’s see how it’s done.

Enabling SOPS in Flux v2

Now that we know how SOPS works offline, we will be able to apply the same principe to our GitOps repositories with Flux.

Flux supports SOPS out of the box, we just need to supply it with correct PGP private keys and its controllers will decrypt SOPS-protected yamls during reconciliation.

Create a Kubernetes secret with your private PGP key

We want to export our PGP private key and store that in a Kubernetes secret that Flux can use on the cluster.

We can do that with the following commands:

# Find the ID of the private key.
gpg --list-secret-keys ${your_email_address}
out:sec   rsa4096 2021-02-04 [SC]
out:CFF53C2B937EAFD676F75C48F70573E9355BF63B
out:uid           [ultimate] Leonid Koftun <leonid.koftun@gmail.com>
out:ssb   rsa4096 2021-02-04 [E]

# Export the PGP secret to a new k8s secret
# in the flux-system namespace.
gpg --export-secret-keys \
out:  --armor CFF53C2B937EAFD676F75C48F70573E9355BF63B |
out:kubectl create secret generic sops-gpg \
out:  --namespace=flux-system \
out:  --from-file=sops.asc=/dev/stdin

🐔 🥚 Note that this manual step will likely need to be a part of your Flux bootstrapping routine. This secret can not be installed by Flux itself because of the implied chicken-egg problem.

Setup cluster-side decryption in Flux

The following examples are based on my recent post about GitOps with Flux v2 where we have bootstrapped the flux-system namespace and deployed a new namespace from our GitOps repository.

Our GitOps repo looks something like this. You can browse it on Github.

.
├── cluster
│   ├── awesome-namespace
│   │   └── namespace.yaml
│   └── flux-system
│       ├── gotk-components.yaml
│       ├── gotk-sync.yaml
│       └── kustomization.yaml
└── README.md

To enable SOPS for our GitOps repository we need to edit the gotk-sync.yaml resource:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: flux-system
  url: ssh://git@github.com/sladkoff/home-cluster
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client
  # Enable decryption
  decryption:
    # Use the sops provider
    provider: sops
    secretRef:
      # Reference the new 'sops-gpg' secret
      name: sops-gpg

Nice. All that is left to do is test this by checking-in an encrypted secret like the one we created earlier and push our changes to the remote git repository.

➡️ Example commit on Github.

Once Flux reconciliation is done, you should see the new my-database-secret inside the awesome-namespace.

Summary

We can use SOPS to encrypt and decrypt Kubernetes secrets. We learnt how to do this manually on our local machine and automatically on a k8s cluster with Flux v2.

If any of this helped you in any way, feel free to buy me a coffee ☕

You can also give me feedback in the comments, on Twitter or Instagram.

Thanks 🙏