AWS API Gateway Authorizers are a powerful tool to secure API Gateway endpoints.

The configuration of Authorizers can be a bit overwhelming at first. There are many options which have an impact on the security, performance and programming model of your API Gateway.

This article is a collection of three patterns that I have identified from my experience with AWS API Gateway Authorizers, from discussions with peers and colleagues and from existing documentation online.

Please note that this is not a comprehensive guide to API Gateway Authorizers. It’s a collection of abstract concepts that can be used to reason about the different patterns:

1. No-Authorizer Pattern
2. Binary Authorizer Pattern
3. Context Authorizer Pattern

Note that the naming of the patterns is not official. I made them up to make it easier to talk about them.

No-Authorizer Pattern

The first pattern consists of an API Gateway and integration Lambdas in the background.

There is no authorizer Lambda configured in the API Gateway. All requests to the API Gateway are authorized by default and forwarded to the backend Lambdas. The Lambdas are responsible for authorization directly.

In the diagram above, there is an exemplary “External User Service” that is queried by the backend Lambdas to authorize the request.

When to use

• When there is a only handful of Lambdas to integrate with the API Gateway
• When it’s possible to re-use the same authorization code across Lambdas in a maintainable way (same code repository or shared library)
• When the authorization call does not take a considerable amount of time to execute
• When authorization logic should be testable together with the business logic
• When each invocation should use the up-to-date authorization information

Cacheable vs Non-Cacheable Authorizers

Before we dive into the next patterns, let’s talk about caching.

API Gateway Authorizers can be configured to cache the authorization result for a given request for a given amount of time. This is useful when the authorization call takes a considerable amount of time to execute and the authorization result is not expected to change within the cache time.

Caching the authorization result can significantly improve the response times of the authorizer but it comes with a trade-off. The cached authorization result is not updated until the cache expires. This means that the authorization result may not be in sync with the actual authorization state of the user.

This needs to be taken into account when choosing the right authorizer pattern.

Example

1. There’s a role based authorization system with 2 roles: admin and user
2. User A has an admin role and requests a resource in our AWS backend
3. The authorization result (“user A has admin role and is allowed to do admin things”) is cached for a given amount of time
4. In the meantime, user A’s role is changed to user
5. User A requests the same resource again where the authorization result is cached => The cached authorization result is used and user A is allowed to do admin things even though he shouldn’t be allowed to

Binary Authorizer Pattern

This pattern consists of an API Gateway, a single authorizer and integration Lambdas.

The authorizer is responsible for the authorization based on a binary decision (e.g. “does the request contain a valid token”). In the diagram above, an exemplary “External User Service” is queried by the authorizer to authorize the request.

Authorized requests are forwarded to backend Lambdas. The Lambdas assume that the requester is authorized to perform the call and can focus on the business logic.

When to use

• When separation of concerns between authorization and business logic is needed (code, test, deploy)

When to cache

• When the authorization call takes a considerable amount of time to execute
• When the security requirements allow the use of caching to improve response times of Authorizer and Lambda invocations

Context Authorizer Pattern

This pattern consists of an API Gateway, a single authorizer and integration Lambdas.

The authorizer is responsible for the authorization based on a binary decision (e.g. “does the request contain a valid token”) and for providing the authorization information to the backend Lambdas.

The Lambdas assume that the requester is authorized to perform the call and receive additional context information to make more nuanced authorization decisions.

In the diagram above, an exemplary “External User Service” is queried by the authorizer to authorize the request and get the role of the user. The role is then forwarded to the backend Lambdas as context information.

When to use

• When the authorization conditions are complex and require evaluation on a per-Lambda basis
• When it’s possible to re-use the same authorization evaluation code across Lambdas in a maintainable way (same code repository or shared library)

When to cache

• When the authorization call takes a considerable amount of time to execute
• When the security requirements allow the use of caching to improve response times of Authorizer and Lambda invocations

Choosing the right authorizer pattern

Here are some questions to ask yourself when choosing the right authorizer pattern.

Please note that these are based on the identity source being a authorization token. This does not take into account other identity sources like request parameters where caching behaves differently (I hope to cover this in a future article).

1. Does my use-case need strict security requirements?

   • Yes:
     • No-Authorizer Pattern
     • Non-Cacheable Binary Authorizer Pattern
     • Non-Cacheable Context Authorizer Pattern

2. Do my authorizer calls take a considerable amount of time to execute?

   • Yes:
     • Cacheable Binary Authorizer Pattern
     • Cacheable Context Authorizer Pattern

3. Do all my Lambdas require the same authorization information?

   • Yes:
     • Binary Authorizer Pattern

4. Do my Lambdas require different authorization information?

   • Yes:
     • Context Authorizer Pattern

Conclusion

We have looked at different patterns for API Gateway Authorizers and learned when to use which pattern.

There is no one-size-fits-all solution. It’s important to understand the trade-offs of each pattern and choose the right one for your use-case.

Further reading and mentions

• AWS Lambda Authorizer Patterns and Caching
• API Gateway Authorizer and Logout (Performance/Security Considerations)

You have a web app that sends out emails using a service like SendGrid or similar when a user signs up. It’s actually not important how the mail delivery is implemented. You want to test that an email is delivered at all using an automated test runner like Cypress.

This is what the sign-up implementation could look like.

Note that we don’t want to reconfigure the app under test to use a different mail API or SDK (like MailTrap) to reroute our whole mailing traffic via a different service. We don’t do that because that essentially prevents us from truly testing the app end-to-end.

The solution

Let’s first have a look at our high level test spec.

1. Given a new email address and password
2. When I sign up to my app
3. Then I want to receive an email

In Step 2 we will need to provide a valid email for our app. In step 3 we will need to check that an email arrived so we need some sort of mailbox. Let’s define some requirements for our mailbox.

1. The mailbox should give us unique addresses to use per test.
2. The mailbox should require minimal manual configuration.
3. The mailbox should be accessible via an API.
4. The mailbox should be cheap af.

1secmail.com to the rescue!

The web version looks quite hideous but it has a really simple API.

It basically functions as a catch-all email mailbox. So you don’t need to create temp addresses. You can use any temp address in your tests (e.g. during sign-up). You can then use the temp address as key to fetch email for the temp address.

So our automated test looks like this under the hood.

• Generate a temp email address like test-user-${unique-suffix}@1secmail.com (no API call necessary)
• Use the generated temp email address in the test to sign up to the app under test
• Check for mails using the 1secmail API

Here’s an example axios implementation of the API call:

const suffix = faker.random.numeric(8)

type Message = {
  id: string
  from: string
  subject: string
  date: string
}

async getMessages(
  login: string,
  domain: string
): Promise<Message[]> {
  const response = await axios.get<Message[]>(
    'https://www.1secmail.com/api/v1/',
    {
      params: {
        login: `test-user-${suffix}`,
        domain: '1secmail.com',
      },
    }
  )
  return response?.data
}

Let’s double check if this approach fulfils our defined requirements.

• We get unique temporary addresses per test.
• The mailbox is catch-all so no configuration is necessary.
• There’s a simple API.
• It’s free.

Note: the public nature of this (anyone can read any temp mailbox) makes this unsuitable for tests where some sort of sensitive data is exchanged.

Why not aliases?

You could create an email address at a public email provider like Protonmail.

You could use unique aliases for each test like test-user+timestamp@protonmail.com.

I know they do have some sort of public API for you to use in your automated tests.

You can get probably get away with this for quite some time. My previous setup involved a prepared email address with aliases at Protonmail. Unfortunately they terminated my account due to a violation of their TOS - apparently you’re not allowed to use automated software to spam their system. Who would have thought.

Why not MailTrap?

MailTrap looks promising because it’s supposed to be an email testing service but it’s not suited for real E2E tests on production if you’re not willing to pay premium cash. You don’t get any public email address up until the Business plan which costs 50$ / month!

In other words, the lower tier plans only allow you to include MailTrap’s SDK into your app or use SMTP to send mail directly to MailTrap. This is not E2E testing because you’re modifying the app’s configuration.

Basically it’s useless and I was disappointed enough to write them a mail when I found out about this…

If you’re not familiar with what Flux is and how it helps you build GitOps workflows on Kubernetes, feel free to read part 1 here: “Introduction to GitOps on Kubernetes with Flux v2”.

In today’s guide we will look at Mozilla SOPS and learn how to incorporate it with Flux v2 to store encrypted secrets in our GitOps repositories and have Flux decrypt them automatically during deployments.

To follow along, you will need access to a Flux-enabled Kubernetes cluster.

Introduction to SOPS

If SOPS is old news to you, you can skip ahead to “Enabling SOPS in Flux v2” ⏩

Suppose you want to store a database password in a Kubernetes yaml and check that in to source control.

You could base64-encode the password and upload it to a private git repository that only your team has access to… If your repository is compromised, the attacker could easily read the password without any further measures in place to protect it. That’s where SOPS comes in.

SOPS stands for “Secrets OPerationS”. SOPS allows us to encrypt and decrypt certain parts of our Kubernetes yamls with PGP. This means that we can even make our yaml declarations containing secret data public and not worry about unauthorized parties peeking into the contents of the secret values because they would require the respective PGP private keys to decrypt the data.

ℹ️ Note that SOPS can also handle different file formats (JSON, ENV, INI, etc). In the Kubernetes context, we mainly use it with yaml files as that’s what we work with most of the time.

SOPS basics

In this section we’ll learn how to use SOPS on your local machine.

First you’ll have to create a PGP key with OpenGPG. The following command will guide you through the creation process.

gpg --full-generate-key

⚠️ Make sure not to specify a passphrase in the prompt if you’re going to use this key with Flux. You can use the default selections for most other parameters and just supply your ID information (name + email).

Verify that the PGP key pair was created and note the secret ID:

gpg --list-secret-keys ${your_email_address}
out:sec   rsa4096 2021-02-04 [SC]
out:CFF53C2B937EAFD676F75C48F70573E9355BF63B
out:uid           [ultimate] Leonid Koftun <leonid.koftun@gmail.com>
out:ssb   rsa4096 2021-02-04 [E]

Let’s create a simple Kubernetes secret yaml to demonstrate how to use SOPS with the new GPG key.

cat <<EOF > secret.yaml
out:apiVersion: v1
out:kind: Secret
out:metadata:
out:    name: my-database-secret
out:    namespace: awesome-namespace
out:stringData:
out:    database-password: Password123
out:EOF

Now we’ll create a SOPS config file in our working directory.

cat <<EOF > .sops.yaml
out:---
out:creation_rules:
out:- encrypted_regex: '^(data|stringData)$'
out:  pgp: >-
out:    CFF53C2B937EAFD676F75C48F70573E9355BF63B
out:EOF

This configuration tells SOPS to only encrypt values under the ‘data’ and ‘stringData’ keys inside of yaml files and to use our previously generated PGP key for the actual encryption.

We can now run sops:

# You can encrypt the file in-place
sops --encrypt --in-place secret.yaml
# Or write to a new file
sops --encrypt secret.yaml > encrypted-secret.yaml

The result of the encrypted yaml will look like this:

apiVersion: v1
kind: Secret
metadata:
    name: my-database-secret
    namespace: awesome-namespace
stringData:
    database-password: ENC[AES256_GCM,data:k8GGkwr4AE/CdlM=,iv:tecWFmg0INNY1vRfpdGLsDc+APd6UmKk6AS//U0OjI4=,tag:EEm7DHO7muNgpLOwUZh1Lw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    lastmodified: '2021-02-22T21:45:13Z'
    mac: ENC[AES256_GCM,data:jhzW3o+XcFZgkvGzMb05GpM3hu1dmhRE74woFIeYQOtOy1jCXCp9WgyHar3XDp+1TkOOaN0myfRMe6uz/WmyyKXMPZNf5i4MlB053UUeL2RFMjaGjFlEgq7kG+aoke7+JVN3vTLCiP9fMb4aV3wfPy3hMp5d10wSmPhcfG6/Fww=,iv:07ToHHvJL5tzI5RZLEkfFj+tqJ1y/3XOlADB9TAIuS0=,tag:xlZWGsK5Isrr6GEpBU7YyA==,type:str]
    pgp:
    -   created_at: '2021-02-22T21:45:13Z'
        enc: |
            -----BEGIN PGP MESSAGE-----

            hQIMAzkIo7JC/ReAAQ/+KV60yKpfRK/qiVaRLbHwu6iTNy59O23vS4+Qt5tv8B9n
            xW4IxTt4IiXeqZMDt2byJUh9KtncnNKectM5gdxDsFdh4QeChFgVYZsgl0CWG3bY
            JBq6Tc/X4udagIuKYIqE+kXiiSwH3+YlKO5VbvcKJPbKg+daeWQvQvXgfghzRZpL
            6auVpdw2E8K59gtADi5T+0JvCy8WXYRWu9JOP9TSQjMx1LRLXmiVITbDQFRokZmb
            7sGbweCVw6ixEDVDbNZx6rh/sH6MGV+yVVd/KMAUWWkKfvezZgbU4M4i7Bfp4sKT
            fp9I21UL1SERHV+6h+jsU379OYb6QdUQ3s+UU7PqejWzuojG4SaHxwPUCT05Doo7
            Bf7syjGta6BJcDgMW8CTfqO/58wpBCI5m1xqri9KYF1/E3T7qP4P1vYO4XLXKJK+
            Q6ee2pgb3W+y8qI0ev2QYh+lFjqH0mk7Z2L5E3JK/Jwsl+mG3jQBPo4fw/N4Iq1j
            WIel4uL9PgPPAgt13Z6UaaYjmYkfiaJIK0rVaY9ArQwU+ShkqHC9nFKB4wxRrGRA
            jJU/6pDLCGOfiHOrGC873qGpOnhnpEZlwxQEQClzFo+uug0bL9fMmD+UBgxaoT1C
            rrXs0tVkgVlyeYDgIpjuwsZEflfxjy/vuH49VeZETn8iQ+4dpqAXFcQyoAAFNWzS
            XgF3Cl2zDs/SIoRMwvZw+GlaWAX9yBLGjxjJGyA5oXkx03i1sL9+5B154O/iPm5q
            QwRTqwmZ22XmtMycV4cJP4tMC/kPKvCHyv3JUO28FkXfhovh+VHCpghzAFySKxc=
            =Pf3r
            -----END PGP MESSAGE-----            
        fp: CFF53C2B937EAFD676F75C48F70573E9355BF63B
    encrypted_regex: ^(data|stringData)$
    version: 3.6.1

Note that the stringData.database-password is no longer readable and the added sops block. The latter contains metadata for SOPS to allow decrypting the secret back to plain-text with the correct private key.

The encrypted yaml file could now be checked into git and distributed. If we try to kubectl apply the encrypted yaml directly to our cluster, it will fail because the secret is not a valid Kubernetes secret in this form.

We have to decrypt it before deploying it like so:

sops --decrypt encrypted-secret.yaml | kubectl apply -f -

The added security of encryption adds at least two extra steps to our DevOps workflow:

1. We need to make sure we only add encrypted secrets to source control.
2. We need to decrypt secrets in our CI/CD scripts before we can deploy to a cluster.

The latter can be automated nicely with Flux v2. Let’s see how it’s done.

Enabling SOPS in Flux v2

Now that we know how SOPS works offline, we will be able to apply the same principe to our GitOps repositories with Flux.

Flux supports SOPS out of the box, we just need to supply it with correct PGP private keys and its controllers will decrypt SOPS-protected yamls during reconciliation.

Create a Kubernetes secret with your private PGP key

We want to export our PGP private key and store that in a Kubernetes secret that Flux can use on the cluster.

We can do that with the following commands:

# Find the ID of the private key.
gpg --list-secret-keys ${your_email_address}
out:sec   rsa4096 2021-02-04 [SC]
out:CFF53C2B937EAFD676F75C48F70573E9355BF63B
out:uid           [ultimate] Leonid Koftun <leonid.koftun@gmail.com>
out:ssb   rsa4096 2021-02-04 [E]

# Export the PGP secret to a new k8s secret
# in the flux-system namespace.
gpg --export-secret-keys \
out:  --armor CFF53C2B937EAFD676F75C48F70573E9355BF63B |
out:kubectl create secret generic sops-gpg \
out:  --namespace=flux-system \
out:  --from-file=sops.asc=/dev/stdin

🐔 🥚 Note that this manual step will likely need to be a part of your Flux bootstrapping routine. This secret can not be installed by Flux itself because of the implied chicken-egg problem.

Setup cluster-side decryption in Flux

The following examples are based on my recent post about GitOps with Flux v2 where we have bootstrapped the flux-system namespace and deployed a new namespace from our GitOps repository.

Our GitOps repo looks something like this. You can browse it on Github.

.
├── cluster
│   ├── awesome-namespace
│   │   └── namespace.yaml
│   └── flux-system
│       ├── gotk-components.yaml
│       ├── gotk-sync.yaml
│       └── kustomization.yaml
└── README.md

To enable SOPS for our GitOps repository we need to edit the gotk-sync.yaml resource:

---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: flux-system
  url: ssh://git@github.com/sladkoff/home-cluster
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client
  # Enable decryption
  decryption:
    # Use the sops provider
    provider: sops
    secretRef:
      # Reference the new 'sops-gpg' secret
      name: sops-gpg

Nice. All that is left to do is test this by checking-in an encrypted secret like the one we created earlier and push our changes to the remote git repository.

➡️ Example commit on Github.

Once Flux reconciliation is done, you should see the new my-database-secret inside the awesome-namespace.

Summary

We can use SOPS to encrypt and decrypt Kubernetes secrets. We learnt how to do this manually on our local machine and automatically on a k8s cluster with Flux v2.

If any of this helped you in any way, feel free to buy me a coffee ☕

You can also give me feedback in the comments, on Twitter or Instagram.

Thanks 🙏

As I’m currently learning about Ansible, I decided to write a playbook that sets up a Headless Raspberry Pi in Grafana Kiosk mode.

This quick guide shows you how to create a Kiosk of your own.

You’ll need:

• A Raspberry Pi with an empty SD card
• Ansible on your local machine
• A display or TV to connect to your Raspberry
• A Grafana instance that is reachable by the Raspberry

Installing the OS

I used the GUI rpi-imager to bootstrap my SD card with Raspberry Pi OS Lite (32bit) for my Raspberry. I think it’s quick and easy and offers you some distro options to choose from.

You can get the tool from the AUR if you’re using Arch:

yay -S rpi-imager

Here’s a screenshot of the GUI (select an image, select SD card, write!):

To make the raspberry auto-connect to your WiFi on boot, create a new file wpa_supplicant.conf inside the boot directory of your flashed SD card with the following contents.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=<Insert 2 letter ISO 3166-1 country code here>

network={
 ssid="<Name of your wireless LAN>"
 psk="<Password for your wireless LAN>"
}

To enable remote access over ssh after boot, create an empty file called ssh inside the boot directory as well.

Installing grafana-kiosk

grafana-kiosk is a simple wrapper script that starts a fullscreen Chrome session and opens a configured Grafana URL with optional authentication. This Grafana URL usually points to a Grafana Playlist which switches between different Grafana dashboards.

I wrote a simple Ansible role to set up Headless Raspberry Pi Grafana Kiosks ™️.

Ansible Playbook

Here’s my personal Ansible playbook.

It assumes that you have a fresh Raspberry with no graphical environment. Like what we’ve set up during “Installing the OS”.

The playbook will install and configure the following things to autostart on boot:

• openbox
• chromium
• grafana-kiosk

To get started on your own RPI, you can fork my repo and apply at least the following changes:

roles/ssh-keys/tasks/main.yml

This role contains tasks to disable password login via ssh and upload ssh keys to the RPI. You can either get rid of this if you don’t need it or provide your own ssh keys:

# roles/ssh-keys/tasks/main.yml
---
- name: Deploy SSH Key
  ansible.posix.authorized_key:
    user: pi
    state: present
    key: <URL-OR-FILE-TO-YOUR-SSH-KEYS>
    validate_certs: False
  notify:
    - Restart sshd

- name: Disable Password Authentication
  lineinfile:
    dest: /etc/ssh/sshd_config
    regexp: '^PasswordAuthentication'
    line: "PasswordAuthentication no"
    state: present
    backup: yes
  notify:
    - Restart sshd

hosts.yaml

Provide the IP of your RPI in the hosts.yaml:

# hosts.yaml
grafana_kiosk:
  hosts:
    grafana_kiosk_rpi_tv:
      ansible_host: "<YOUR-RASPBERRY-IP-HERE>"

all:
  children:
    grafana_kiosk:

files/my-grafana-kiosk-config.yaml

Provide a configuration for the grafana-kiosk utility script, this is where you specify the path to the Grafana instance and Playlist ID that you want to display on your kiosk:

# my-grafana-kiosk-config.yaml
general:
  kiosk-mode: full
  autofit: true
  lxde: false

target:
  login-method: local
  username: pi-grafana-kiosk
  password: pi-grafana-kiosk-password
  playlist: true
  URL: https://<YOUR-GRAFANA-HOST-HERE>/playlists/play/<YOUR-GRAFANA-PLAYLIST-ID>?kiosk&autofitpanels
  ignore-certificate-errors: false

⏯ Run the playbook in the repo directory

ansible-galaxy install -r requirements.yaml
ansible-playbook install-grafana-kiosk -i hosts.yaml --ask-pass

You’ll need to connecto the RPI to a screen and reboot it. If all went well, you should be greeted with your Grafana Playlist after booting.

If there are still issues with my Ansible voodoo, feel free to create a PR on Github 😸

We will first go through some core concepts of Flux and then create our first GitOps workflow.

You will need access to a Kubernetes cluster, a shell interface and a Github account to follow this guide. Note that you can use any git provider (Gitlab, Bitbucket, custom) but you’ll have to modify the provided example commands.

What is GitOps?

GitOps is a way of managing your infrastructure and applications so that whole system is described declaratively and version controlled (most likely in a Git repository), and having an automated process that ensures that the deployed environment matches the state specified in a repository.

– https://toolkit.fluxcd.io/core-concepts/#gitops

In Kubernetes practice this means that git is used over kubectl (helm, etc) to perform operations tasks against the cluster.

Pushing to master triggers a deployment to the cluster. We can work with branches and Merge Requests to diff and review changes to the desired cluster state. We can audit past cluster states with git log. We can rollback a change using git revert.

What is Flux v2?

Flux v2 is a Toolkit for building GitOps workflows on Kubernetes.

In simplified terms, Flux v2 is deployed to a Kubernetes cluster and configured to watch git repositories containing Kubernetes manifests.

As Flux watches all our repos and pulls the changes into the cluster, we can focus on writing our Kubernetes manifests. We don’t have to worry about client-side tooling and pushing the changes from every git repo into the cluster. This is what GitOps is about.

We will come back to the individual Flux components once we’ve installed them in the upcoming steps.

Installing Flux v2

We’ll follow the official docs and use the flux CLI tool.

You can install it with:

curl -s https://toolkit.fluxcd.io/install.sh | sudo bash

# enable completions in ~/.bash_profile
. <(flux completion bash)

The flux bootstrap command will ensure that:

1. A new GitOps repository for our manifests is created on Github
2. A flux-system namespace with all Flux components is configured on our cluster
3. The Flux controllers are set up to sync with our new git repository

ℹ️ Note that there’s also Terraform Provider as an alternative installation method.

The following snippet shows an annotated bootstrap command. Feel free to play around with flux bootstrap --help to get to know all available options.

export GITHUB_TOKEN=<your-gitlab-token-here> # 1

flux bootstrap \
  github \                      # 2
  --owner <your-github-user> \  # 3
  --repository cluster-name \   # 4
  --personal \                  # 5
  --private \                   # 6
  --path cluster \              # 7
  --branch master               # 8

1. We’re going to need to give flux access to our Github account so that it can manage our GitOps repository on our behalf. You can generate a new Personal Access Token in Github Settings.
2. We need to tell Flux that we’re using Github as git provider. You can also use Gitlab and generic git servers.
3. The Github username.
4. The name of our git repository.
5. (Optional) We’re creating a personal Github repo.
6. (Optional) We’re starting out with a private Github repo.
7. (Optional) A path inside the repository to be watched by Flux.
8. (Optional) The git branch to be watched by Flux.

The bootstrap command is interactive and will let you know about the progress and state of the installation.

This is all you need to get up and running with Flux. Note that the boostrap command is idempotent and can also be used with an existing GitOps repository.

Inspecting the Flux components 🕵

The flux-system namespace

Let’s take a look at what was deployed to the k8s cluster for us.

# Let's have a look at the Flux controllers...
kubectl get pods -n flux-system
out:NAME                                      READY   STATUS    RESTARTS   AGE
out:helm-controller-c858b9c65-v8pgr           1/1     Running   28         17d
out:kustomize-controller-6767b8fd78-8ptd4     1/1     Running   30         17d
out:notification-controller-db467b4fb-pt9tc   1/1     Running   29         17d
out:source-controller-7b4d748b45-2blvd        1/1     Running   28         17d
# And the custom resource definitions...
kubectl get crd -o name | grep flux
out:customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io
out:customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io

Great! The Flux controllers are running and we have a bunch of new CustomResourceDefinitions available to us. Read ahead to learn more about how those play together.

The GitOps repository

To better understand how Flux works, let’s now take a look at the directory structure of our bootstrapped GitOps repository on Github. We’re going to see that the bootstrap command has created some Kubernetes yamls for us.

git clone git@github.com:${username}/cluster-name.git
cd cluster-name
tree
out:.
out:├── cluster
out:│   └── flux-system
out:│       ├── gotk-components.yaml
out:│       ├── gotk-sync.yaml
out:│       └── kustomization.yaml
out:└── README.md
out:
out:2 directories, 4 files

Cool. We’ll go through the individual files.

The cluster directory

This directory was created because we used flux bootstrap ... --path cluster ... during bootstrapping. We’ll see in a second that this folder is also special because it’s being watched by the Flux kustomization-controller.

Any Kubernetes yaml that we throw in this directory will be deployed to our cluster. It also means that we can have different files and directories in our repository that will never make it to the cluster. This is useful because we could have one single git repository for our infrastructure declarations and divide it into sub-paths for different k8s clusters or environments. For example, we could bootstrap multiple Flux environments and use paths like dev-cluster, stage-cluster, prod-cluster.

That’s just one way to organize. It’s nice that we have this flexibility.

The flux-system directory

This directory represents the flux-system k8s namespace and contains Kubernetes declarations.

I like to continue this pattern and create a subdirectory for each namespace that I deploy with Flux.

Example:

└── cluster
    ├── flux-system
    ├── monitoring
    ├── cool-app-namespace
    └── monitoring

This isn’t required. You’re flexible and can structure the /cluster directory like you want!

kustomization.yaml

As we can see Flux created a simple kustomization.yaml for us. If you’re not familiar with Kustomize you can just ignore this for now. All it does is include the other two files in the cluster/flux-system directory.

# cluster/flux-system/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- gotk-components.yaml
- gotk-sync.yaml

gotk-components.yaml

This file contains the GitOps ToolKit components 🤯 - in here you will find the Deployments and Services for the Flux controllers and all the CustomResourceDefinitions that we’ve already seen when we inspected the flux-system namespace.

This file is generated by the flux bootstrap command and you shouldn’t have to deal with it manually.

gotk-sync.yaml

The file gotk-sync.yaml is more interesting to us. This file declares our first two custom resources for Flux.

# cluster/flux-system/gotk-sync.yaml
---
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: master
  secretRef:
    name: flux-system
  url: ssh://git@github.com/your-user-name-here/cluster-name
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: flux-system
  namespace: flux-system
spec:
  interval: 10m0s
  path: ./cluster
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  validation: client

Let’s take a closer look at these two…

The flux-system GitRepository

The first custom resource is of type GitRepository. A GitRepository is one of the possible Sources that are picked up by the Flux source-controller. Once a repository is registered with this controller, the Flux toolkit is able to watch it for changes and notify other Flux controllers to apply the contents to the cluster.

This specific GitRepository definition tells Flux to check the master branch of our new GitOps repository from Github. every 1m0s using a flux-system secret for authentication (this secret contains your Github Personal Access Token that we created during bootstrapping).

Note that we could create more GitRepository resources and effectively sync any number of GitOps repositories with our Flux-enabled cluster.

Example use-case: You have multiple teams that want to deploy to one cluster. You want each team to maintain their own infrastructure repository. You could set up Flux on the cluster such that it pulls each team’s repository.

Read more in the Flux Docs.

The flux-system Kustomization

The second custom resource in this file is a (Flux-) Kustomization.

⚠️ Not to be confused with a Kustomize resource.

A Flux Kustomization describes a path inside of a Source that Flux should apply to the cluster. It’s assumed that there are Kubernetes yamls and/or Kustomize yamls located in the directory at the named path.

The Flux kustomize-controller is going to periodically apply the yamls at the given location to our cluster.

Here’s a simplified walk-through of how the kustomize-controller is going to install our k8s resources.

• Checkout the configured branch from

  ssh://git@github.com/${username}/cluster-name
  

• Change the working directory to the specified path: ./cluster
• Build a root Kustomize yaml at this location if no explicit one is available:

  kustomize create --autodetect --recursive > kustomization.yaml
  

• Apply the result:

  kubectl apply -k kustomization.yaml`
  

I think it’s good to know how kustomize is being used to build the final Kubernetes manifest from our GitOps repo. The commands above can be used for testing locally. This is especially helpful when you mix plain Kubernetes yamls with Kustomize yamls as this has an impact on the recursive auto-detection.

Deploying own resources

Let’s do a really simple example on how to deploy stuff to our cluster.

All we want to do is create a new namespace “awesome-namespace”.

# Create a new directory for our namespace
mkdir cluster/awesome-namespace

# Create the Kubernetes namespace descriptor
cat <<EOF > cluster/awesome-namespace/namespace.yaml
out:---
out:apiVersion: v1
out:kind: Namespace
out:metadata:
out:  name: awesome-namespace
out:EOF

# Commit and push the change
git add .
git commit -m "Add 'awesome-namespace'"
git push

Lean back and watch as the new namespace is shipped to our cluster 🛳️

Summary

We installed the Flux GitOps toolkit to a Kubernetes cluster and deployed a Kubernetes manifest by only using git. Along the way we learnt about the Flux components.

I hope this gave you an introduction on what Flux v2 is, how it works and how you can use it to enhance your CI/CD workflows.

I’m planning to write more about this topic in the future. We’ll take a look at how to integrate SOPS for secret handling, how to deploy Helm Charts and how to set up Flux notifications.

If you read this far, hit me up in the comments, on Twitter or Instagram to let me know how I did.

Thanks 🙏

After stumbling upon this Techlead video on YouTube I was convinced that I didn’t need any fancy layouts for my CV anyway and that I would go with a minimalist plain-text approach.

If you’re like me and you don’t need pie charts on your resume, you might be asking:

Why not write my resume in good old Markdown? Why not just make it open-source on Github as well? Why not use Github Actions to build a distributable PDF version of it? 🤔

Hey, that’s what I did! And I find it pretty cool. So I recommend you do it, too. Here’s why.

The Opensource Markdown Resume ™️

Easy to write and maintain!

It’s one file. It’s simple af. I only used headings and bullet lists and decided that it was enough to get my points across.

You can see it on Github and fork it or whatever.

Automated PDF export!

I found this markdown-to-pdf Github Action that exports - you guessed it - Markdown to PDF. It renders the PDF with the default Github styling, so the result (preview available here) looks very similar to the web version.

Here’s the full annotated Github Action that I’m using now:

on:
  push:
    tags:
      - '*'                                                 # (1)

name: Upload Release Asset

jobs:
  build:
    name: Upload Release Asset
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Remove swag line                              # (2)
        run: sed -i '1d' README.md
      - name: Build PDF from Markdown                       # (3)
        uses: BaileyJM02/markdown-to-pdf@v1
        with:
          input_dir: .
          output_dir: out
      - name: Rename pdf
        run: cp out/README.pdf ./leonid_koftun_resume.pdf
      - name: Create Release                                # (4)
        id: create_release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          tag_name: ${{ github.ref }}
          release_name: Release ${{ github.ref }}
          draft: false
          prerelease: false
      - name: Upload Release Asset                          # (5)
        id: upload-release-asset
        uses: actions/upload-release-asset@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          upload_url: ${{ steps.create_release.outputs.upload_url }}
          asset_path: ./leonid_koftun_resume.pdf
          asset_name: leonid_koftun_resume.pdf
          asset_content_type: application/pdf

Annotations

1. Build PDF when any tag is pushed
2. (Optional) This is just ‘pre-processing’ to remove some HTML content that I don’t want to be rendered to PDF (the first line of the Markdown contains some links that only make sense on the Github web view)
3. The actual markdown-to-pdf step with minimal config
4. Creates a Github release for the tag
5. Uploads the generated PDF to the tagged release

The opensource aspect

As you probably noticed by now, my resume is publicly available on Github.

Potential employers and recruiters can grab a copy there. My peers and colleagues can review it and tell me what I suck at the most (come at me).

Any issues with this?

Yes.

• If you want to use fancy column layouts and colors and unicorns 🦄 in your resume, then this probably isn’t for you (you could make it complicated and add CSS and templating but that kind of defeats the purpose of simplicity in the .md approach IMO).
• If you’re afraid of me or one of my cats copying some lines from your own personal resume, I do not recommend that you make it public.

Conclusion

Keep your resume simple by using Markdown, publish it on Github if you want to be a cool kid.

What do you think of this nonsense? #hmu on Twitter or Instagram. 🤙

This means that I have some source files on my local machine and I run hugo to generate static html/css files. Once I have these static files I needed to decide how to publish them to the internet.

This post is a subjective comparison of possible hosting solutions when working with Hugo.

Gitlab Pages

My initial plan was to use Gitlab Pages because my source files are already checked into a private git repository on Gitlab.

I used a simple .gitlab-ci.yaml found on https://gitlab.com/pages/hugo:

image: registry.gitlab.com/pages/hugo:latest

variables:
  GIT_SUBMODULE_STRATEGY: recursive

pages:
  script:
  - hugo
  artifacts:
    paths:
    - public
  only:
  - master

This would run on my master branch and publish the generated content to Gitlab Pages. The setup is simple and everything was in one place, a custom domain can be used and we get a Let’s Encrypt TLS certificate as well.

What made me look for an alternative solution? Two things:

• For some reason I could not get custom 404 pages to work with this setup. When accessing a non-existent URL it would always load a generic Gitlab 404 page instead of my custom Hugo one. It may be related to this issue.
• The loading times seemed atrociously slow for what I was serving 🐌. 5-10 seconds on an empty cache was quite a deal-breaker.

Github Pages

Everybody and their mom has a Github account. Github also has Pages. So I tried out hosting my hugo content there.

The setup is not so elegant anymore. I followed the instructions on https://gohugo.io/hosting-and-deployment/hosting-on-github/ and ended up with two git repos. One for my source files (I used my existing Gitlab repo) and one for the generated content under /public which needs to be set up as a git submodule.

This could still be automated with Gitlab CI pretty easily with something like (not tested):

image: registry.gitlab.com/pages/hugo:latest

variables:
  GIT_SUBMODULE_STRATEGY: recursive

github-pages:
  script:
  - hugo
  - cd public && git -am "Build for Github Pages" && git push
  artifacts:
    paths:
    - public
  only:
  - master

This tells Gitlab to build the static page and then push the git submodule to Github as well.

Github Pages were subjectively faster but there are downsides:

• I still didn’t get a custom 404 page 😥
• I don’t like having one repo for the source and one for the generated content
• Github Pages works only with public repositories (unless you have a paid account AFAIK)

Netlify

I haven’t heard of Netlify before. I think it came up in some Gitlab issue while I was looking for alternatives.

I used their web UI to set everything up but I read that there’s also a CLI available which should be cool. Anyway, I just had to connect my Gitlab account and select the source repository which I had already been working on.

Netlify automagically recognized that the contents of the repo had to be build with hugo and suggested that for the build options:

Whenever I push to master in Gitlab a hook on Netlify builds and publishes the site. I could get rid of the .gitlab-ci.yml altogether.

I like about this setup:

• I have one private git repo on Gitlab
• I haven’t had issues with loading times on Netlify so far

I understand that there could be one big downside to this approach for other people. On Gitlab / Github you get a cool xyz.gitlab.io / xyz.github.io domain for free. A free xyz.netfliy.com domain is not so cool for a tech blog - so I’d say that you need a custom domain if you choose to go that route.

Conclusion

If you’re using hugo to generate your static site, you can use Gitlab Pages, Github Pages and Netlify as a hosting provider quite easily.

Gitlab turned out to have slow loading times. Github requires a weird repo / branching setup. Netlify works nicely together with Gitlab but you’ll probably want a custom domain.