Gitops Implementation Continued

It takes a long time to make no visible progress.

Well that’s only partially true. Certainly no visible progress in that the same result is happening now that was before, but behind the scenes a few things have moved forwards. As of now, the end-to-end process from pushing code from my notebook to GitHub, building the container/s for the relevant services, updating the GitOps Kubernetes manifest, pulling the new container image and getting it running and available to the world is fully automated, no need to me to manually update repository links or log into the server to get things going.

Adding in automatic image deployment

Previously, once a new piece of code was sent to GitHub, I needed to manually update my repo to see the new image and deploy it. The process of actually building the image was simple and already covered within GitHub Actions workflows within the CalCount repo, split out such that each service can build its own container only when the relevant code is upgraded. The next part, linking the newly built artifact to whatever is being monitored by Flux was the challenge for this weekend.

Adding image monitoring

Out of the box, Flux doesn’t include modules for checking image repositories, so step one was to recreate my gotk-components manifest to add in the additional components. This was simple enough, just re-run the Flux install command and pipe the output to my gotk-components file. From there the next step was to get the actual Flux components configured to install everything on update.

Adding Flux components

There were three components to add to get the image updating recognised. First was the ImageRepository resource, which is defined per-container that is being deployed, not per repository as I first assumed. So for my five services in CalCount I needed to define five ImageRepository instances, one for each service. Same again for the ImagePolicies, which define what specific image gets pulled from the ImageRepository resource. Because the concept of “most recent” doesn’t seem to exist in these environments, the most effective way I found to get the SHA hash of the most recent image was to have the latest image built in GitHub to be tagged with latest, as is convention, then have the following ImagePolicy configuration

  policy:
    alphabetical:
      order: asc
  filterTags:
    pattern: "^latest$"

Initially Gemini had suggested doing a purely alphabetical ordering, but that showed to be immediately problematic when an older version of my application OTel collector ran into a configuration issue due to not having a ClickStack URI to send data to and failing to start. Luckily with such a clear failure process it was an immediate piece of feedback that let me add the filter tag in. I could probably get away with just the filter tag and omit the ordering, but for now I’m happy to keep it around just in case.

Updating kustomizations

The need to reference the SHA hash instead of just the ’latest" tag is that unless there is a change to the manifest, Flux won’t identify that the manifest in changed and reconcile the differences. So instead it’s necessary to run the process to find the SHA hash of the container tagged “latest”, append that SHA to the kustomization for the container reference in the kustomization pipeline, then use that to reconcile the old and new instances that need to be changed. Now that I had updated the Flux components, I had to remove the reference to the ’latest’ tag in my app definitions and add in references for Flux to update these instead. I ended up with

images:
  - name: ghcr.io/s-k-hassan/calcount/api
    newTag: latest # {"$imagepolicy": "flux-system:calcount-api-policy:tag"}

Where the tag ’latest’ is just a placeholder value to get past YAML linting and validation of the kustomization file, and as Flux passes over the kustomization during the application process, it replaces that value with the value on the right of the #. The placeholder tag could be literally anything, but given I’m using the “latest” tag, it seemed like a sensible idea to have it set as a failsafe in the event Flux failed.

Adding the image update automations

Lastly I needed an ImageUpdateAutomation. Because Flux will reconcile itself against the actual GitOps manifest, not the customisations it makes, it’s necessary that once it identifies the SHA hash and applies it, it can then commit the value back into the GitOps pipeline and reconcile itself against it. Which seems pointlessly recursive at first glance but made sense once I had thought it through - given my GitOps pipeline removes anything that is not defined in the repo, if it didn’t undertake this step it would provision and then destroy the container during its next reconciliation step.

All this automation does is allow Flux to create a new commit with the updated values and push them to the source repository with a default message.

Conceptually, super quick and easy to get done. In reality, this took several attempts to get right due to a combination of factors, including some DNS issues which might have been a result of the next section or a patch implementation issue, as well as the complexity of troubleshooting when multiple pull requests are made in a short time, and not all of them are pulled by Flux when it runs a reconciliation - if in doubt, git pull again and re-run the reconciliation command. Especially when you have Flux now also adding commits or if you are simultaneously doing other incidental housekeeping in the repo, you can have a new request come in while a request is being reconciled and not notice it unless you look at the SHA of what is being processed.

DNS-01 challenges via Acme

Tangetially related to the image deployment process, I want to eventually expand this GitOps configuration from my VPS to my wider homelab environment. Two key wants as part of this are to have it all deployable as a GitOps process, and to have automatic SSL certificate creation for any services that I spin up within that process. Usually getting an SSL certificate via Acme happens with a HTTP challenge, where the web server is issued a challenge to create a certain payload that is publically available and once Acme verifies it, it is assumed you own the website and a certificate is granted. For public sites this is great, for my homelab behind a CG-NAT and floating IP address this is somewhere between hard and impossible without adding in even more middlemen. So instead there is the DNS-01 challenge, where Acme verifies via a txt entry in your DNS registry to prove ownership of a domain and then issues a wildcard certificate, similar to AWS and Azure. Only issue with this is that the certificates are quite short-lived and are best automated via an API, something that my DNS provider doesn’t offer. So in comes the acme-dns server, which can be delegated nameserver authority from your main DNS provider using a NS record and has an API that can in turn automate the certificate generation process. It still needs a publicly accessible endpoint, which I can host on my VPS, but allows me to have full SSL certificates for all my home services.

The end-to-end workflow for this is still in progress, but the first part at least is done. The first part is getting the Acme-DNS container up and running. This was a fairly quick and easy setup building on the existing GitOps pattern. Step 1, define the application specification. Step 2, create the kustomizations and add in the certbot certificate, Traefik ingress route and define the config of the subdomain it will manage. Step 3, deploy. Of course nothing is ever that easy and while the container was up and running without issue, I ran into a problem where it seemed to override my existing DNS resolver config and meant my other services couldn’t resolve anything. Where this became a problem is that I was unable to pull any updates to my Flux configuration files or pull new images. After a lot of fluffing around I eventually updated my server to use a public DNS resolver by default instead of looking locally, then after a couple of restarts of the systemd service things seemed to start working.

While the homelab is still yet to be deployed, conceptually at least once it is I should be able to just extend the GitOps repo and get it to pull its config down and I’ll up and running. Reality of course will be a half-day of troubleshooting, but for now I can pretend it’ll all go smoothly.

OTel collector config update

Not specifically related to GitOps, but an important development regardless was the inclusion of log monitoring in my cluster-level OTel collector. Initially the collector was only grabbing traces and defining the endpoint to send them on to, but now Im also collecting my kubernetes logs with it. given I have a comparatively lightweight environment, it made more sense to me to hook into my kubernetes deployment at the per-host level and collect all the logs from all the applications instead of configuring a multi-step collection per app to send to the host collectors. This way I can have the applications send their traces to their app OTel agent and send all logs to stdout, which is then captured by the host-level agent along with the traces from the app-level collector and processed to send on to whatever backend it needs to go to. Next step will be configuring metric scraping and configuration of /health endpoints on each of the microservices to be able to collect instance data and process this at an app-level collector, send it to the host-level collector that will also grab the host-level metrics before passing all of it onwards to the chosen endpoint.

After several days of configuration, I’ve got YAML coming out my ears. Over 65 different configuration files spread across the GitOps repository and a current structure looking like:

├───apps
│   ├───acme-dns
│   ├───blog
│   ├───calCount
│   └───core-infra
│       ├───fluxcd
│       └───otel-collector
└───clusters
    ├───homelab-cluster
    │   ├───environments
    │   └───_core
    │       ├───coreDNS
    │       ├───fluxcd
    │       ├───init
    │       └───observability
    └───vps-server
        ├───environments
        │   ├───dev
        │   │   └───calcount
        │   └───prod
        │       └───acme-dns
        └───_core
            ├───fluxcd
            ├───init
            └───observability

A few more steps to go, but there’s definitely progress being made and plenty being learned alongthe way.