Sam's Space

Gitops Implementation

As part of building calCount, I wanted to get to the point of having the service available on the internet for wider access, ideally powered by Kubernetes and declarative infrastructure. As with many things related to this wider project, it was to build familiarity with a new technology stack and find how they work and how they break. Having finally completed the MVP of this process, I wanted to jot some notes down of learnings along the way.

Kubernetes up and running

Step one was installing a kubernetes distribution on my existing VPS. I have two VPS servers that I use, a small one hosting this blog and a second one that was running a demo environment on Minikube that ended up detonating itself due to OOM errors in one of the containers (I think, I never spent time investigating the issue at length). My experience with Minikube was… uninspiring, and while it is fine for a local testing instance or perhaps for someone who has more familiarity with bare-bones Kubernetes, I wanted to instead go for a more batteries-included approach for a publicly available app. To this end, I opted to try out k3s as a single-node cluster to help deal with some of the low-level details and remove the decision fatigue on what components to include as part of the deployment.

Starting with the end in mind

In the longer-term, I wanted to have my (still packed away) homelab and blog server running on their own Kubernetes clusters, and some of the applications within them replicated across dev/test/prod environments and across multiple clusters, such as OTel collectors. Given the concept of “Don’t Repeat Yourself”, and a want to implement GitOps in future for all of these deployments, the structure of the environment needed consideration. Some prompting via Gemini pointed me to a structure that made sense to me, and I ended up with the following structure:

├───apps
│   ├───blog
│   ├───calCount
│   └───core-infra
│       ├───fluxcd
│       └───otel-collector
└───clusters
    └───vps-server
        ├───environments
        │   └───dev
        │       └───calcount
        └───_core
            ├───fluxcd
            ├───init
            └───observability

The apps folder contains the core manifests for each of the apps I run, which for calCount included a Deployment and a Service for each container, plus a StatefulSet for the PostgreSQL databases and a ConfigMap for the various env variables. The manifests are then collected into a single kustomization file which is referenced by a deployment in clusters. The manifests defined here are the components less likely to need changing, such as which container to pull, the names of the services, what ports are open and so on. Anything that will be defined on implementation such as PostgreSQL passwords, DNS addresses, replica counts and Namespaces that the containers will deploy to are defined elsewhere. The logic here is that anything that changes that will need to update the core of the service, like a new env variable or a change in the components of the application (I want to add a “reports” container in future for example) can be defined once and then referenced by any implementations, instead of updating multiple copies of the same config. If needed, the config definitions can be overwritten at runtime, for example changing replicas from 1 to 2 for a production deployment.

Building the first node

Within the clusters directory, I created a subdirectory for my VPS which will be its own single-node cluster. Within that, I have my _core and environments subdirectories. _core contains the base infrastructure and cluster-wide settings, such as Namespaces used and the OTel collector gateway configured for the cluster. _core/init builds out the certificate manager and namespaces, then other services can build on top of those for anything they specifically require. _core/observability contains a kustomization file that pulls in the Secrets for the cluster, in this case the Datadog API key I will send data to, and references the /apps/otel-collector kustomization, which at runtime then collates all the referenced files together for the deployment.

With k3s installed and a basic implementation structure for my apps created, I moved to get calCount itself up and running. The app isn’t quite ready for primetime yet, so I opted to deploy it to a dev.calcount subdomain while I play around with it to get things right. Given k3s has a default ingress provider included, Traefik, it made implementation much easier as I only needed to define the Traefik IngressRoute, not the full CRD implementation on top of everything else. The definition is pretty simple, Traefik listens for traffic directed to dev.calcount, and passes it on to the calcount Nginx container. My understanding of the inheritance process is that because the IngressRoute is deployed via a kustomization that specifies a namespace (calcount-dev), the IngressRoute will route to the container within its own namespace, so I can also create a near-identical configuration for my production instance, changing the DNS entry to listen for, place it in the /prod/calcount location and there should be no collisions as the two calcount-service entities are in different namespaces and have different internal DNS entries as a result. If it turns out not to be the case, I’ll cross that bridge as it comes, but for now I’m fairly confident it should work.

So at this point we have a single-node k3s cluster, with calCount and an OTel collector running, accessible via HTTPS from the wider internet. As and when I want to update or change anything, I can update my Git manifest, pull it locally and run kubectl apply -k <directory> to make any changes. Which is awesome, and made the process of troubleshooting as I went through the build process so much easier as I could just run the change as I made it, and if I made any adjustments on the server-side instance of the repo I could push it back up to Github and from there sync it back to my notebook. The next step was to remove the need to me to be in the server to run the commands, I wanted to get to a full GitOps-style deployment structure that would update itself on push to the repository without me needing to touch anything within the VPS.

Picking a GitOps provider

There are two main GitOps systems people seem to gravitate to, FluxCD and ArgoCD. ArgoCD is easily the more popular service and has an impressive array of capabilities. For what I’m using it for and the system I’m going to be using it on, it seemed like overkill and too heavy on resources compared to the benefits offered. Even if I opted to remove the UI and installed core only, I wasn’t entirely sold on the whole thing. I ended up going with FluxCD as a lightweight alternative, partly for the lower resource usage and because as I read through the philosophy’s of the two offerings, FluxCD seemed to make more sense to me overall - I was better able to comprehend what it was doing and why.

Installing Flux

In what may have been my own downfall, I decided I wanted to use Flux to manage Flux, such that once it was up and running, it will continue to monitor itself and correct its own configuration drift. Conceptually it makes sense and I was sure it could be done, Gemini assured the same.

First things first, in my /apps/core-infra folder I created the Flux app definition. This is a pretty simple process - the Flux CLI generates the installation file and exports it to gotk-components.yaml. From there, create a kustomization that incorporates the aforementioned file and you’re good to go.

Deployment onto the server was a different experience, mostly due to my own misunderstanding of how Flux operates. Within _core/fluxcd went flux-sync.yaml and a secrets file with the access key to connect to the Github repo that the configuration was stored on. flux-sync defines the Github repository/repositories to be monitored by Flux for changes, as well as the specific kustomizations to apply to the cluster. Given I’m using a monorepo for the deployment, I only need to monitor the one repo, but if I expanded over time or had a specific app that was defined elsewhere I could include that too. The other component is a set of kustomizations, which are the specific deployments to monitor. You note which Github repo has the deployment you want to refer to and the path to check and it will review all the subdirectories within it for changes and apply them if any are detected. This is what threw me into disarray for a couple of hours. I had two paths that I was monitoring, ./clusters/vps-server, which was the configuration for the overall server, and ./clusters/vps-server/_core/fluxcd/, which had the Flux configuration for itself to monitor. There were ultimately two problems with this.

Fixing Flux

The first problem is in how Flux assesses systems that need to be remediated. From the path specified, it walks down the full hierarchy and triggers all the kustomize files that it sees. In my situation, the kustomize file in ./clusters/vps-server/_core/fluxcd/ was being referenced twice, once by each path. On seeing this, Flux found a different number of kustomizations to run than actually existed and would run into a panic condition and fail to sync. My troubleshooting with Gemini never seemed to pick up this conflict and it wasn’t until I fixed the issue that I could see what was happening. The fix was to break the checks down and have each service in ./clusters/vps-server/_core be its own check, and then have a single apps check for the contents of ./clusters/vps-server/environments/. This removed the double-assessment of kustomizations and fixed one of the issues. The second issue came in the form of dependencies. Unless explicitly specified otherwise, all kustomizations run effectively at the same time. This caused an issue where services that relied on others to function would fail if the dependent service hadn’t started yet, again causing a panic. The resolution for this was to define the dependencies within Flux to force ordering when running kustomizations. First to run would be Flux itself to reconcile the repository. From there, the init service runs to define the namespaces and install certbot. Once that is completed, the observability and environments kustomizations run to deploy the OTel cluster collector and service apps respectively.

Secrets management

Last thing to do was find a way to manage secrets within the GitOps deployment. Because each Git pull will overwrite any existing secrets that are manually written on the server in their configmaps, the two options available were to use a KMS or SOPS. The server isn’t running on AWS or Azure to use their key vaults, and I didn’t particularly want to run one locally for this environment, so SOPS it was. Actually using SOPS to get up and running was surprisingly easy. Create a public/private keypair with age, then encrypt your secrets file as documented. Add the encrypted file to your kustomization, ensure that the private key is on your server and accessible and you’re away. Except. By default, SOPS will encrypt every field in your YAML configuration, including kind and apiVersion. When your kustomization tries to then read the file, it can’t understand what the values are for this as validation of the content structure occurs before decryption of the values, so it will fail. So instead you need to adjust the RegEx for SOPS to it only encrypts the data and stringData fields. I ended up creating a sops.yaml configuration to automate this part:

creation_rules:
  - path_regex: .*\.enc\.yaml$
    encrypted_regex: "^(data|stringData)$"

So I don’t need to define it each time as part of the SOPS command. It took a few goes to understand what was happening here, as all the documentation I was looking at just implied it would work out of the box, and it took a few tries to get it right. I implemented this at the same time I did the Flux deployment, so the issues with flux-sync and SOPS combined in the worst way to make things harder to fix.

In the end though, I have the system up and running, and a full GitOps workflow for the end-to-end deployment process. A few changes are likely to come up in time, namely automating the ingest of the SOPS key into memory on install of the environment now that I understand the inheritance process, as well as expanding the services under management and clusters configured with Flux as I get them deployed. For now though, the system is up and running and seems to be doing so without issue.