Sam's Space

Building Calcount

Following on from my first minor project, building a simple log-generator for work, I wanted to expand to a “meatier” project that would take a bit more effort to pull together. The basic requirements were to make a CRUD application of some sort, with a database, application and some form of interface. I wanted it to also be something that I actually needed to use, which lead me towards building a simple calorie and macro tracker. At the same time, I also wanted to have a sample application I could use to build out more familiarity with OpenTelemetry and be able to use OTel contrib to pass telemetry to multiple provider backends, so being able to include this would be a cherry-on-top. So what could really have been a simple single app has instead evolved into a pseudo-microservice with far more complexity than is necessary for the complexity and scale involved with the application.

The High-Level Design

The overall application has the following components:

NGINX

Used as the entrypoint into the application and performs redirection of traffic to the microservices.

AUTH

The user management service, tracking new registrations and user logins.

API

The main database of logs entered by the users.

WEB

It’s not technically a backend service, but I’m very broadly defining backend as “on my server” in this case. This in particular is the website users will access over the internet.

OTEL COLLECTOR

An internal OTel collector bringing in traces for each of the services in the environment and processing them.

FRONTEND/CLI

The next step once the server-side components are in a good-enough state will be to create a simple desktop application to query and update the services. In itself it’s not at all necessary, but will provide some additional opportunities to learn and explore how OTel works in more depth.

Each of the server-side components is built as its own Docker container, and in the medium-term will be migrated to a Kubernetes deployment. At this point in time the repo configuration looks like this:

+---src
    +---backend
    ¦   ¦   .env
    ¦   ¦   .env.template
    ¦   ¦   compose.yaml
    ¦   ¦   
    ¦   +---api
    ¦   ¦   ¦   dockerfile
    ¦   ¦   ¦   requirements.txt
    ¦   ¦   ¦   
    ¦   ¦   +---app
    ¦   ¦           database.py
    ¦   ¦           logging_config.yaml
    ¦   ¦           main.py
    ¦   ¦           models.py
    ¦   ¦           schemas.py
    ¦   ¦           
    ¦   +---auth
    ¦   ¦   ¦   dockerfile
    ¦   ¦   ¦   requirements.txt
    ¦   ¦   ¦   
    ¦   ¦   +---app
    ¦   ¦           database.py
    ¦   ¦           logging_config.yaml
    ¦   ¦           main.py
    ¦   ¦           models.py
    ¦   ¦           schemas.py
    ¦   ¦           
    ¦   +---nginx
    ¦   ¦   ¦   dockerfile
    ¦   ¦   ¦   
    ¦   ¦   +---conf
    ¦   ¦           nginx.conf
    ¦   ¦           
    ¦   +---otel-collector
    ¦   ¦       config.yaml
    ¦   ¦       dockerfile
    ¦   ¦       
    ¦   +---web
    ¦       ¦   dockerfile
    ¦       ¦   requirements.txt
    ¦       ¦   
    ¦       +---app
    ¦           ¦   helpers.py
    ¦           ¦   logging_config.yaml
    ¦           ¦   main.py
    ¦           ¦   
    ¦           +---templates
    ¦                   base.html
    ¦                   dashboard.html
    ¦                   home.html
    ¦                   index.html
    ¦                   login.html
    ¦                   register.html
    ¦                   
    +---frontend
        +---cli

Each service has its own Dockerfile to build, and Github workflows will in-turn be used to auto-create the containers once it gets pushed into Github. Currently the database for both the API and Auth services are in-memory SQLite instances, but I will expand these to use PostgreSQL and possibly a MySQL instance in future to expand OTel opportunities.

AI Assistance in Developing

I would call myself an ‘advanced beginner’ in terms of development skill. I feel comfortable in what need to get done and what the design and considerations are in building things, but gaps exist in the details of how to do it. A lot of questions to Gemini and Co-Pilot have helped flesh out those gaps, particularly for the more simple or standardised questions such as how to set up the database initialisation using SQL Alchemy or simple FastAPI autocompletions. I didn’t like the idea of having AI actually write the files out for me though, so it was usually a discussion with an AI chat and then copying and tweaking the output from the chat into the file. having to actually read through each function before placing it into the file helped to also cement understanding of how the project was structured and flowed. It works, but it definitely has it’s issues. It might just be the limitations of the free plan I’m currently on, but there were several instances where the answer was enough to get started with an answer but needed significant ongoing refining to work correctly and as intended.

Current State

At the moment, I’d say we’ve reached MVP stage. The services all appear to work, as of today the OTel instrumentation works and is sending traces from Nginx and the application containers to the OTel collector, and it builds and runs consistently with Compose without any errors. From here there are a few areas to continue to work on:

  • Pushing traces to an external provider. Now that the OTel collector is processing everything, I need to work through pushing it up to Datadog, Clickstack or another service. Good news is this should only be a matter of updating my OTel config. Bad news is the OTel configs are very opaque about what is going wrong and how to correct it.
  • Build the GitHub workflows to auto-build the containers on push. Not a big step, but an important one.
  • Tests. Which will be a new experience for me as I’ve not done much in this area, but it will be a good challenge to work on, and hopefully expanding the above workflows to gate on failed tests as well.
  • Kubernetes deployments of the app via k3s and ArgoCD. I have the server ready, it will just be getting the actual configuration created and deployed. I’ve worked with similar in the past, but not as a full end-to-end GitOps-style workflow.