My current self-hosted Bluebuild solution

It just feels wrong/off for me to have my Bluebuild images hosted and built with GitHub CI. I now have a self-hosted solution for this that is, I believe, less failure-prone than using ordinary CI/CD.

The basic ingredients are as follows:

Private-facing server with adequate storage, accessed only via Tailscale, to serve as a build host

Docker or Podman

docker-registry-ui without authentication (because Tailscale)

Caddy web server to reverse-proxy the container registry

bluebuild-cli

Bash script to run the bluebuild program with logging and notifying on build failures

systemd service and systemd timer to run the Bash script on a schedule

Cosign package installed on the build host

The container registry is configured to allow pushing without authentication. I don’t need to worry about authentication because it is not open to the public internet. The bluebuild build command is simply supplied with the registry URL on my Tailnet.

Here is the Bash script I wrote to build Bazzite and Bluefin images and push them to the container registry:

#!/usr/bin/env bash

set -x

DATE=$(date '+%Y%m%d_%H%M%S')
LOGFILE="${HOME}/bluebuild-logs/${DATE}.log"
NTFY_SERVER="https://nas-aux.carp-wyvern.ts.net"

if [ ! -d "${HOME}/bluebuild-logs" ]; then
mkdir -p "${HOME}/bluebuild-logs"
fi

notify_fail() {
curl \
-H prio:urgent \
-H tags:warning \
-d "${1}: failed to build. See logs for details." \
"${NTFY_SERVER}/bluebuild"
}

print_header() {
echo "===================================================================="
echo "Start build of ${1}"
echo "Date: $(date '+%Y-%m-%d %H:%M:%S')"
echo "Commit: $(git log --oneline --no-decorate -n 1)"
echo "===================================================================="
}

print_footer() {
echo "===================================================================="
echo "End build of ${1}"
echo "Time elapsed: ${2}"
}

if [ ! -f "$(pwd)/cosign.key" ]; then
echo "cosign.key not found. Copying it from backup."
cp -fv "${HOME}/.cosign.key" "$(pwd)/cosign.key"
fi

for recipe in $(/bin/ls ./recipes/recipe-*.yml); do
print_header "$(basename -s .yml "$recipe")" >>"$LOGFILE"
start_time=$(date '+%s')
if ! bluebuild build \
-p \
--registry aux-remote.carp-wyvern.ts.net \
-B docker \
-S cosign \
"${recipe}" >>"$LOGFILE" 2>&1; then
notify_fail "$(basename -s .yml "$recipe")"
fi
end_time=$(date '+%s')
runtime=$((end_time - start_time))
time_elapsed=$(date -d @$runtime -u '+%H:%M:%S')
print_footer "$(basename -s .yml "$recipe")" "$time_elapsed" >>"$LOGFILE"
done

# vim: ts=4 sts=4 sw=4 et ai ft=bash

This script is then run with a systemd service and timer every day at 01:00 (CST). In terms of results, it is very similar to running it in a CI/CD pipeline. Each recipe is slated to run, and if one fails, a notification is sent to my ntfy server (also on my Tailnet), and the script continues to build the next recipe. Build output is redirected to logfiles. Everything is run as an unprivileged user from the repo’s root as the working directory.

git pull is run by systemd before the execution of the script, so when updates are pushed to the repo, they are reflected in the next scheduled build. If I need to build them immediately, I can just run systemctl --user start bluebuild_build.service manually to trigger a build. Nothing in the script changes the state of the git repository during the build process, so each run will be ensured a clean git repo.

So, yeah, that’s it. I think it’s kinda neat.