Duncan's Blog

Ephemeral Gitlab Runners

1,736 Charactersabout 5 minutesGitLabLightwhaleGitLabDockerProxmox

Lightwhale is an immutable Linux distro that is meant to run Docker containers. By (ab)using the default ephemeral nature, we can set up disposable Gitlab Runner VMs on a Proxmox hypervisor. And, by keeping all the scripting in the Hypervisor host, updating Lightwhale is as easy as changing the ISO file!

Prerequisites:

Installation

First, you'll have to set-up your Proxmox server to support snippets in the first place. Log in, and then navigate to Datacenter → Storage→ local. Click the dropdown beside 'Content' and ensure 'Snippets' is selected.

Now, you'll have to set up the hookscript on your Proxmox server. You'll need root access, the GUI alone can't configure it:

apt install sshpass
cd /var/lib/vz/snippets
editor lightwhale-gitlab-runner.sh

Now, copy and paste the following into your editor. All the user configuration is at the very top of the file, and the full config.toml is towards the end.

#!/bin/bash
# Configuration:
export GITLAB_URL="https://www.gitlab.com"
export GITLAB_RUNNER_TOKEN="<<your runner token here!>>"
export PASSWORD_SALT="<<salt, perhaps from `openssl rand -hex 32`?>>"
export NETWORK_BRIDGE="vmbr0"

# Proxmox hookscript passes the VM ID in position 1
if [[ -z $1 ]]; then
	echo "No ID passed to the script!"
	exit 1
fi

# Proxmox hookscript passes the state in position 2
# We only want to be called after the VM has successfully started
if [[ "$2" != "pre-stop" && "$2" != "post-start" ]]; then
	exit 0
fi

# Wait for the VM to boot and negotiate a network address
# We'll use the IPv6 link-local address as this comes up first
for i in {1..10}; do
	export ADDRESS=$(qm agent $1 network-get-interfaces | grep -m 1 fe80 | awk -F'"' '{print $4}')
	if [[ -n $ADDRESS ]]; then
		break
	fi
	sleep 5
done

if [[ -z $ADDRESS ]]; then
	echo "Could not determine IP Address!"
	exit 2
fi

export PASSWORD=$(echo "$ADDRESS,$1,$PASSWORD_SALT" | sha256sum - | cut -d ' ' -f 1)
export HOST="$ADDRESS%$NETWORK_BRIDGE"
export GITLAB_DIR="/mnt/lightwhale-data/gitlab-runner"

# If stopping, unregister the runner so it doesn't clutter gitlab
if [[ "$2" == "pre-stop" ]]; then
	sshpass -p $PASSWORD ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null op@$HOST 'docker exec -i gitlab-runner gitlab-runner unregister --all-runners'
	exit 0;
fi

sshpass -p opsecret ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null op@$HOST <<-EOF
	echo -e "opsecret\n$PASSWORD\n$PASSWORD" | passwd

	mkdir -p "$GITLAB_DIR"
	# The default Lightwhale network uses Google, but by removing this we default to the one provided by the DHCP server
	echo "$PASSWORD" | sudo -S sed -i '/1.1.1.2/d' /etc/resolv.conf

	# TODO: Can this be simplified to a GitLab register command?
	cat <<-CONFIG > "$GITLAB_DIR/config.toml"
		concurrent = 4
		check_interval = 0
		connection_max_age = "15m0s"
		shutdown_timeout = 0

		[session_server]
			session_timeout = 1800

		[[runners]]
			request_concurrency = 2
			name = "gitlab-runner"
			url = "$GITLAB_URL"
			token = "$GITLAB_RUNNER_TOKEN"
			token_obtained_at = 2026-03-17T22:07:02Z
			token_expires_at = 0001-01-01T00:00:00Z
			executor = "docker"
			[runners.cache]
				MaxUploadedArchiveSize = 0
			[runners.cache.s3]
			[runners.cache.gcs]
			[runners.cache.azure]
			[runners.docker]
				host = "unix:///var/run/docker.sock"
				cap_add = ["SYS_ADMIN"]
				devices = ["/dev/fuse"]
				image = "debian:stable"
				privileged = true
				disable_entrypoint_overwrite = false
				oom_kill_disable = false
				disable_cache = false
				volumes = ["/cache"]
				shm_size = 0
				network_mtu = 0
	CONFIG

	docker run -d --name gitlab-runner --restart always \
		-v $GITLAB_DIR:/etc/gitlab-runner \
		-v /var/run/docker.sock:/var/run/docker.sock \
		gitlab/gitlab-runner:latest

EOF

exit 0

Make sure it is executable!

chmod +x lightwhale-gitlab-runner.sh

Now, create a standard VM in Proxmox - Ensure that 'qemu agent' is checked on the System tab and remove the disk that is automatically created. Once created, make a note of the VM_ID and execute the following command:

qm set <VM_ID> --hookscript local:snippets/lightwhale-gitlab-runner.sh

And now you are all set up! You should be able to start the VM, and after about 30 seconds it should show up in your GitLab Instance/Group/Project Runner portal. This hookscript is also reusable across multiple VMs - just clone the VM and the rest is taken care of automatically!

Limitations

I am aware of the current limitations with these VMs:

  • Docker does not automatically prune images, wasting RAM
    To compensate for this, you may want to automatically shutdown / startup the VMs on a daily basis.
  • On an unclean shutdown, the Runner remains in GitLab.
    The only solution I'm aware of is to delete the entire Gitlab Runner, which is counterproductive.

By default, this allows for Docker-In-Docker for GitLab, including giving all containers SYS_ADMIN privileges. Concurrently running jobs may be at risk, so either use this only in a trusted context or limit concurrency to 1 and reboot the VM after each job.