Setting up a personal server in 2023
Sometimes I just want to do a small thing, like creating a blog, and then end up noticing everything is horrible and I should probably start fixing things at the root.
In this instance, my personal server from a hoster didn’t receive an update in a long time.
$ cat /etc/os-release | grep PRETTY_NAME
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
Debian 9… that seems a bit old.
Stretch also had benefited from Long Term Support (LTS) until the end of June 2022.
Crap.
I also didn’t update my mail server, mail frontend, rss reader and anything else really in a long time. Time for some change.
Hetzner cloud
For whatever reason I decided to go with Hetzner as my hoster. They have some reasonably priced low-end on-demand servers in their cloud. Both the Cloud and DNS services have ansible plugins to work with them and expose a nice API.
What they do not provide however is an official way to install CoreOS on their servers. That won’t stop us though: one can boot the server into a recovery mode where one can download an image and write it to the disk.
# CoreOS
- name: Check if CoreOS is installed
shell: |
ssh -oStrictHostKeyChecking=no root@{{ swick_server.hcloud_server.ipv4_address }} << 'EOF'
cat /etc/os-release | grep CoreOS
EOF
register: check_os
failed_when: check_os.rc != 1 and check_os.rc != 0
- name: Install CoreOS
block:
- name: Boot into rescue system
hcloud_server:
name: "{{ swick_server.hcloud_server.name }}"
rescue_mode: linux64
ssh_keys: "{{ project.ssh.name }}"
state: restarted
- name: Wait for SSH to come up
wait_for:
host: "{{ swick_server.hcloud_server.ipv4_address }}"
port: 22
timeout: 1800
- name: Copy ignition config
command: scp -oStrictHostKeyChecking=no {{ cache }}/config.ign root@{{ swick_server.hcloud_server.ipv4_address }}:./config.ign
# TODO: make the install medium selection more robust (look at id, label?)
- name: Install CoreOS
shell: |
ssh -oStrictHostKeyChecking=no root@{{ swick_server.hcloud_server.ipv4_address }} << 'EOF'
wget "{{ fcos_image_url }}"
echo "{{ fcos_image_checksum }}" > checksum
sha256sum -c checksum || reboot
xz -dc "{{ fcos_image_filename }}" | dd of={{ server_install_device }} status=progress
mount {{ focs_image_boot_partition }} /mnt/
mkdir -p /mnt/ignition
cp config.ign /mnt/ignition/
umount /mnt
EOF
- name: Reboot to CoreOS
hcloud_server:
name: "{{ swick_server.hcloud_server.name }}"
state: restarted
- name: Wait for SSH to come up
wait_for:
host: "{{ swick_server.hcloud_server.ipv4_address }}"
port: 22
sleep: 10
timeout: 1800
when: check_os.rc
Ignition, podman and docker-compose
Now all we need is our ignition file for CoreOS to configure itself. I’m going for a docker-compose setup which gets controlled by systemd and ansible. All docker services should run as the core user in the user session and connect to an unprivileged podman service. I also need a way to store podman volumes on persistent memory.
So, first of all we need some tools that are not on the default CoreOS install, such as python for ansible and semanage for setting up the correct labels for our podman volume storage.
systemd:
units:
# disable ssh, enable when we're ready
- name: sshd.service
enabled: false
# install python and semanage as a layered package with rpm-ostree
# also re-enable ssh when we're done
- name: rpm-ostree-install-tools.service
enabled: true
contents: |
[Unit]
Description=Layer python and semanage with rpm-ostree
Wants=network-online.target
After=network-online.target
Before=zincati.service
ConditionPathExists=!/var/lib/%N.stamp
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/rpm-ostree install --apply-live --allow-inactive python3 policycoreutils-python-utils
ExecStart=/usr/bin/systemctl enable --now sshd.service
ExecStart=/bin/touch /var/lib/%N.stamp
[Install]
WantedBy=default.target
One trick I’m using here is to disable sshd in the ignition file to avoid ansible connecting before python is installed. Only after rpm-ostree install --apply-live succeeds sshd will get enabled and started. This allows ansible to use a simple wait_for task to wait for the server to become fully functional.
The other trick here is using the ConditionPathExists together with ExecStart=/bin/touch to avoid unnecessary work on later boots.
storage:
files:
# store volumes on persistent storage
- path: /home/core/.config/containers/storage.conf
mode: 0644
user:
name: core
group:
name: core
contents:
inline: |
[storage]
graphroot = "/var/storage/core-volumes"
systemd:
units:
# mount persistent storage
- name: var-storage.mount
enabled: true
contents: |
[Unit]
Description=Mount persistent storage
[Install]
WantedBy=storage-directories.service
[Mount]
What={{ swick_server_volume.hcloud_volume.linux_device }}
Where=/var/storage
# create volume storage directory on persistent storage
- name: storage-directories.service
enabled: true
contents: |
[Unit]
Description=Create volume storage directory on persistent storage
After=var-storage.mount
After=rpm-ostree-install-tools.service
[Install]
WantedBy=default.target
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/mkdir -p /var/storage/core-volumes
ExecStart=/usr/bin/chown core:core /var/storage/core-volumes
ExecStart=/usr/sbin/semanage fcontext -a -e /var/home/core/.local/share/containers/storage /var/storage/core-volumes
For storing the podman volumes on persistent memory we change the container storage config file for the core user to point the graphroot to a directory on our hetzner volume.
This volume is mounted by a system service, a mount point is created and the selinux context from the default graphroot is copied to that mount point.
At this point it should be possible to create podman volumes, pull images and run containers as the core user. We can even throw away the entire server, create a new one, attach the hetzner volume to the new server and all the volumes and images will still be available on the new server.
storage:
files:
# docker-compose
- path: /usr/local/bin/docker-compose
mode: 0755
user:
name: root
group:
name: root
contents:
source: {{ docker_compose_binary_url }}
verification:
hash: sha256-{{ docker_compose_binary_sha256 }}
# docker-compose instanced service
- path: /etc/systemd/user/docker-compose@.service
mode: 0644
contents:
inline: |
[Unit]
Description=%i service with docker compose
After=podman.socket
[Service]
Type=exec
WorkingDirectory=%E/docker-compose/%i
ExecStart=/usr/local/bin/docker-compose up --remove-orphans
[Install]
WantedBy=default.target
# set DOCKER_HOST environemnt variable to the podman socket
- path: /home/core/.config/environment.d/podman-docker-socket.conf
mode: 0644
user:
name: core
group:
name: core
contents:
inline: "DOCKER_HOST=unix://{{ docker.host }}"
links:
# create the podman system service socket
- path: /etc/systemd/user/sockets.target.wants/podman.socket
target: /usr/lib/systemd/user/podman.socket
mode: 0644
hard: false
systemd:
units:
# disable podman socket
- name: podman.socket
enabled: false
# disable docker socket
- name: docker.socket
enabled: false
It’s time to get docker-compose running. We can just download the release binary but docker-compose needs to communicate with the docker service, or in our case with the compatible podman service. Our podman service has to run in the core user session which we can achieve simply by symlinking the socket to /etc/systemd/user/sockets.target.wants/podman.socket. We can also just disable the system podman and docker sockets to avoid accidentally talking with them.
We also have to make sure docker-compose picks up the socket which should be available as /run/user/1000/podman/podman.sock. For that we create the podman-docker-socket.conf file in ~/.config/environment.d/ and set DOCKER_HOST as required.
The last piece of the puzzle is the docker-compose@.service instanced service: with it we can just throw a docker-compose.yaml file into ~/.config/docker-compose/$SERVICE and start the service with systemctl --user enable --now docker-compose@$SERVICE.
Running docker-compose services
version: "3.3"
services:
traefik:
image: "traefik:v2.10"
container_name: "traefik"
command:
- "--log.level=WARN"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.web.http.redirections.entrypoint.to=websecure"
- "--entrypoints.web.http.redirections.entrypoint.scheme=https"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.httpchallenge=true"
- "--certificatesresolvers.myresolver.acme.httpchallenge.entrypoint=web"
- "--certificatesresolvers.myresolver.acme.email={{ admin.mail }}"
- "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
security_opt:
- label=disable
volumes:
- "./letsencrypt:/letsencrypt"
- "{{ docker.host }}:/var/run/docker.sock:ro"
whoami:
image: "traefik/whoami"
container_name: "simple-service"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`{{ domain }}`)"
- "traefik.http.routers.whoami.entrypoints=web"
This docker-compose.yaml file starts traefik as reverse proxy with SSL/let’s encrypt support and the whoami service which will print something when connected to. Traefik will monitors for containers with specific labels to create certificates and route traffic and thus needs access to the docker socket. For this to work we need to turn off labels with security_opt: label=disable.
After all of this it should be possible to start the service with systemctl --user enable --now docker-compose@traefik and then see the whoami output with curl {{ domain }}.
Conclusion
All in all, this setup seems to be robust and as low maintenance as possible. CoreOS will update itself. Watchtower can be installed to automatically update containers when new images become available. The podman volumes are nicely separated on a detachable, persistent storage device with snapshot and backup functionality. The most unreliable part here is installing CoreOS via the recovery mode.
Do you have a comment?
Toot at me on mastodon or send me a mail!