Skip to content

Bootstrapping K3s on Bare Metal Heztner Running on Tailnet

This guide walks you through the steps to bootstrap a k3s cluster on a bare metal heztner server. Most significantly we will be:

  • Make sure that the k3s is only privately accessible within the Tailscale tailnet, with only port 443 ports open for the public internet.
  • Despite the server is standalone, clustering it with worker nodes will be trivial.

Prerequisites

  • A heztner dedicated server
  • A tailscale that has been installed on both of your workstation and the heztner server
  • k3sup for bootstrapping the k3s cluster.

Tailscale setup

On your workstation make sure that the tailscale is up and running. On the heztner server, you can pretty much follow the instruction here.

To ensure smooth bootstrapping, please make sure that the tailscale on the server is bootstrapped with

tailscale up --ssh

so that you can ssh into the server during bootstrap.

k3sup setup

This is as simple as

k3sup install \
    --host $SERVER_HOST \
    --user "root" \
    --k3s-extra-args "--flannel-iface tailscale0 \
        --advertise-address $TAILSCALE_IP \
        --node-ip $TAILSCALE_IP \
        --node-external-ip $SERVER_EXTERNAL_IP \
        --tls-san $SERVER_HOST" \
    --k3s-channel latest \
    --context $SERVER_HOST \
    --local-path $HOME/.kube/config \
    --merge

There are a few things to note here:

  • --user root is made possible by the --ssh flag on tailscale up. You can also use custom user but you have to make sure that the user has passwordless sudo access.
  • --flannel-iface tailscale0 is the interface that flannel will use to create the overlay network. We use tailscale0 as the interface.
  • --advertise-address $TAILSCALE_IP is the apiserver IP that are advertised for other nodes to reach out to.
  • --tls-san $SERVER_HOST allows the kubectl client to reach out to the server via the hostname thus fewer ip to be hardcoded.
  • --node-external-ip $SERVER_EXTERNAL_IP is the public ip address of the server. This is useful if you want public access to your metallb loadbalancer on the k3s cluster.

Cluster join

TBD (This has been done before for the home cluster but somehow I haven't documented it).

Hetzner Firewall configuration

For incoming traffic

  • Accept IPV4 ICMP
  • Accept IPV4 TCP 443 for external access
  • Accept IPV4 TCP 0-65535 ACK.

Notes the default template from Hetzner's range for TCP ACK is 32768-65535 TCP. This matches the Linux default of ephemeral ports range, however the CNI plugin bypass the port range while performing SNAT. As a result, we need to open up the entire port range for TCP ACK to be accepted. I was not aware of this initially, and ended up with intermittent ACK blackhole where traffic from high ephemeral TCP ports ranges work fine while the lower ranges are entirely hang. I learned the lession the painful way via tcpdump :/

For outgoing traffic

  • Accept all outgoing traffic (since we are not doing any air-gapping).

Extract the kubeconfig to be used somewhere else

kubectl config view -o yaml --raw --minify --context=$THE_CLUSTER_CONTEXT

Personally I extracted the cluster kubeconfig and stored it to the gcp secret manager to be used by terraform.