Suckless Admission Webhook Setup
Here are some tricks and tips for setting up k8s webhook.
To be clear this is not the best practice, but more as a MVP guide to get things off the ground with minimum effort while maintaining reasonable standard.
When it comes to k8s admission webhook setup you can always:
- setup a cert manager
- provision a CA and master private key
- for the cert provision a private key, CRS and cert
but it always feel a bit overkill to introduce the cert manager just for a pair of self signed certs.
Here are the config I use recently to setup the admission wenhook e2e without the need for cert-manager.
variable "service_name" {
type = string
default = "webhooks"
}
variable "image_name" {
type = string
description = "name of the image to deploy"
}
resource "kubernetes_namespace" "thewebhook" {
metadata {
name = "thewebhook"
}
}
resource "kubernetes_service_account" "thewebhook" {
metadata {
name = "thewebhook"
namespace = kubernetes_namespace.thewebhook.metadata[0].name
}
}
resource "kubernetes_cluster_role" "thewebhook" {
metadata {
name = "thewebhook"
}
rule {
api_groups = [""]
resources = ["pods"]
verbs = ["create", "get", "list", "watch"]
}
rule {
api_groups = ["apigroup.domain"]
resources = ["sidecarprofiles"]
verbs = ["get", "list", "watch"]
}
}
resource "kubernetes_cluster_role_binding" "thewebhook" {
metadata {
name = "thewebhook"
}
role_ref {
api_group = "rbac.authorization.k8s.io"
kind = "ClusterRole"
name = kubernetes_cluster_role.thewebhook.metadata[0].name
}
subject {
kind = "ServiceAccount"
name = kubernetes_service_account.thewebhook.metadata[0].name
namespace = kubernetes_namespace.thewebhook.metadata[0].name
}
}
resource "tls_private_key" "ca_priv" {
algorithm = "RSA"
}
resource "tls_self_signed_cert" "ca" {
private_key_pem = tls_private_key.ca_priv.private_key_pem
is_ca_certificate = true
subject {
common_name = "thewebhook-ca"
country = "GB"
organization = "THE ORG"
organizational_unit = "THE OU"
}
allowed_uses = [
"digital_signature",
"cert_signing",
"crl_signing",
]
validity_period_hours = 43800 // 5 years
}
resource "tls_private_key" "server_priv" {
algorithm = "RSA"
}
resource "tls_cert_request" "server" {
private_key_pem = tls_private_key.server_priv.private_key_pem
subject {
common_name = "${var.service_name}.thewebhook.svc.cluster.local"
country = "GB"
organization = "THE ORG"
organizational_unit = "THE OU"
}
dns_names = [
"${var.service_name}.thewebhook.svc.cluster.local",
"${var.service_name}.thewebhook.svc",
"${var.service_name}.thewebhook",
]
}
resource "tls_locally_signed_cert" "server" {
cert_request_pem = tls_cert_request.server.cert_request_pem
ca_private_key_pem = tls_private_key.ca_priv.private_key_pem
ca_cert_pem = tls_self_signed_cert.ca.cert_pem
validity_period_hours = 17520 // 2 years
allowed_uses = [
"digital_signature",
"key_encipherment",
"server_auth",
"client_auth",
]
}
resource "kubernetes_secret" "tls" {
metadata {
name = "tls"
namespace = kubernetes_namespace.thewebhook.metadata[0].name
}
data = {
"tls.crt" = join("", [tls_locally_signed_cert.server.cert_pem, tls_locally_signed_cert.server.ca_cert_pem])
"tls.key" = tls_private_key.server_priv.private_key_pem
}
}
resource "kubernetes_deployment" "thewebhook" {
metadata {
name = "thewebhook"
namespace = kubernetes_namespace.thewebhook.metadata[0].name
}
spec {
replicas = 1
selector {
match_labels = {
app = "thewebhook"
}
}
template {
metadata {
labels = {
app = "thewebhook"
}
}
spec {
service_account_name = kubernetes_service_account.thewebhook.metadata[0].name
container {
name = "thewebhook"
image = var.image_name
command = [
"/manager",
"--metrics-bind-address=:8080",
"--sidecar-profile-namespace=${kubernetes_namespace.thewebhook.metadata[0].name}",
"--cert-path=/etc/thewebhook/certs",
]
env {
name = "TLS_SHA"
value = base64sha512(jsonencode(kubernetes_secret.tls.data))
}
port {
name = "webhook"
container_port = 9443
}
port {
name = "metrics"
container_port = 8080
}
volume_mount {
name = "tls"
mount_path = "/etc/thewebhook/certs"
read_only = true
}
}
volume {
name = "tls"
secret {
secret_name = kubernetes_secret.tls.metadata[0].name
}
}
}
}
}
}
resource "kubernetes_service" "thewebhook" {
metadata {
name = var.service_name
namespace = kubernetes_namespace.thewebhook.metadata[0].name
}
spec {
selector = {
app = "thewebhook"
}
port {
port = 9443
name = "webhook"
target_port = "webhook"
}
}
}
resource "kubernetes_mutating_webhook_configuration" "thewebhook" {
metadata {
name = "thewebhook"
}
webhook {
name = "thewebhook.APIGROUP.apigroup.DOMAIN"
client_config {
# url = "https://${var.service_name}.thewebhook.svc.cluster.local:9443/mutate-v1-pod"
service {
name = var.service_name
namespace = kubernetes_namespace.thewebhook.metadata[0].name
port = 9443
path = "/mutate-v1-pod"
}
ca_bundle = tls_self_signed_cert.ca.cert_pem
}
admission_review_versions = ["v1"]
failure_policy = "Ignore"
side_effects = "None"
rule {
api_groups = [""]
api_versions = ["v1"]
operations = ["CREATE"]
resources = ["pods"]
}
}
}
some caveats:
By the default the controller-runtime sets /tmp/k8s-webhook-server/serving-certs as the default directory for certificate. It's a bizarre location you probably want to change it to somewhere else that is more obvious via:
webhookServer := webhook.NewServer(webhook.Options{
CertDir: certPath,
TLSOpts: tlsOpts,
})
By default the webhook runs on port :9443 which makes sense but not very obvious (convention over configuration I guess!)
If you use service as part of your client config in the mutating webhook config, out of box the api server is going to reach out to https://${var.service_name}.thewebhook.svc:9443 as your webhook endpoint - make sure it's part of your cert's CN or SANs.
It's attempting to set https://${var.service_name}.thewebhook.svc.cluster.local:9443 as the URL as part of your client config in the mutating webhook configuration, but based on my experience I get a DNS resolve error on GKE. As the result avoid it if you can.
I set the failure_policy = "Ignore" so that the webhook doesn't turn into a single point of failure, otherwise if your webhook endpoint (which happen to be mutating or validating pods) is ever unavailable all the pod recreation will be rejected, which can be detrimental during a cluster upgrade where everything just get stuck.
The downside of failure_policy = "Ignore" is as I understand there is no way of debugging tls or dns issue in case you webhook is not setup properly in the first place. So you might want to set it to Fail so that in case of any DNS or TLS issue occur, you can always get feedback from running kubectl -n XXX get events.