Rotating webhooks certificates

Linkerd uses the Kubernetes admission webhooks and extension API server to implement some of its core features like automatic proxy injection, service profiles validation and tap.

To secure the connections between the Kubernetes API server and the webhooks, all the webhooks are TLS-enabled. The x509 certificates used by these webhooks are issued by the self-signed CA certificates embedded in the webhooks configuration.

By default, these certificates have a validity period of 365 days. They are stored in the following secrets, in the linkerd namespace: linkerd-proxy-injector-tls, linkerd-sp-validator-tls, linkerd-tap-tls.

The rest of this documentation provides instructions on how to renew these certificates.

Renewing the webhook certificates

To check the validity of all the TLS secrets (using step):

for secret in "linkerd-proxy-injector-k8s-tls" "linkerd-sp-validator-k8s-tls" "linkerd-tap-k8s-tls"; do \
  kubectl -n linkerd get secret "${secret}" -ojsonpath='{.data.tls\.crt}' | \
    base64 --decode - | \
    step certificate inspect - | \
    grep -iA2 validity; \
done

Manually delete these secrets and use linkerd upgrade to recreate them:

for secret in "linkerd-proxy-injector-k8s-tls" "linkerd-sp-validator-k8s-tls" "linkerd-tap-k8s-tls"; do \
  kubectl -n linkerd delete secret "${secret}"; \
done

linkerd upgrade | kubectl apply -f -

The above command will recreate the secrets without restarting Linkerd.

Note

For Helm users, use the helm upgrade command to recreate the deleted secrets.

If you render the helm charts externally and apply them with kubectl apply (e.g. in a CI/CD pipeline), you do not need to delete the secrets manually, as they wil be overwritten by a new cert and key generated by the helm chart.

Confirm that the secrets are recreated with new certificates:

for secret in "linkerd-proxy-injector-tls" "linkerd-sp-validator-tls" "linkerd-tap-tls"; do \
  kubectl -n linkerd get secret "${secret}" -ojsonpath='{.data.crt\.pem}' | \
    base64 --decode - | \
    step certificate inspect - | \
    grep -iA2 validity; \
done

Ensure that Linkerd remains healthy:

linkerd check

Restarting the pods that implement the webhooks and API services is usually not necessary. But if the cluster is large, or has a high pod churn, it may be advisable to restart the pods manually, to avoid cascading failures.

If you observe certificate expiry errors or mismatched CA certs, restart their pods with:

kubectl -n linkerd rollout restart deploy \
  linkerd-proxy-injector \
  linkerd-sp-validator \
  linkerd-tap