Grafana on k3s (with metallb and nginx)

This is a follow-on from my earlier post, but hopefully k8s is consistent enough that you don’t need to know that.

Sadly, this is less helmy :(

Use the Prometheus-operator

We’ll use the prometheus operator to install and configure prometheus. First, let’s get everything created. This will clone a git repo then create a bunch of customResourceDefinitions and a monitoring namespace, and then install a lot of things into that namespace:

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus
kubectl create -f manifests/setup
kubectl create -f manifests/

After a while, things should be running:

NAME                                   READY   STATUS    RESTARTS   AGE
node-exporter-kldcv                    2/2     Running   0          82s
blackbox-exporter-6b6c8d67b8-bdwq5     3/3     Running   0          86s
prometheus-operator-58899ccbf4-nqzwd   2/2     Running   0          80s
kube-state-metrics-f76ffbf8b-ccmmd     3/3     Running   0          84s
prometheus-adapter-6b88dfd544-g6p8p    1/1     Running   0          80s
prometheus-adapter-6b88dfd544-shxxm    1/1     Running   0          80s
alertmanager-main-0                    2/2     Running   0          66s
alertmanager-main-1                    2/2     Running   0          66s
alertmanager-main-2                    2/2     Running   0          66s
prometheus-k8s-1                       2/2     Running   0          64s
prometheus-k8s-0                       2/2     Running   0          64s
grafana-59dd6bcdff-qlnck               1/1     Running   0          84s

There’s now a lot of services, and everything should be working given some way to access them:

$ kubectl get --namespace monitoring services
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-main       ClusterIP   10.43.190.168   <none>        9093/TCP,8080/TCP            3m11s
blackbox-exporter       ClusterIP   10.43.15.198    <none>        9115/TCP,19115/TCP           3m11s
grafana                 ClusterIP   10.43.222.161   <none>        3000/TCP                     3m9s
kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP            3m8s
node-exporter           ClusterIP   None            <none>        9100/TCP                     3m7s
prometheus-k8s          ClusterIP   10.43.171.195   <none>        9090/TCP,8080/TCP            3m6s
prometheus-adapter      ClusterIP   10.43.135.61    <none>        443/TCP                      3m5s
prometheus-operator     ClusterIP   None            <none>        8443/TCP                     3m4s
alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   2m50s
prometheus-operated     ClusterIP   None            <none>        9090/TCP                     2m48s

There’s a few options, but since I installed metallb in my last post I’m going to use a load balancer :) For this we’ll modify each service to be of type LoadBalancer:

$ for servicename in prometheus-k8s alertmanager-main grafana; do kubectl --namespace monitoring patch svc $servicename -p '{"spec": {"type": "LoadBalancer"}}' ; done

Now you should have some loadbalancers, but there’s some networkPolicies that block your access from a loadBalancer. I’ve just deleted them all but need to think of a better way to handle this:

kubectl -n monitoring delete networkpolicies.networking.k8s.io --all

Now you can access the services by their combination of load balancer IP and port; my grafana is at http://192.168.2.104:3000 here:

$ kubectl get service -n monitoring grafana
NAME      TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
grafana   LoadBalancer   10.43.222.161   192.168.2.104   3000:32085/TCP   154m