Why?

kube-prometheus is an open-source project created by the maintainers of Prometheus Operator. It provides a set of configuration files and resource definitions for deploying a complete Prometheus monitoring stack on Kubernetes.

kube-prometheus utilizes Prometheus Operator to set up and manage the following components:

  • Prometheus: Used for collecting and storing metric data.
  • Alertmanager: Used for handling alerts.
  • Grafana: Used for visualizing metric data.
  • Prometheus node-exporter: Used for collecting metrics from k8s nodes.
  • kube-state-metrics: Used for collecting metrics from the k8s API server.
  • Prometheus Adapter: Used for exposing Prometheus metrics to the k8s custom metrics API.

Pros:

  • Complete solution: It provides everything needed to establish a comprehensive monitoring infrastructure on Kubernetes.
  • Out-of-the-box functionality: Preconfigured alerts and dashboards allow you to quickly start monitoring your cluster and applications without the need to configure everything from scratch.
  • Customizable: Although kube-prometheus provides a complete starting point, you can freely adjust and extend it according to your specific requirements.
  • Community support: As part of the Prometheus Operator ecosystem, kube-prometheus benefits from an active community that continuously contributes new features, bug fixes, and improvements.

Cons:

  1. Complexity: kube-prometheus introduces a considerable number of components and configurations. Understanding and managing these require some time and effort.
  2. Resource overhead: Running a full monitoring stack can consume a significant amount of resources, however does not seem to be the troubling for my little k3s cluster
  3. Upgrade management: Over time, managing upgrades of the kube-prometheus components can become challenging.

Prepare kube-prometheus Library

The prerequisites specified that kubelet configuration must contains following flags:

  • --authentication-token-webhook=true
  • --authorization-mode=Webhook From what I checked, the k3s agent from the latest version (1.22 and above) already have them included, so we should be good.

kube-prometheus provided the installing guide for anyone who wants to customize the configuration:

1
2
3
4
5
6
7
8

$ mkdir my-kube-prometheus; cd my-kube-prometheus
$ jb init  # Creates the initial/empty `jsonnetfile.json`
# Install the kube-prometheus dependency
$ jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@<release-version> # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json`

$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/example.jsonnet -O example.jsonnet
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
$ chmod +x build.sh

Note that when jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@<release-version> choose the release that’s compatible with the cluster version

Customization for Metallb

I have deployed Metallb for the services on my k3s cluster which works great, and I want to use it to expose Grafana, Prometheus and Alertmanager, so I can access them easily without needing to port-forward lol

With kube-prometheus, it’s pretty simple, just override the service config:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132

local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/addons/anti-affinity.libsonnet') +
  // (import 'kube-prometheus/addons/managed-cluster.libsonnet') +
  // (import 'kube-prometheus/addons/node-ports.libsonnet') +
  // (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  // (import 'kube-prometheus/addons/custom-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/external-metrics.libsonnet') +
  // (import 'kube-prometheus/addons/pyrra.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },
    },
    grafana+:: {
      service+: {
        metadata+: {
          annotations+: {
            'metallb.universe.tf/allow-shared-ip': 'grafana',
          },
        },
        spec+: {
          type: 'LoadBalancer',
          loadBalancerIP: '192.168.10.247',
          ports: [
            {
              name: 'grafana-tcp',
              protocol: 'TCP',
              port: 3000,
              targetPort: 'http',
            },
          ],
        },
      },
      networkPolicy+: {
        spec+: {
          ingress: [{}],  // allow all
        },
      },
    },
    prometheus+:: {
      prometheus+: {
        spec+: {
          retention: '7d',  // default is 24h
        },
      },
      service+: {
        metadata+: {
          annotations+: {
            'metallb.universe.tf/allow-shared-ip': 'prometheus',
          },
        },
        spec+: {
          type: 'LoadBalancer',
          loadBalancerIP: '192.168.10.246',
          ports: [
            {
              name: 'web-tcp',
              protocol: 'TCP',
              port: 9090,
              targetPort: 'web',
            },
            {
              name: 'reloader-web-tcp',
              protocol: 'TCP',
              port: 8080,
              targetPort: 'reloader-web',
            },
          ],
        },
      },
      networkPolicy+: {
        spec+: {
          ingress: [{}],  // allow all
        },
      },
    },
    alertmanager+:: {
      service+: {
        metadata+: {
          annotations+: {
            'metallb.universe.tf/allow-shared-ip': 'alertmanager',
          },
        },
        spec+: {
          type: 'LoadBalancer',
          loadBalancerIP: '192.168.10.245',
          ports: [
            {
              name: 'web-tcp',
              protocol: 'TCP',
              port: 9093,
              targetPort: 'web',
            },
            {
              name: 'reloader-web-tcp',
              protocol: 'TCP',
              port: 8080,
              targetPort: 'reloader-web',
            },
          ],
        },
      },
      networkPolicy+: {
        spec+: {
          ingress: [{}],  // allow all
        },
      },
    },
  };

{ 'setup/0namespace-namespace': kp.kubePrometheus.namespace } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor' && name != 'prometheusRule'), std.objectFields(kp.prometheusOperator))
} +
// { 'setup/pyrra-slo-CustomResourceDefinition': kp.pyrra.crd } +
// serviceMonitor and prometheusRule are separated so that they can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ 'prometheus-operator-prometheusRule': kp.prometheusOperator.prometheusRule } +
{ 'kube-prometheus-prometheusRule': kp.kubePrometheus.prometheusRule } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
// { ['pyrra-' + name]: kp.pyrra[name] for name in std.objectFields(kp.pyrra) if name != 'crd' } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }

Note that I also override NetworkPolicy for them to allow all ingress traffic, otherwise the default config only allows Prometheus to access them. Technically I should fine grain this config, but since everything runs in my LAN which has no exposure to Internet, I think it should be fine.

Build and Install

Run ./build.sh in the folder (e.g. my-kube-prometheus) will generate all the manifests in manifests folder. Then just install with kubectl

1
2
3
4
5
6

$ kubectl apply --server-side -f manifests/setup
$ kubectl wait \
	--for condition=Established \
	--all CustomResourceDefinition \
	--namespace=monitoring
$ kubectl apply -f manifests/

That’s it.

Node Exporter