If you are using Prometheus Operator, you have probably encountered an occasion where you wish that you could declare your own custom configuration.  Currently, there are a couple of use cases that require custom configurations because Prometheus Operator does not cover them.  For example, if your use case is relabeling metrics or black box probing, you will need declare your own custom configuration to accomplish these tasks.

When this article was published, custom configuration relied on a hack rather than a dedicated path method for including custom configurations.  However, since v0.19.0, this hack is not longer needed.  Prometheus Operator now allows you to include additional configs that will be merged with the configs that Prometheus Operator automatically generates.  If you are using this version or later, use the additional scrape configs feature rather than the method described here.

Before going down this rabbit hole, be warned that I am presenting a temporary work around until Prometheus Operator has better support for your use case that requires custom configuration.  Also be aware that this work around relies on hack that isn’t explicitly supported by Prometheus Operator, although the maintainers of Prometheus Operator acknowledge that the work around exists and provide some sparse documentation on how to use it.  Additionally, I have not had an opportunity to figure out how to mount the alerting rules in kubernetes 1.8+.  This issue was reported by Vinayak Saokar (@saokar).  If you can contribute to the solution, please post to the issue.

To understand how this workaround works, it helps to know how Prometheus Operator manages the configuration of each Prometheus instance deployed to your Kubernetes cluster.  When a value is assigned to the serviceMonitorSelector, Prometheus Operator creates a Kubernetes Secret object which contains the base64 encoded prometheus.yaml file used to configure Prometheus.  This secret can be inspected just like any other Kubernetes Secret.  Assuming you have an instance of prometheus deployed to the monitoring namespace via Prometheus Operator, you can inspect the secret with command below.

kubectl -n monitoring get secret -o yaml

You should see a file like the one below:

apiVersion: v1
data:
  configmaps.json:
    'base64 encoded string'
  prometheus.yaml:
    'base64 encoded string'
kind: Secret
metadata:
  annotations:
    generated: "true"
  creationTimestamp: 2037-10-10T20:35:47Z
  labels:
    managed-by: prometheus-operator
  name: prometheus-<my-deployment>
  namespace: monitoring
  resourceVersion: "0000"
  selfLink: /api/v1/namespaces/monitoring/secrets/prometheus-<my-deployment>
  uid: <some-uid>
type: Opaque

Notice that there are two base64 encoded files that are passed as data in the secret.  The first is a configmaps.json file that contains a checksum for the version of prometheus.yaml that passed in the data object of the secret.  If you don’t pass Prometheus Operator a serviceMonitorSelector, you can pass your own secret in place of this one.

Let’s take advantage of this to set up blackbox probing of my website joecreager.com.  Blackbox probing simply means to probe a machine that you can’t instrument internally.  For example, I can’t instrument the applications that make google’s search engine work, but I can probe google.com to see if it is up, and to see how long it takes to respond.  The company Pingdom actually sells premium services to do just that for your own website if you are concerned with things like uptime.

To do the blackbox probing, the first thing that we need is an application to handle the probes.  I am going to use blackbox_exporter.  You can get your own copies of the configuration that I am using for blackbox_exporter here.

To run blackbox_exporter in my prometheus cluster, I am using the deployment.yml and services.yml files below:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: blackbox-exporter
  namespace: default
  labels:
    app: blackbox-exporter
spec:
  replicas: 1
  strategy: 
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  progressDeadlineSeconds: 100
  minReadySeconds: 5
  revisionHistoryLimit: 10
  template:
    metadata:
      labels:
        app: blackbox-exporter
    spec:
      containers:
      - name: blackbox
        image: josephcreager/blackbox_exporter:0.11.0 
        imagePullPolicy: Always
        ports:
        - name: blackbox
          protocol: TCP
          containerPort: 9115
        livenessProbe:
          tcpSocket:
            port: blackbox
          initialDelaySeconds: 10
          timeoutSeconds: 10
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 2

apiVersion: v1
kind: Service
metadata:
  name: blackbox-exporter
  namespace: default
  labels:
    app: blackbox-exporter
spec:
  type: ClusterIP
  selector:
    app: blackbox-exporter
  ports:
  - name: blackbox
    port: 9115
    protocol: TCP
    targetPort: blackbox

If you copy these files to a directory, you can run kubectl apply -f . in that directory to create the resources in your own cluster or minkube instance.  This particular configuration is very simple, and only checks to see if probe returns 200.

After setting up our exporter, it’s time to create a prometheus instance to record the probe results.  You can get a copy of the files that I will use to create these resources here.  Let’s take care of some of the more straightforward stuff first.  We can create our configMaps, storage, and services before we need to worry about how to manage the secret.  Go ahead and kubectl create -f these files if you are following along.

apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: default
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard

The storage.yml file tells kubernetes to provision a persistent disk.  I am using GCE.  Refer to the kubernetes docs on StorageClass if you are using another provisioner.

apiVersion: v1
kind: Service
metadata:
  name: blackbox-prometheus
  namespace: default
  labels:
    #meta-monitoring: TODO enable monitoring of this service by another service
    #monitoring: custom
spec:
  selector:
    prometheus: blackbox 
  type: ClusterIP
  ports:
  - name: prometheus
    port: 9090
    targetPort: web
    protocol: TCP

There isn’t much to say about the services.yml file, but keep the port number 9090 in mind as we will use it later to access the prometheus UI.

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-rulefiles-blackbox
  namespace: default
  labels:
    role: prometheus-rulefiles
    prometheus: blackbox
data:
  recording.rules: |-
  up.rules: |-

I’m not doing much with my configMap.yml yet, but later on it will house any alerting or recording rules that I’d like to have.  It is mainly important because the secret we will create for our custom configuration relies on this file existing.

Finally, we can write our prometheusConfig.yml file.  This file won’t be consumed by kubernetes directly.  Instead, we will pass this file as a secret to kubernetes so that prometheus can consume it for it’s configs.  Normally, I would say don’t share your kubernetes secrets, but in this case, there isn’t much that is secret about this file.

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
alerting:
  alertmanagers:
  - kubernetes_sd_configs:
    - api_server: null
      role: endpoints
      namespaces:
        names:
        - monitoring
    scheme: http
    path_prefix: /
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      separator: ;
      regex: alertmanager-operated
      replacement: $1
      action: keep
rule_files:
- /etc/prometheus/rules/rules-0/*.rules
scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - http://localhost:8080/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter.default.svc.cluster.local:9115

If we wanted to, we could stop here and make a secrets.yml file that has a base64 encoded version of this file, plus a base64 encoded version of a configmaps.json file.  The first time I set blackbox_exporter up it was for work, and this is how I did it because I was in a crunch for time.  But manually editing and applying the secrets was a huge pain in the neck, so I wised up and wrote a little bash script to take care of generating the secret for me.  Bash may not be able to do floating point multiplication, but it’s great for templating.  I recommend that you use this script as well.  If you are on a mac, you may need to run brew update && brew install shasum.  If you are on linux, you should replace the shasum command with the linux equivalent in this script.

#!/bin/bash

CHECKSUM="$(cat configMap.yml | shasum -a 256)"
if [ "$?" -ne 0 ]; then
  echo "cannot find configMap.yml"
  exit 1
fi

CONFIGMAPS_JSON="{\"items\":[{\"key\":\"monitoring/prometheus-rulefiles-blackbox\",\"checksum\":\""$CHECKSUM"\"}]}"

PROMETHEUS_YAML="$(cat prometheusConfig.yml)"
if [ "$?" -ne 0 ]; then
  echo "cannot find PrometheusConfig.yml"
  exit 1
fi

SECRET="
kind: Secret
apiVersion: v1
metadata:
    name: prometheus-blackbox
    namespace: default
data:
    configmaps.json:
      \""$(echo "$CONFIGMAPS_JSON" | base64)"\"
    prometheus.yaml:
      \""$(echo "$PROMETHEUS_YAML" | base64)"\"
"

cat <<EOF | kubectl apply -f -
$(echo "$SECRET")
EOF

As you can see, I am generating the configmaps.json file and the secret dynamically, and then applying the secret.

Finally, we can create the prometheus instance by running kubectl apply -f on the the file below.  But remember, this only works if you have prometheus-operator running.  If you haven’t set up prometheus-operator, it only takes one minute.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: blackbox
  namespace: default
spec:
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: default
        selector:
          matchLabels:
            storage: blackbox-prometheus
        resources:
          requests:
            storage: 10Gi
  resources:
    limits:
      cpu: 0.5
      memory: 0.5Gi
    requests:
      cpu: 0.25
      memory: 0.25Gi
  ruleSelector:
    matchLabels:
      role: prometheus-rulefiles
      prometheus: blackbox

We could also make a public facing interface for our prometheus instance using nginx+basic auth.  But I’m not going to go through the trouble, because we can also use port forwarding to access the prometheus UI.  After applying the prometheus.yml file, run kubectl port-forward prometheus-blackbox-0 9090:9090 to forward the application port to your local machine.  Then, we can visit localhost:9090 to visit the prometheus UI.

The image above shows the probe_http_duration_seconds metrics for my site.  The response times are pretty good.  Eventually, I’ll set up alerting so that I can be notified if my site is unavailable for any reason.  In addition to monitoring my personal website, I use the same strategy to probe critical endpoints at work so that our team can be alerted if any endpoints are unavailable.  Hopefully this information saves you a lot of time as you set up your own blackbox probes.