基于 Pod 和 Service 注解的服务发现

背景

很多应用会为 Pod 或 Service 打上一些注解用于 Prometheus 的服务发现，如 prometheus.io/scrape: "true"，这种注解并不是 Prometheus 官方支持的，而是社区的习惯性用法，要使这种注解生效，还需结合 Prometheus 的采集配置，本文介绍具体的配置方法。

注意

如果你使用 kube-prometheus-stack 部署的监控系统，默认就会对自身的一些组件创建采集规则，比如会给 kube-state-metrics 创建 ServiceMonitor，并且 kube-state-metrics 的 Service 上也有 prometheus.io/scrape: "true" 的注解，如果配置了基于 Service 注解的服务发现，就会导致重复采集。

真实案例

istio 指标采集

istio 使用了这种 Pod 注解，当 Pod 被自动注入 sidecar 的同时也会被自动注入以下注解：

    prometheus.io/path: /stats/prometheus
    prometheus.io/port: "15020"
    prometheus.io/scrape: "true"

表示声明让 Prometheus 采集 Envoy Sidecar 暴露的 metrics，端口是 15020，路径是 /stats/prometheus。

除此之外，控制面组件 istiod 的 Pod 也会有类似注解：

    prometheus.io/port: "15014"
    prometheus.io/scrape: "true"

Kubernetes Addon 指标采集

Kubenretes 源码仓库中的一些 addon 组件也使用了这种注解，有的是 Pod 注解，有的是 Service 注解。

• coredns 使用 Service 注解：

  apiVersion: v1
  kind: Service
  metadata:
    name: kube-dns
    namespace: kube-system
    annotations:
      prometheus.io/port: "9153"
      prometheus.io/scrape: "true"  

• nodelocaldns 使用 Pod 注解：

  apiVersion: apps/v1
  kind: DaemonSet
  metadata:
    name: node-local-dns
    namespace: kube-system
    labels:
      k8s-app: node-local-dns
      kubernetes.io/cluster-service: "true"
      addonmanager.kubernetes.io/mode: Reconcile
  spec:
    updateStrategy:
      rollingUpdate:
        maxUnavailable: 10%
    selector:
      matchLabels:
        k8s-app: node-local-dns
    template:
      metadata:
        labels:
          k8s-app: node-local-dns
        annotations:
          prometheus.io/port: "9253"
          prometheus.io/scrape: "true"

Prometheus 采集配置

根据 Pod 注解动态采集

- job_name: "kubernetes-pods"

  kubernetes_sd_configs:
    - role: pod

  relabel_configs:
    # prometheus relabel to scrape only pods that have
    # `prometheus.io/scrape: "true"` annotation.
    - source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      action: keep
      regex: true

    # prometheus relabel to customize metric path based on pod
    # `prometheus.io/path: <metric path>` annotation.
    - source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
      action: replace
      target_label: __metrics_path__
      regex: (.+)

    # prometheus relabel to scrape only single, desired port for the pod
    # based on pod `prometheus.io/port: <port>` annotation.
    - source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: pod

根据 Service 注解动态采集

- job_name: "kubernetes-service-endpoints"

  kubernetes_sd_configs:
    - role: endpoints

  relabel_configs:
    # prometheus relabel to scrape only endpoints that have
    # `prometheus.io/scrape: "true"` annotation.
    - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      action: keep
      regex: true

    # prometheus relabel to customize metric path based on endpoints
    # `prometheus.io/path: <metric path>` annotation.
    - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
      action: replace
      target_label: __metrics_path__
      regex: (.+)

    # prometheus relabel to scrape only single, desired port for the service based
    # on endpoints `prometheus.io/port: <port>` annotation.
    - source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__

    # prometheus relabel to configure scrape scheme for all service scrape targets
    # based on endpoints `prometheus.io/scheme: <scheme>` annotation.
    - source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
      action: replace
      target_label: __scheme__
      regex: (https?)
    - action: labelmap
      regex: __meta_kubernetes_service_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: namespace
    - source_labels: [__meta_kubernetes_service_name]
      action: replace
      target_label: service

kube-prometheus-stack 采集配置方法

如果你使用 kube-prometheus-stack 来安装 Prometheus，需要在 additionalScrapeConfigs 里加上采集配置，示例:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      -  job_name: "kubernetes-service-endpoints"
         ...
      -  job_name: "kubernetes-pods"
         ...