跳到主要内容

监控系统：VictoriaMetrics + Grafana

云原生家庭网络监控系统​

如何打造一个云原生家庭网络的监控系统？该如何选型？毫无疑问，在云原生领域，Prometheus 生态已成为监控标配，通常都会部署 Prometheus 来采集监控数据，配合 Grafana 来展示监控面板。

但是，Prometheus 的监控数据会占用很大的存储空间，采集数据时也会占用很大的内存，而路由器的计算资源往往都比较小，可不能像服务器那样奢侈。

好在 VictoriaMetrics 提供了很好的解决方案，兼容 Prometheus 的同时，有更强的性能，占用更小的存储空间和内存，在路由器上替代 Prometheus 再合适不过了，所以，本文将基于 VictoriaMetrics 在路由器上部署监控系统。

victoria-metrics-k8s-stack​

Prometheus 社区提供了 kube-prometheus-stack 这个 helm chart 来部署 Prometheus 及其生态内相关的产品，相应的，VictoriaMetrics 社区提供了对等的 chart: victoria-metrics-k8s-stack。

实际上路由器上的监控需求相对监控，也可以直接自己手工写 VictoriaMetrics + Grafana 的部署 YAML，但这个不利于后续的升级，升级不单单是升级容器镜像，相应的 YAML 和应用配置都可能改动，所以如果是要打算后续持续升级，享受最新的功能，建议是用社区的 victoria-metrics-k8s-stack 这个 helm chart 安装。

部署 EnvoyGateway​

由于使用社区的 chart 部署，但 chart 中不直接指定 Grafana 使用 HostNetwork 部署，那我们可以使用

使用 victoria-metrics-k8s-stack 部署监控系统​

目录结构​

monitoring
├── dashboards
│   ├── kustomization.yaml
│   └── router.json
├── kustomization.yaml
├── namespace.yaml
├── values.yaml
└── vm-hostpath-pv.yaml

配置监控面板​

dashboards 目录的作用就是配置我们需要自定义的 Grafana 监控面板，router.json 就是监控面板的定义，通过 kustomize 将 json 文件存储到 ConfigMap 中，以下是 dashboards/kustomization.yaml 的定义：

dashboards/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generatorOptions:
  disableNameSuffixHash: true
  labels:
    grafana_dashboard: "1"

configMapGenerator:
  - files:
      - router.json
    name: router

要点解析：

  • 通过 configMapGenerator 引用 router.json，将 Grafana 监控面板保存到名为 router 的 ConfigMap 中。后续有需要其它的自定义面板，也可以直接在这里增加。
  • 通过 generatorOptions 为自动生成的 ConfigMap 加上 grafana_dashboard: "1" 的 label，这个很关键，用作 Grafana 的 Sidecar 对监控面板的自动发现，以便让 Grafana 能够展示我们自定义的面板。

通常使用 node-exporter 提供的监控指标来捏监控面板，社区有开源的 Node Exporter Full 这样的监控面板，但太多了，一般我们可以根据自己的重点关注内容，自己来捏一个适合自己的面板，这里分享下我自己捏的面板(router.json)：

dashboards/router.json
{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "description": "",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "id": 68,
      "panels": [],
      "title": "概览",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "description": "",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "max": 1,
          "min": 0,
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 6,
        "w": 24,
        "x": 0,
        "y": 1
      },
      "id": 88,
      "options": {
        "displayMode": "lcd",
        "maxVizHeight": 300,
        "minVizHeight": 16,
        "minVizWidth": 8,
        "namePlacement": "auto",
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
            "lastNotNull"
          ],
          "fields": "",
          "values": false
        },
        "showUnfilled": true,
        "sizing": "auto",
        "valueMode": "color"
      },
      "pluginVersion": "10.4.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "1 - avg by (instance)(irate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval]))",
          "format": "table",
          "instant": true,
          "legendFormat": "CPU 使用率",
          "range": false,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "内存使用率",
          "range": false,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "1 - ( node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"} )",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "磁盘空间使用率",
          "range": false,
          "refId": "C"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "1 - ( node_filesystem_files_free{mountpoint=\"/\"} / node_filesystem_files{mountpoint=\"/\"} )",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "磁盘 inode 使用率",
          "range": false,
          "refId": "D"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "avg by (instance)(node_load5) / count by (instance)(node_cpu_seconds_total{mode=\"idle\"})",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "5分钟平均负载",
          "range": false,
          "refId": "E"
        }
      ],
      "title": "使用率",
      "transformations": [
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "names": [
                "instance",
                "Value #A",
                "Value #B",
                "Value #C",
                "Value #D",
                "Value #E"
              ]
            }
          }
        },
        {
          "id": "joinByField",
          "options": {
            "byField": "instance",
            "mode": "outer"
          }
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {},
            "includeByName": {},
            "indexByName": {},
            "renameByName": {
              "Value #A": "CPU 使用率",
              "Value #B": "内存使用率",
              "Value #C": "磁盘空间使用率",
              "Value #D": "磁盘 inode 使用率",
              "Value #E": "5 分钟平均负载"
            }
          }
        }
      ],
      "type": "bargauge"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "fixed"
          },
          "custom": {
            "align": "left",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "CPU 核数"
            },
            "properties": [
              {
                "id": "unit",
                "value": "none"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "内存大小"
            },
            "properties": [
              {
                "id": "unit",
                "value": "bytes"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "磁盘容量"
            },
            "properties": [
              {
                "id": "unit",
                "value": "bytes"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "当前温度"
            },
            "properties": [
              {
                "id": "unit",
                "value": "celsius"
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "启动时长"
            },
            "properties": [
              {
                "id": "unit",
                "value": "s"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 7
      },
      "id": 87,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true,
        "sortBy": []
      },
      "pluginVersion": "10.4.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_time_seconds - node_boot_time_seconds",
          "format": "table",
          "instant": true,
          "legendFormat": "uptime",
          "range": false,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "count by (instance) (node_cpu_seconds_total{mode=\"idle\"})",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "cpu",
          "range": false,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_memory_MemTotal_bytes",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "总内存",
          "range": false,
          "refId": "C"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_size_bytes{mountpoint=\"/\"}",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "总容量",
          "range": false,
          "refId": "D"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "avg by (instance) (node_hwmon_temp_celsius{chip=\"platform_coretemp_0\"})",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "temp",
          "range": false,
          "refId": "E"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_memory_MemAvailable_bytes",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "可用内存",
          "range": false,
          "refId": "F"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_memory_MemTotal_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes - node_memory_MemFree_bytes - node_memory_SReclaimable_bytes",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "已用内存",
          "range": false,
          "refId": "G"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_memory_Buffers_bytes + node_memory_Cached_bytes + node_memory_SReclaimable_bytes",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "buff/cache",
          "range": false,
          "refId": "H"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_memory_MemFree_bytes",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "空闲内存",
          "range": false,
          "refId": "I"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_memory_Shmem_bytes",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "共享内存",
          "range": false,
          "refId": "J"
        }
      ],
      "title": "系统信息",
      "transformations": [
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "names": [
                "instance",
                "Value #A",
                "Value #B",
                "Value #C",
                "Value #D",
                "Value #E",
                "Value #F",
                "Value #G",
                "Value #H",
                "Value #I",
                "Value #J"
              ]
            }
          }
        },
        {
          "id": "joinByField",
          "options": {
            "byField": "instance",
            "mode": "outer"
          }
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "instance": true
            },
            "includeByName": {},
            "indexByName": {
              "Value #A": 4,
              "Value #B": 1,
              "Value #C": 2,
              "Value #D": 3,
              "Value #E": 5,
              "Value #F": 7,
              "Value #G": 6,
              "Value #H": 9,
              "Value #I": 8,
              "Value #J": 10,
              "instance": 0
            },
            "renameByName": {
              "Value #A": "启动时长",
              "Value #B": "CPU 核数",
              "Value #C": "内存大小",
              "Value #D": "磁盘容量",
              "Value #E": "当前温度",
              "Value #F": "可用内存",
              "Value #G": "已用内存",
              "Value #H": "buff/cache 内存",
              "Value #I": "空闲内存",
              "Value #J": "共享内存"
            }
          }
        }
      ],
      "type": "table"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "custom": {
            "align": "left",
            "cellOptions": {
              "type": "auto"
            },
            "inspect": false,
            "minWidth": 50
          },
          "mappings": [],
          "thresholds": {
            "mode": "percentage",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "空间使用率"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "basic",
                  "type": "gauge",
                  "valueDisplayMode": "color"
                }
              },
              {
                "id": "max",
                "value": 1
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "inode 使用率"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "mode": "basic",
                  "type": "gauge",
                  "valueDisplayMode": "color"
                }
              },
              {
                "id": "max",
                "value": 1
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "inode 总量"
            },
            "properties": [
              {
                "id": "unit",
                "value": "none"
              },
              {
                "id": "custom.width",
                "value": 114
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "剩余 inode 数量"
            },
            "properties": [
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": null
                    },
                    {
                      "color": "green",
                      "value": 10480
                    }
                  ]
                }
              },
              {
                "id": "unit",
                "value": "none"
              },
              {
                "id": "custom.cellOptions",
                "value": {
                  "type": "color-text"
                }
              },
              {
                "id": "custom.width",
                "value": 139
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "剩余空间"
            },
            "properties": [
              {
                "id": "custom.cellOptions",
                "value": {
                  "type": "color-text"
                }
              },
              {
                "id": "thresholds",
                "value": {
                  "mode": "absolute",
                  "steps": [
                    {
                      "color": "red",
                      "value": null
                    },
                    {
                      "color": "green",
                      "value": 10737418240
                    }
                  ]
                }
              },
              {
                "id": "custom.width",
                "value": 110
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "设备"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 118
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "文件系统"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 103
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "挂载点"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 110
              }
            ]
          },
          {
            "matcher": {
              "id": "byName",
              "options": "总容量"
            },
            "properties": [
              {
                "id": "custom.width",
                "value": 94
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 4,
        "w": 24,
        "x": 0,
        "y": 11
      },
      "id": 29,
      "options": {
        "cellHeight": "sm",
        "footer": {
          "countRows": false,
          "fields": "",
          "reducer": [
            "sum"
          ],
          "show": false
        },
        "showHeader": true,
        "sortBy": []
      },
      "pluginVersion": "10.4.1",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_size_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"}",
          "format": "table",
          "instant": true,
          "legendFormat": "总容量",
          "range": false,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_avail_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"}",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "剩余空间",
          "range": false,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "1 - ( node_filesystem_avail_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"} / node_filesystem_size_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"} )",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "空间使用率",
          "range": false,
          "refId": "C"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "1 - ( node_filesystem_files_free{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"} / node_filesystem_files{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"} )",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "inode使用率",
          "range": false,
          "refId": "D"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_files{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"}",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "inode 总量",
          "range": false,
          "refId": "E"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_files_free{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"}",
          "format": "table",
          "hide": false,
          "instant": true,
          "legendFormat": "剩余 inode 数量",
          "range": false,
          "refId": "F"
        }
      ],
      "title": "磁盘",
      "transformations": [
        {
          "id": "filterFieldsByName",
          "options": {
            "include": {
              "names": [
                "device",
                "fstype",
                "mountpoint",
                "Value #A",
                "Value #B",
                "Value #C",
                "Value #D",
                "Value #E",
                "Value #F"
              ]
            }
          }
        },
        {
          "id": "seriesToColumns",
          "options": {
            "byField": "device"
          }
        },
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Value #E": false,
              "Value #F": false,
              "fstype 2": true,
              "fstype 3": true,
              "fstype 4": true,
              "fstype 5": true,
              "fstype 6": true,
              "mountpoint 1": false,
              "mountpoint 2": true,
              "mountpoint 3": true,
              "mountpoint 4": true,
              "mountpoint 5": true,
              "mountpoint 6": true
            },
            "includeByName": {},
            "indexByName": {
              "Value #A": 3,
              "Value #B": 4,
              "Value #C": 5,
              "Value #D": 14,
              "Value #E": 12,
              "Value #F": 13,
              "device": 0,
              "fstype 1": 1,
              "fstype 2": 6,
              "fstype 3": 8,
              "fstype 4": 10,
              "fstype 5": 15,
              "fstype 6": 17,
              "mountpoint 1": 2,
              "mountpoint 2": 7,
              "mountpoint 3": 9,
              "mountpoint 4": 11,
              "mountpoint 5": 16,
              "mountpoint 6": 18
            },
            "renameByName": {
              "Value #A": "总容量",
              "Value #B": "剩余空间",
              "Value #C": "空间使用率",
              "Value #D": "inode 使用率",
              "Value #E": "inode 总量",
              "Value #F": "剩余 inode 数量",
              "device": "设备",
              "fstype": "文件系统",
              "fstype 1": "文件系统",
              "fstype 2": "",
              "mountpoint": "挂载点",
              "mountpoint 1": "挂载点",
              "mountpoint 4": ""
            }
          }
        }
      ],
      "type": "table"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "celsius"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 15
      },
      "id": 89,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "avg by (sensor, instance) (node_hwmon_temp_celsius{chip=\"platform_coretemp_0\"})",
          "instant": false,
          "legendFormat": "传感器 {{sensor}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "温度",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "left",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "使用量"
            },
            "properties": [
              {
                "id": "unit",
                "value": "short"
              },
              {
                "id": "custom.axisPlacement",
                "value": "right"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 23
      },
      "id": 25,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "(1 - avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])))",
          "hide": false,
          "legendFormat": "使用率",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "sum by (instance) (irate(node_cpu_seconds_total{mode!=\"idle\"}[$__rate_interval])) ",
          "hide": false,
          "instant": false,
          "legendFormat": "使用量",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "CPU",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "left",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byName",
              "options": "使用量"
            },
            "properties": [
              {
                "id": "custom.axisPlacement",
                "value": "right"
              },
              {
                "id": "unit",
                "value": "bytes"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 23
      },
      "id": 53,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "1 - (node_memory_MemAvailable_bytes/node_memory_MemTotal_bytes)",
          "hide": false,
          "legendFormat": "使用率",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "node_memory_MemTotal_bytes-node_memory_MemAvailable_bytes",
          "hide": false,
          "instant": false,
          "legendFormat": "使用量",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "内存",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "left",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 32
      },
      "id": 80,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "(node_filesystem_size_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"} - node_filesystem_free_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"}) /\nnode_filesystem_size_bytes{fstype!~\"nfs.*|tmpfs|ramfs|vfat\", mountpoint!=\"/boot\"}",
          "hide": false,
          "instant": false,
          "legendFormat": "{{ mountpoint }}",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "磁盘使用率",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 30,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "binBps"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 32
      },
      "id": 56,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "sum(irate(node_network_receive_bytes_total{device=\"ppp0\"}[$__rate_interval]))",
          "instant": false,
          "legendFormat": "入流量",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "-sum(irate(node_network_transmit_bytes_total{device=\"ppp0\"}[$__rate_interval]))",
          "hide": false,
          "instant": false,
          "legendFormat": "出流量",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "流量",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 41
      },
      "id": 19,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "node_load1",
          "hide": false,
          "legendFormat": "1min",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "node_load5",
          "hide": false,
          "legendFormat": "5min",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "node_load15",
          "hide": false,
          "legendFormat": "15min",
          "range": true,
          "refId": "C"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "count(node_cpu_seconds_total{mode=\"idle\"})",
          "hide": false,
          "instant": false,
          "legendFormat": "核数",
          "range": true,
          "refId": "D"
        }
      ],
      "title": "平均负载",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "left",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 41
      },
      "id": 79,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "node_filefd_allocated",
          "hide": false,
          "instant": false,
          "legendFormat": "FD 数量",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "FD 数量",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 50
      },
      "id": 16,
      "panels": [],
      "title": "CPU 明细",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 51
      },
      "id": 27,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "(1 - avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])) by (cpu))",
          "legendFormat": "{{cpu}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "CPU 使用率(单核)",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "description": "",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "percentunit"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 12,
        "y": 51
      },
      "id": 22,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "avg(irate(node_cpu_seconds_total{mode!=\"idle\"}[$__rate_interval])) by (instance, mode)",
          "hide": false,
          "legendFormat": "{{mode}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "CPU 使用率明细",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 60
      },
      "id": 31,
      "panels": [],
      "title": "磁盘明细",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red"
              },
              {
                "color": "green",
                "value": 10
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byFrameRefID",
              "options": "B"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 61
      },
      "id": 40,
      "maxPerRow": 2,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "9.0.4",
      "repeat": "device",
      "repeatDirection": "h",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_size_bytes{device=~\"$device\"} - node_filesystem_free_bytes{device=~\"$device\"}",
          "format": "time_series",
          "instant": false,
          "legendFormat": "磁盘使用量",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "(node_filesystem_size_bytes{device=~\"$device\"} - node_filesystem_free_bytes{device=~\"$device\"}) /\nnode_filesystem_size_bytes{device=~\"$device\"}",
          "hide": false,
          "legendFormat": "磁盘使用率",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "磁盘空间使用情况 - $device",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "red"
              },
              {
                "color": "green",
                "value": 10
              }
            ]
          },
          "unit": "none"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byFrameRefID",
              "options": "B"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 69
      },
      "id": 41,
      "maxPerRow": 2,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "desc"
        }
      },
      "pluginVersion": "9.0.4",
      "repeat": "device",
      "repeatDirection": "h",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "exemplar": false,
          "expr": "node_filesystem_files{device=~\"$device\"} - node_filesystem_files_free{device=~\"$device\"}",
          "format": "time_series",
          "instant": false,
          "legendFormat": "使用量",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "(node_filesystem_files{device=~\"$device\"} - node_filesystem_files_free{device=~\"$device\"}) /\nnode_filesystem_files{device=~\"$device\"}",
          "hide": false,
          "legendFormat": "使用率",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "磁盘 inode 使用情况 - $device",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "binBps"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byRegexp",
              "options": "/IO 时间/"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 0,
        "y": 77
      },
      "id": 65,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "irate(node_disk_read_bytes_total{device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}[$__rate_interval])",
          "instant": false,
          "legendFormat": "{{device}} 读流量",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "irate(node_disk_written_bytes_total{device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}[$__rate_interval])",
          "hide": false,
          "instant": false,
          "legendFormat": "{{device}} 写流量",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "irate(node_disk_io_time_seconds_total{device=~\"(/dev/)?(mmcblk.p.+|nvme.+|rbd.+|sd.+|vd.+|xvd.+|dm-.+|md.+|dasd.+)\"}[$__rate_interval])",
          "hide": false,
          "instant": false,
          "legendFormat": "{{device}} IO 时间",
          "range": true,
          "refId": "C"
        }
      ],
      "title": "磁盘 IO",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "iops"
        },
        "overrides": [
          {
            "matcher": {
              "id": "byRegexp",
              "options": "/IO 时间/"
            },
            "properties": [
              {
                "id": "unit",
                "value": "percentunit"
              }
            ]
          }
        ]
      },
      "gridPos": {
        "h": 8,
        "w": 12,
        "x": 12,
        "y": 77
      },
      "id": 83,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "multi",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "irate(node_disk_reads_completed_total[$__rate_interval])",
          "instant": false,
          "legendFormat": "{{device}} 读 IOPS",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {
            "type": "prometheus",
            "uid": "${datasource}"
          },
          "editorMode": "code",
          "expr": "irate(node_disk_writes_completed_total[$__rate_interval])",
          "hide": false,
          "instant": false,
          "legendFormat": "{{device}} 写 IOPS",
          "range": true,
          "refId": "B"
        }
      ],
      "title": "磁盘 IOPS",
      "type": "timeseries"
    },
    {
      "collapsed": false,
      "gridPos": {
        "h": 1,
        "w": 24,
        "x": 0,
        "y": 85
      },
      "id": 71,
      "panels": [],
      "title": "Pod 监控",
      "type": "row"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 11,
        "w": 24,
        "x": 0,
        "y": 86
      },
      "id": 72,
      "options": {
        "legend": {
          "calcs": [
            "max"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true,
          "width": 550
        },
        "tooltip": {
          "mode": "single",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "sum(irate(container_cpu_usage_seconds_total{pod!=\"\"}[$__rate_interval])) by (namespace, pod)",
          "instant": false,
          "legendFormat": "{{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "CPU 使用量",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "bytes"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 11,
        "w": 24,
        "x": 0,
        "y": 97
      },
      "id": 73,
      "options": {
        "legend": {
          "calcs": [
            "max"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true,
          "width": 550
        },
        "tooltip": {
          "mode": "single",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "container_memory_usage_bytes{container!=\"\", pod!=\"\"}",
          "instant": false,
          "legendFormat": "{{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "内存使用量",
      "type": "timeseries"
    },
    {
      "datasource": {
        "type": "prometheus",
        "uid": "${datasource}"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green"
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          },
          "unit": "none"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 11,
        "w": 24,
        "x": 0,
        "y": 108
      },
      "id": 76,
      "options": {
        "legend": {
          "calcs": [
            "max"
          ],
          "displayMode": "table",
          "placement": "right",
          "showLegend": true,
          "width": 550
        },
        "tooltip": {
          "mode": "single",
          "sort": "desc"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P4169E866C3094E38"
          },
          "editorMode": "code",
          "expr": "sum by (pod) (container_file_descriptors{pod!=\"\", container!=\"\"})",
          "instant": false,
          "legendFormat": "{{pod}}",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "FD使用量",
      "type": "timeseries"
    }
  ],
  "refresh": "5s",
  "schemaVersion": 39,
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": false,
          "text": "VictoriaMetrics",
          "value": "P4169E866C3094E38"
        },
        "hide": 2,
        "includeAll": false,
        "multi": false,
        "name": "datasource",
        "options": [],
        "query": "prometheus",
        "queryValue": "",
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "type": "datasource"
      },
      {
        "allValue": "",
        "current": {
          "selected": true,
          "text": [
            "All"
          ],
          "value": [
            "$__all"
          ]
        },
        "datasource": {
          "type": "prometheus",
          "uid": "P4169E866C3094E38"
        },
        "definition": "label_values(node_filesystem_size_bytes{mountpoint!=\"/boot\", fstype!~\"nfs.*|tmpfs|ramfs|vfat\"},device)",
        "hide": 0,
        "includeAll": true,
        "label": "磁盘",
        "multi": true,
        "name": "device",
        "options": [],
        "query": {
          "qryType": 1,
          "query": "label_values(node_filesystem_size_bytes{mountpoint!=\"/boot\", fstype!~\"nfs.*|tmpfs|ramfs|vfat\"},device)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-6h",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "路由器监控大盘",
  "uid": "VCx4Fck4k",
  "version": 4,
  "weekStart": ""
}

展示效果：

配置主要的 kustomization.yaml​

下面我们来配主要的 kustomization.yaml：

kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: monitoring

resources:
  - namespace.yaml
  - vm-hostpath-pv.yaml
  - dashboards

helmCharts:
  - repo: https://victoriametrics.github.io/helm-charts
    name: victoria-metrics-k8s-stack
    releaseName: monitoring
    namespace: monitoring
    valuesFile: values.yaml

要点解析：

  • namespace 指定为 monitoring，因为监控系统涉及多个组件，单独放到一个命名空间里，方便管理。
  • resources 引用 namespace.yaml，用于创建 monitoring 命名空间。
  • resources 引用 vm-hostpath-pv.yaml，用于为 VictoriaMetrics 挂载 hostPath 的 PV（因为 chart 里没有提供直接定义 volume 类型的选项，只有手动定一个 hostPath 类型的 PV 来被 PVC 绑定）。
  • resources 引用 dashboards 目录，相当于将前面配置的监控面板加进来。
  • helmCharts 引用 victoria-metrics-k8s-stack 这个 chart，并提供 values.yaml 来自定义配置。

配置 namespace.yaml​

namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

配置 values.yaml​

value.syaml
defaultRules:
  create: false
defaultDashboardsEnabled: false
experimentalDashboardsEnabled: false
coreDns:
  enabled: false
alertmanager:
  enabled: false
vmalert:
  enabled: false
kubeApiServer:
  enabled: false
kubeControllerManager:
  enabled: false
kubeEtcd:
  enabled: false
kubeScheduler:
  enabled: false

victoria-metrics-operator:
  createCRD: true

vmagent:
  enabled: true
  spec:
    scrapeInterval: 10s
    externalLabels: null

vmsingle:
  enabled: true
  spec:
    resources:
      limits:
        cpu: 1200m
        memory: 1500Mi
    storage:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: ""
      volumeMode: Filesystem
      volumeName: vm-storage

grafana:
  service:
    type: LoadBalancer
    port: 3000
  #grafana.ini:
  #  users:
  #    home_page: /d/VCx4Fck4k/e8b7af-e794b1-e599a8-e79b91-e68ea7-e5a4a7-e79b98
  adminPassword: "123456"
  adminUser: "admin"
  defaultDashboardsTimezone: "Asia/Shanghai"
  auth:
    anonymous:
      enabled: true
      org_name: Main Org.
      org_role: Viewer
      hide_version: false
  testFramework:
    enabled: false

要点解析：

  • 通过很多 enabled: false 禁用掉很多不需要的组件和功能。
  • 启用 vmagent 用于采集监控配置，配置 scrapeInterval 可调整采集时间间隔。
  • 启用 vmsingle 用于存储监控数据，storage 配置 PVC 的定义，指定 volumeName 为 vm-storage，与后续我们手动定义的 hostPath 类型的 PV 同名，以便让 PVC 能够自动绑定该 PV。
  • grafana 的 service.type 指定为 LoadBalancer，k3s 内置了 servicelb 作为 LoadBalancer 类型 Service 的实现，会自动为 LoadBalancer 类型的 Service 在集群中每个节点都开一个 Service 中声明的端口。这里端口指定为 3000，即访问路由器的 3000 端口就能访问到 Grafana 面板。
  • grafana 的 adminUser 和 adminPassword 分别指定管理员的账号密码，用于登录 Grafana。
  • grafana 的 defaultDashboardsTimezone 指定 Grafana 页面显示时所用的时区，国内我们使用 Asia/Shanghai。
  • 我们可能希望进入 Grafana 时默认显示我们自定义的监控面板，可以在进入自定义面板页面后，复制其路径，然后取消上面的 grafana.ini 的注释，替换 home_page 的值为复制的路径，这样后续我们进入 Grafana 页面就默认显示自定义的监控面板了。

配置 vm-hostpath-pv.yaml​

apiVersion: v1
kind: PersistentVolume
metadata:
  name: vm-storage
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /data/victoria
    type: DirectoryOrCreate
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  volumeMode: Filesystem