Cilium with NodeLocal DNSCache Coexistence

Overview

NodeLocal DNS Cache is used for DNS caching and acceleration, which can reduce the pressure on CoreDNS and improve DNS query performance.

This article describes how to achieve coexistence between Cilium and NodeLocal DNSCache in TKE clusters where Cilium is installed.

Incompatibility with TKE's NodeLocalDNSCache Addon

When Cilium is installed and replaces kube-proxy, requests accessing CoreDNS are intercepted and forwarded by Cilium's eBPF program, making them unable to be intercepted by the node-local-dns Pod on the node. This prevents the direct implementation of DNS caching capabilities, rendering the addon's functionality ineffective.

Cilium officially provides a method to achieve coexistence with NodeLocal DNSCache by configuring CiliumLocalRedirectPolicy. However, if you are using TKE's NodeLocalDNSCache addon, even configuring CiliumLocalRedirectPolicy cannot achieve coexistence. This is because the addon uses HostNetwork and does not listen on node/Pod IP addresses (it listens on 169.254.20.10 and CoreDNS's Cluster IP), preventing DNS traffic from being redirected by CiliumLocalRedirectPolicy to the local node-local-dns Pod.

Therefore, if you want to use NodeLocal DNSCache in a cluster with Cilium installed, it is recommended to build your own NodeLocal DNSCache. The specific method is described below.

Self-built NodeLocal DNSCache

Save the following content to a file node-local-dns.yaml:

Note

The following content is based on the Manual Configuration method from Cilium's official documentation Node-local DNS cache, modified from the official NodeLocal DNS deployment YAML file nodelocaldns.yaml. Additionally, the image address has been replaced with a mirror image on Docker Hub for convenient internal network pulls in TKE environments, and HINFO requests have been disabled to avoid continuous log errors (VPC's DNS service does not support HINFO requests).

node-local-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns-upstream
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNSUpstream"
spec:
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  selector:
    k8s-app: kube-dns
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  Corefile: |
    __PILLAR__DNS__DOMAIN__:53 {
        errors
        cache {
                success 9984 30
                denial 9984 5
        }
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        health
        }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    .:53 {
        template ANY HINFO . {
            rcode NXDOMAIN
        }
        errors
        cache 30
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__UPSTREAM__SERVERS__
        prometheus :9253
        }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
      annotations:
        prometheus.io/port: "9253"
        prometheus.io/scrape: "true"
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: node-local-dns
      hostNetwork: false
      dnsPolicy: Default # Don't use cluster DNS.
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - effect: "NoExecute"
        operator: "Exists"
      - effect: "NoSchedule"
        operator: "Exists"
      containers:
      - name: node-cache
        image: docker.io/k8smirror/k8s-dns-node-cache:1.26.4 # 官方镜像在 registry.k8s.io，这里替换为 dockerhub 上的 mirror 镜像（TKE 环境默认有 dockerhub 加速，可直接内网拉取）
        resources:
          requests:
            cpu: 25m
            memory: 5Mi
        args: ["-localip", "__PILLAR__DNS__SERVER__", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream", "-skipteardown=true", "-setupinterface=false", "-setupiptables=false"]
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9253
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /run/xtables.lock
          name: xtables-lock
          readOnly: false
        - name: config-volume
          mountPath: /etc/coredns
        - name: kube-dns-config
          mountPath: /etc/kube-dns
      volumes:
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      - name: config-volume
        configMap:
          name: node-local-dns
          items:
          - key: Corefile
            path: Corefile.base
---
# A headless service is a service with a service IP but instead of load-balancing it will return the IPs of our associated Pods.
# We use this to expose metrics to Prometheus.
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9253"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: node-local-dns
  name: node-local-dns
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - name: metrics
    port: 9253
    targetPort: 9253
  selector:
    k8s-app: node-local-dns

Install:

kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}) && sed -i "s/__PILLAR__DNS__SERVER__/$kubedns/g;" node-local-dns.yaml
kubectl apply -f node-local-dns.yaml

Save the following content to a file localdns-redirect-policy.yaml:

localdns-redirect-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
  name: nodelocaldns
  namespace: kube-system
spec:
  redirectFrontend:
    serviceMatcher:
      serviceName: kube-dns
      namespace: kube-system
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        k8s-app: node-local-dns
    toPorts:
    - port: "53"
      name: dns
      protocol: UDP
    - port: "53"
      name: dns-tcp
      protocol: TCP

Create CiliumLocalRedirectPolicy (redirects DNS requests to the local node-local-dns pod):
```
kubectl apply -f localdns-redirect-policy.yaml
```

Common Issues

sed Error: extra characters at the end of n command

When installing NodeLocal DNSCache on macOS, sed may report an error:

$ kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}) && sed -i "s/__PILLAR__DNS__SERVER__/$kubedns/g;" node-local-dns.yaml

sed: 1: "node-local-dns.yaml
": extra characters at the end of n command

This occurs because macOS's built-in sed command is not standard (GNU) and has different syntax. Install GNU version of sed:

brew install gnu-sed

And set the PATH:

PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

Then open a new terminal and re-execute the installation command.

Unable to Create CiliumLocalRedirectPolicy

The CiliumLocalRedirectPolicy capability is not enabled by default. It needs to be enabled during installation by adding the parameter --set localRedirectPolicies.enabled=true.

If Cilium is already installed, update the Cilium configuration to enable it:

helm upgrade cilium cilium/cilium --version 1.18.3 \
  --namespace kube-system \
  --reuse-values \
  --set localRedirectPolicies.enabled=true

Then restart the operator and agent for the changes to take effect:

kubectl rollout restart deploy cilium-operator -n kube-system
kubectl rollout restart ds cilium -n kube-system

Overview​

Incompatibility with TKE's NodeLocalDNSCache Addon​

Self-built NodeLocal DNSCache​

Common Issues​

sed Error: extra characters at the end of n command​

Unable to Create CiliumLocalRedirectPolicy​

References​