Cilium with Nodelocal DNSCache

Overview

Nodelocal DNS Cache is used for DNS caching and acceleration, reducing coredns pressure and improving DNS query performance.

This article describes how to make Nodelocal DNSCache coexist with Cilium in TKE clusters.

Incompatibility with TKE's NodeLocalDNSCache Addon

When Cilium is installed and replaces kube-proxy, requests to coredns are intercepted and forwarded by Cilium's eBPF programs, and cannot be intercepted by the node-local-dns Pod on the node. Therefore, DNS caching capability cannot be achieved directly, and the addon's functionality becomes ineffective.

Cilium provides an official method to coexist with Nodelocal DNSCache by configuring CiliumLocalRedirectPolicy. However, if you are using TKE's NodeLocalDNSCache addon, coexistence with Cilium cannot be achieved even with CiliumLocalRedirectPolicy, because this addon uses HostNetwork and does not listen on node/Pod IPs (it listens on 169.254.20.10 and the kube-dns Cluster IP), making it impossible for CiliumLocalRedirectPolicy to redirect DNS traffic to the local node-local-dns Pod.

Therefore, if you want to use Nodelocal DNSCache in a cluster with Cilium installed, it is recommended to self-build Nodelocal DNSCache. Refer to the section below for details.

Self-Build Nodelocal DNSCache

One-Click Installation

You can use a script to automatically install and configure coexistence with Cilium:

bash -c "$(curl -sfL https://raw.githubusercontent.com/imroc/tke-guide/main/static/scripts/cilium.sh)" -- install-localdns

If your network environment cannot reach GitHub, use the site address:

bash -c "$(curl -sfL https://imroc.cc/tke/scripts/cilium.sh)" -- install-localdns

Manual Installation

For manual installation, follow these steps:

Save the following content to node-local-dns.yaml:

Note

The content below is adapted from the Manual Configuration approach in Cilium's official documentation Node-local DNS cache, modified from the official node-local-dns deployment YAML nodelocaldns.yaml. The image addresses have been replaced with Docker Hub mirror images for easy internal network pulling in TKE environments, and HINFO requests are disabled to prevent continuous log errors (VPC's DNS service does not support HINFO requests).

node-local-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns-upstream
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNSUpstream"
spec:
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  selector:
    k8s-app: kube-dns
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  Corefile: |
    __PILLAR__DNS__DOMAIN__:53 {
        errors
        cache {
                success 9984 30
                denial 9984 5
        }
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        health
        }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    .:53 {
        template ANY HINFO . {
            rcode NXDOMAIN
        }
        errors
        cache 30
        reload
        loop
        bind 0.0.0.0
        forward . __PILLAR__UPSTREAM__SERVERS__
        prometheus :9253
        }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
      annotations:
        prometheus.io/port: "9253"
        prometheus.io/scrape: "true"
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: node-local-dns
      hostNetwork: false
      dnsPolicy: Default # Don't use cluster DNS.
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - effect: "NoExecute"
        operator: "Exists"
      - effect: "NoSchedule"
        operator: "Exists"
      containers:
      - name: node-cache
        image: docker.io/k8smirror/k8s-dns-node-cache:1.26.4
        resources:
          requests:
            cpu: 25m
            memory: 5Mi
        args: ["-localip", "__PILLAR__DNS__SERVER__", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream", "-skipteardown=true", "-setupinterface=false", "-setupiptables=false"]
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9253
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /run/xtables.lock
          name: xtables-lock
          readOnly: false
        - name: config-volume
          mountPath: /etc/coredns
        - name: kube-dns-config
          mountPath: /etc/kube-dns
      volumes:
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
      - name: kube-dns-config
        configMap:
          name: kube-dns
          optional: true
      - name: config-volume
        configMap:
          name: node-local-dns
          items:
          - key: Corefile
            path: Corefile.base
---
# A headless service is a service with a service IP but instead of load-balancing it will return the IPs of our associated Pods.
# We use this to expose metrics to Prometheus.
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9253"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: node-local-dns
  name: node-local-dns
  namespace: kube-system
spec:
  clusterIP: None
  ports:
  - name: metrics
    port: 9253
    targetPort: 9253
  selector:
    k8s-app: node-local-dns

Install:

kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}) && sed -i "s/__PILLAR__DNS__SERVER__/$kubedns/g;" node-local-dns.yaml
kubectl apply -f node-local-dns.yaml

Save the following content to localdns-redirect-policy.yaml:

localdns-redirect-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
  name: nodelocaldns
  namespace: kube-system
spec:
  redirectFrontend:
    serviceMatcher:
      serviceName: kube-dns
      namespace: kube-system
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        k8s-app: node-local-dns
    toPorts:
    - port: "53"
      name: dns
      protocol: UDP
    - port: "53"
      name: dns-tcp
      protocol: TCP

Create the CiliumLocalRedirectPolicy (redirect DNS requests to the local node-local-dns pod):
```
kubectl apply -f localdns-redirect-policy.yaml
```

FAQ

sed Error: extra characters at the end of n command

When installing Nodelocal DNSCache on macOS, sed reports an error:

$ kubedns=$(kubectl get svc kube-dns -n kube-system -o jsonpath={.spec.clusterIP}) && sed -i "s/__PILLAR__DNS__SERVER__/$kubedns/g;" node-local-dns.yaml

sed: 1: "node-local-dns.yaml
": extra characters at the end of n command

This is because the built-in sed on macOS is not standard (GNU) and has slightly different syntax. Install the GNU version of sed:

brew install gnu-sed

And set the PATH:

PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"

Then open a new terminal and re-run the installation command.

Cannot Create CiliumLocalRedirectPolicy

The CiliumLocalRedirectPolicy feature is not enabled by default. You need to add --set localRedirectPolicies.enabled=true during installation to enable it.

If Cilium is already installed, update the Cilium configuration to enable it:

helm upgrade cilium cilium/cilium --version 1.19.4 \
  --namespace kube-system \
  --reuse-values \
  --set localRedirectPolicies.enabled=true

Then restart the operator and agent for the changes to take effect:

kubectl rollout restart deploy cilium-operator -n kube-system
kubectl rollout restart ds cilium -n kube-system

Overview​

Incompatibility with TKE's NodeLocalDNSCache Addon​

Self-Build Nodelocal DNSCache​

One-Click Installation​

Manual Installation​

FAQ​

sed Error: extra characters at the end of n command​

Cannot Create CiliumLocalRedirectPolicy​

References​