ACK 的 Terway 网络插件

Terway 插件的三种模式

Terway 插件支持三种模式：

默认模式（不勾选 DataPath V2 和 NetworkPolicy 支持）
勾选 DataPath V2。
勾选 NetworkPolicy （依赖勾选 DataPath V2）

组件部署 YAML

默认
启用 DataPath V2
启用 NetworkPolicy 支持

terway-eniip
eni-config

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: terway-eniip
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: terway-eniip
  template:
    metadata:
      annotations:
        k8s.alibabacloud.com/change-id: yutzvutv
      labels:
        app: terway-eniip
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
                - arm64
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
              - key: alibabacloud.com/nodepool-type
                operator: NotIn
                values:
                - hybridcloud
              - key: k8s.aliyun.com/ignore-by-terway
                operator: NotIn
                values:
                - "true"
      containers:
      - command:
        - /usr/bin/terwayd
        - -log-level
        - info
        - -daemon-mode
        - ENIMultiIP
        - -config
        - /etc/eni/eni_conf
        env:
        - name: TERWAY_GC_RULES
          value: "true"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/terway:v1.15.3
        imagePullPolicy: IfNotPresent
        name: terway
        resources:
          limits:
            cpu: 100m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - DAC_OVERRIDE
            drop:
            - ALL
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/eni
          name: eni-config
          readOnly: true
        - mountPath: /var/run/
          name: eni-run
        - mountPath: /lib/modules
          name: lib-modules
        - mountPath: /var/lib/cni/terway
          name: cni-terway
        - mountPath: /etc/cni/net.d
          name: cni
          readOnly: true
        - mountPath: /host-etc-net.d
          name: host-cni
        - mountPath: /var/lib/kubelet/device-plugins
          name: device-plugin-path
      - command:
        - /bin/policyinit.sh
        env:
        - name: NODENAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: DISABLE_POLICY
          valueFrom:
            configMapKeyRef:
              key: disable_network_policy
              name: eni-config
              optional: true
        - name: FELIX_TYPHAK8SSERVICENAME
          valueFrom:
            configMapKeyRef:
              key: felix_relay_service
              name: eni-config
              optional: true
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: CILIUM_K8S_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CILIUM_CNI_CHAINING_MODE
          value: terway-chainer
        - name: IN_CLUSTER_LOADBALANCE
          valueFrom:
            configMapKeyRef:
              key: in_cluster_loadbalance
              name: eni-config
              optional: true
        image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/terway:v1.15.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            host: 127.0.0.1
            port: 9099
          timeoutSeconds: 1
        name: policy
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            host: 127.0.0.1
            port: 9099
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 250m
            memory: 100Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - DAC_OVERRIDE
            - SYS_ADMIN
            - NET_RAW
            - SYS_MODULE
            - CHOWN
            - IPC_LOCK
            - SYS_RESOURCE
            drop:
            - ALL
          privileged: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/fs
          name: sys-fs
        - mountPath: /etc/eni
          name: eni-config
          readOnly: true
        - mountPath: /var-run-eni
          name: var-run-eni
          readOnly: true
        - mountPath: /lib/modules
          name: lib-modules
        - mountPath: /etc/cni/net.d
          name: cni
          readOnly: true
        - mountPath: /sys/fs/bpf
          mountPropagation: HostToContainer
          name: bpf-maps
        - mountPath: /var/run/cilium
          name: cilium-run
        - mountPath: /run/xtables.lock
          name: xtables-lock
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      hostNetwork: true
      initContainers:
      - command:
        - /bin/init.sh
        env:
        - name: TERWAY_DAEMON_MODE
          value: ENIMultiIP
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: DISABLE_POLICY
          valueFrom:
            configMapKeyRef:
              key: disable_network_policy
              name: eni-config
              optional: true
        image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/terway:v1.15.3
        imagePullPolicy: IfNotPresent
        name: terway-init
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /tmp/fs
          name: sys-fs
        - mountPath: /sys/fs/bpf
          mountPropagation: Bidirectional
          name: bpf-maps
        - mountPath: /etc/eni
          name: eni-config
        - mountPath: /var-run-eni
          name: var-run-eni
        - mountPath: /opt/cni/bin/
          name: cni-bin
        - mountPath: /etc/cni/net.d/
          name: cni
        - mountPath: /lib/modules
          name: lib-modules
        - mountPath: /host
          name: host-root
        - mountPath: /var/run/
          name: eni-run
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: terway
      serviceAccountName: terway
      terminationGracePeriodSeconds: 10
      tolerations:
      - operator: Exists
      volumes:
      - name: var-run-eni
      - configMap:
          defaultMode: 420
          name: eni-config
        name: eni-config
      - hostPath:
          path: /opt/cni/bin
          type: Directory
        name: cni-bin
      - hostPath:
          path: /etc/cni/net.d
          type: ""
        name: host-cni
      - name: cni
      - hostPath:
          path: /var/run/
          type: Directory
        name: eni-run
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /var/lib/cni/terway
          type: ""
        name: cni-terway
      - hostPath:
          path: /var/lib/kubelet/device-plugins
          type: Directory
        name: device-plugin-path
      - hostPath:
          path: /
          type: Directory
        name: host-root
      - hostPath:
          path: /var/run/cilium
          type: DirectoryOrCreate
        name: cilium-run
      - hostPath:
          path: /sys/fs/bpf
          type: DirectoryOrCreate
        name: bpf-maps
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
      - name: sys-fs
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 3%
    type: RollingUpdate

apiVersion: v1
kind: ConfigMap
metadata:
  name: eni-config
  namespace: kube-system
data:
  10-terway.conf: |
    {
      "cniVersion": "0.4.0",
      "name": "terway",
      "capabilities": {"bandwidth": true},
      "network_policy_provider": "ebpf",
      "type": "terway"
    }
  disable_network_policy: "true"
  eni_conf: |
    {
      "version": "1",
      "max_pool_size": 5,
      "min_pool_size": 0,


      "credential_path": "/var/addon/token-config",



      "enable_eni_trunking": true,
      "ipam_type": "crd",
      "vswitches": {"cn-hangzhou-b":["vsw-bp1hoez45sv94zkycvu29"],"cn-hangzhou-h":["vsw-bp15cvfngxdqwyue9845c"],"cn-hangzhou-i":["vsw-bp1w327bfwlwq94tts1jq"]},
      "eni_tags": {"ack.aliyun.com":"ca133aaf80fd542038acda778fbbf93a1"},
      "service_cidr": "192.168.0.0/16",
      "security_group": "sg-bp19ovaxk2551xdssn3q",

      "ip_stack": "ipv4",

      "resource_group_id": "rg-acfm23cvm4tmmjy",

      "vswitch_selection_policy": "ordered"
    }
  in_cluster_loadbalance: "true"

terway-eniip
eni-config

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: terway-eniip
  namespace: kube-system
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: terway-eniip
  template:
    metadata:
      annotations:
        k8s.alibabacloud.com/change-id: oifomoso
      labels:
        app: terway-eniip
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
                - arm64
              - key: kubernetes.io/os
                operator: In
                values:
                - linux
              - key: alibabacloud.com/nodepool-type
                operator: NotIn
                values:
                - hybridcloud
              - key: k8s.aliyun.com/ignore-by-terway
                operator: NotIn
                values:
                - "true"
      containers:
      - command:
        - /usr/bin/terwayd
        - -log-level
        - info
        - -daemon-mode
        - ENIMultiIP
        - -config
        - /etc/eni/eni_conf
        env:
        - name: KUBECONFIG
          value: /etc/terway/kubeconfig.conf
        - name: TERWAY_GC_RULES
          value: "true"
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/terway:v1.15.3
        imagePullPolicy: IfNotPresent
        name: terway
        resources:
          limits:
            cpu: 100m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - DAC_OVERRIDE
            drop:
            - ALL
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/eni
          name: eni-config
          readOnly: true
        - mountPath: /etc/terway
          name: terway-config
          readOnly: true
        - mountPath: /var/run/
          name: eni-run
        - mountPath: /lib/modules
          name: lib-modules
        - mountPath: /var/lib/cni/terway
          name: cni-terway
        - mountPath: /etc/cni/net.d
          name: cni
          readOnly: true
        - mountPath: /host-etc-net.d
          name: host-cni
        - mountPath: /var/lib/kubelet/device-plugins
          name: device-plugin-path
      - command:
        - /bin/policyinit.sh
        env:
        - name: KUBECONFIG
          value: /etc/terway/kubeconfig.conf
        - name: CILIUM_K8S_KUBECONFIG_PATH
          value: /etc/terway/kubeconfig.conf
        - name: NODENAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: DISABLE_POLICY
          valueFrom:
            configMapKeyRef:
              key: disable_network_policy
              name: eni-config
              optional: true
        - name: FELIX_TYPHAK8SSERVICENAME
          valueFrom:
            configMapKeyRef:
              key: felix_relay_service
              name: eni-config
              optional: true
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: CILIUM_K8S_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CILIUM_CNI_CHAINING_MODE
          value: terway-chainer
        - name: IN_CLUSTER_LOADBALANCE
          valueFrom:
            configMapKeyRef:
              key: in_cluster_loadbalance
              name: eni-config
              optional: true
        image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/terway:v1.15.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            host: 127.0.0.1
            port: 9099
          timeoutSeconds: 1
        name: policy
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            host: 127.0.0.1
            port: 9099
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 250m
            memory: 100Mi
        securityContext:
          capabilities:
            add:
            - NET_ADMIN
            - DAC_OVERRIDE
            - SYS_ADMIN
            - NET_RAW
            - SYS_MODULE
            - CHOWN
            - IPC_LOCK
            - SYS_RESOURCE
            drop:
            - ALL
          privileged: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/terway
          name: terway-config
          readOnly: true
        - mountPath: /tmp/fs
          name: sys-fs
        - mountPath: /etc/eni
          name: eni-config
          readOnly: true
        - mountPath: /var-run-eni
          name: var-run-eni
          readOnly: true
        - mountPath: /lib/modules
          name: lib-modules
        - mountPath: /etc/cni/net.d
          name: cni
          readOnly: true
        - mountPath: /sys/fs/bpf
          mountPropagation: HostToContainer
          name: bpf-maps
        - mountPath: /var/run/cilium
          name: cilium-run
        - mountPath: /run/xtables.lock
          name: xtables-lock
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      hostNetwork: true
      initContainers:
      - command:
        - /bin/init.sh
        - --feature-gates=KubeProxyReplacement=true
        env:
        - name: TERWAY_DAEMON_MODE
          value: ENIMultiIP
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: DISABLE_POLICY
          valueFrom:
            configMapKeyRef:
              key: disable_network_policy
              name: eni-config
              optional: true
        - name: KUBECONFIG
          value: /etc/terway/kubeconfig.conf
        image: registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/terway:v1.15.3
        imagePullPolicy: IfNotPresent
        name: terway-init
        securityContext:
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/terway
          name: terway-config
          readOnly: true
        - mountPath: /tmp/fs
          name: sys-fs
        - mountPath: /sys/fs/bpf
          mountPropagation: Bidirectional
          name: bpf-maps
        - mountPath: /etc/eni
          name: eni-config
        - mountPath: /var-run-eni
          name: var-run-eni
        - mountPath: /opt/cni/bin/
          name: cni-bin
        - mountPath: /etc/cni/net.d/
          name: cni
        - mountPath: /lib/modules
          name: lib-modules
        - mountPath: /host
          name: host-root
        - mountPath: /var/run/
          name: eni-run
      priorityClassName: system-node-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: terway
      serviceAccountName: terway
      terminationGracePeriodSeconds: 10
      tolerations:
      - operator: Exists
      volumes:
      - configMap:
          defaultMode: 420
          name: terway-config
        name: terway-config
      - name: var-run-eni
      - configMap:
          defaultMode: 420
          name: eni-config
        name: eni-config
      - hostPath:
          path: /opt/cni/bin
          type: Directory
        name: cni-bin
      - hostPath:
          path: /etc/cni/net.d
          type: ""
        name: host-cni
      - name: cni
      - hostPath:
          path: /var/run/
          type: Directory
        name: eni-run
      - hostPath:
          path: /lib/modules
          type: ""
        name: lib-modules
      - hostPath:
          path: /var/lib/cni/terway
          type: ""
        name: cni-terway
      - hostPath:
          path: /var/lib/kubelet/device-plugins
          type: Directory
        name: device-plugin-path
      - hostPath:
          path: /
          type: Directory
        name: host-root
      - hostPath:
          path: /var/run/cilium
          type: DirectoryOrCreate
        name: cilium-run
      - hostPath:
          path: /sys/fs/bpf
          type: DirectoryOrCreate
        name: bpf-maps
      - hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
        name: xtables-lock
      - name: sys-fs
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 3%
    type: RollingUpdate

apiVersion: v1
kind: ConfigMap
metadata:
  name: eni-config
  namespace: kube-system
data:
  10-terway.conf: |
    {
      "cniVersion": "0.4.0",
      "name": "terway",
      "capabilities": {"bandwidth": true},
      "network_policy_provider": "ebpf",
      "eniip_virtual_type": "datapathv2",
      "host_stack_cidrs": ["169.254.20.10/32"],
      "type": "terway"
    }
  disable_network_policy: "true"
  eni_conf: |
    {
      "version": "1",
      "max_pool_size": 5,
      "min_pool_size": 0,


      "credential_path": "/var/addon/token-config",



      "enable_eni_trunking": true,
      "ipam_type": "crd",
      "vswitches": {"cn-hangzhou-b":["vsw-bp1hoez45sv94zkycvu29"],"cn-hangzhou-h":["vsw-bp15cvfngxdqwyue9845c"],"cn-hangzhou-i":["vsw-bp1w327bfwlwq94tts1jq"]},
      "eni_tags": {"ack.aliyun.com":"cbd0e2f1cca304f6eb46b1485b78d573f"},
      "service_cidr": "192.168.0.0/16",
      "security_group": "sg-bp1hidwwqyg6nncqy2vo",

      "ip_stack": "ipv4",

      "resource_group_id": "rg-acfm23cvm4tmmjy",

      "vswitch_selection_policy": "ordered"
    }
  in_cluster_loadbalance: "true"

terway-eniip
eni-config

与只勾选 DataPath V2 时的 YAML 一致。

相比只勾选 DataPath V2，disable_network_policy 为 false：

apiVersion: v1
kind: ConfigMap
metadata:
  name: eni-config
  namespace: kube-system
data:
  10-terway.conf: |
    {
      "cniVersion": "0.4.0",
      "name": "terway",
      "capabilities": {"bandwidth": true},
      "network_policy_provider": "ebpf",
      "eniip_virtual_type": "datapathv2",
      "host_stack_cidrs": ["169.254.20.10/32"],
      "type": "terway"
    }
  disable_network_policy: "false"
  eni_conf: |
    {
      "version": "1",
      "max_pool_size": 5,
      "min_pool_size": 0,


      "credential_path": "/var/addon/token-config",



      "enable_eni_trunking": true,
      "ipam_type": "crd",
      "vswitches": {"cn-hangzhou-b":["vsw-bp1hoez45sv94zkycvu29"],"cn-hangzhou-h":["vsw-bp15cvfngxdqwyue9845c"],"cn-hangzhou-i":["vsw-bp1w327bfwlwq94tts1jq"]},
      "eni_tags": {"ack.aliyun.com":"c6a070d5c64f94f39bc3175c5a226477c"},
      "service_cidr": "192.168.0.0/16",
      "security_group": "sg-bp1gwurls5f0v0t2z81r",

      "ip_stack": "ipv4",

      "resource_group_id": "rg-acfm23cvm4tmmjy",

      "vswitch_selection_policy": "ordered"
    }
  in_cluster_loadbalance: "true"

CNI 配置

默认
勾选 DataPath V2 或 NetworkPolicy

/etc/cni/net.d/10-terway.conflist
{
  "cniVersion": "0.4.0",
  "name": "terway-chainer",
  "plugins": [
    {
      "bandwidth_mode": "tc",
      "capabilities": {
        "bandwidth": true
      },
      "eniip_virtual_type": "veth",
      "network_policy_provider": "ebpf",
      "type": "terway"
    }
  ]
}

/etc/cni/net.d/10-terway.conflist
{
  "cniVersion": "0.4.0",
  "name": "terway-chainer",
  "plugins": [
    {
      "bandwidth_mode": "edt",
      "capabilities": {
        "bandwidth": true
      },
      "eniip_virtual_type": "datapathv2",
      "host_stack_cidrs": [
        "169.254.20.10/32"
      ],
      "network_policy_provider": "ebpf",
      "type": "terway"
    },
    {
      "data-path": "datapathv2",
      "enable-debug": false,
      "log-file": "/var/run/cilium/cilium-cni.log",
      "type": "cilium-cni"
    }
  ]
}

CNI 二进制

$ ls /opt/cni/bin
bandwidth  bridge  cilium-cni  dhcp  dummy  firewall  host-device  host-local  ipvlan  LICENSE  loopback  macvlan  portmap  ptp  README.md  sbr  static  tap  terway  tuning  vlan  vrf

启动参数

默认
启用 DataPath V2
启用 NetworkPolicy

$ kubectl exec -i -t terway-eniip-kncv6 -c terway -- ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 07:33 ?        00:00:00 /usr/bin/terwayd -log-level info -daemon-mode ENIMultiIP -config /etc/eni/eni_conf
$ kubectl exec -i -t terway-eniip-kncv6 -c policy -- ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 08:45 ?        00:00:00 terway-cli policy

$ kubectl exec -i -t terway-eniip-kncv6 -c terway -- ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 07:33 ?        00:00:00 /usr/bin/terwayd -log-level info -daemon-mode ENIMultiIP -config /etc/eni/eni_conf
$ kubectl exec -i -t terway-eniip-kncv6 -c policy -- ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 07:33 ?        00:00:15 cilium-agent --routing-mode=native --cni-chaining-mode=terway-chainer --enable-ipv4-masquerade=false --enable-ipv6-masquerade=false --disable-envoy-version-check=true --local-router-ipv4=169.254.10.1 --local-router-ipv6=fe80:2400:3200:baba::1 --enable-local-node-route=false --enable-endpoint-health-checking=false --enable-health-checking=false --enable-service-topology=true --k8s-heartbeat-timeout=0 --enable-session-affinity=true --install-iptables-rules=false --enable-l7-proxy=false --ipam=delegated-plugin --enable-bandwidth-manager=true --agent-health-port=9099 --enable-policy=never --labels=k8s:io\.kubernetes\.pod\.namespace --datapath-mode=veth --kube-proxy-replacement=true --bpf-lb-sock=true --bpf-lb-sock-hostns-only=true --enable-node-port=true --enable-host-port=true --enable-external-ips=true --enable-endpoint-routes=true --enable-l2-neigh-discovery=false --enable-in-cluster-loadbalance=true --terway-host-stack-cidr=169.254.20.10/32

相比默认，policy 容器换成了 cilium-agent 进程启动。

$ kubectl exec -i -t terway-eniip-kncv6 -c terway -- ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 07:33 ?        00:00:00 /usr/bin/terwayd -log-level info -daemon-mode ENIMultiIP -config /etc/eni/eni_conf
$ kubectl exec -i -t terway-eniip-kncv6 -c policy -- ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 07:42 ?        00:00:08 cilium-agent --routing-mode=native --cni-chaining-mode=terway-chainer --enable-ipv4-masquerade=false --enable-ipv6-masquerade=false --disable-envoy-version-check=true --local-router-ipv4=169.254.10.1 --local-router-ipv6=fe80:2400:3200:baba::1 --enable-local-node-route=false --enable-endpoint-health-checking=false --enable-health-checking=false --enable-service-topology=true --k8s-heartbeat-timeout=0 --enable-session-affinity=true --install-iptables-rules=false --enable-l7-proxy=false --ipam=delegated-plugin --enable-bandwidth-manager=true --agent-health-port=9099 --enable-policy=default --datapath-mode=veth --kube-proxy-replacement=true --bpf-lb-sock=true --bpf-lb-sock-hostns-only=true --enable-node-port=true --enable-host-port=true --enable-external-ips=true --enable-endpoint-routes=true --enable-l2-neigh-discovery=false --enable-in-cluster-loadbalance=true --terway-host-stack-cidr=169.254.20.10/32

相比只启用 DataPath V2，policy 容器的 cilium-agent 的启动参数 --enable-policy=never 被修改为 --enable-policy=default。

网络实现分析

默认

网卡

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:2c:42:10 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 10.0.5.102/24 brd 10.0.5.255 scope global dynamic noprefixroute eth0
       valid_lft 1892159507sec preferred_lft 1892159507sec
    inet6 fe80::216:3eff:fe2c:4210/64 scope link
       valid_lft forever preferred_lft forever
3: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default
    link/ether 2e:da:ca:cc:de:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.60.84/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 192.168.0.1/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 192.168.188.57/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 192.168.0.10/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
    inet 192.168.155.221/32 scope global kube-ipvs0
       valid_lft forever preferred_lft forever
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:2c:7b:7a brd ff:ff:ff:ff:ff:ff
    altname enp0s8
    altname ens8
    inet 10.0.5.102/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe2c:7b7a/64 scope link
       valid_lft forever preferred_lft forever
5: caliaec797095c6@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 92:1a:36:d3:57:ed brd ff:ff:ff:ff:ff:ff link-netns cni-174631ed-0647-1e1b-5eef-6efabd9dd8b0
    inet6 fe80::901a:36ff:fed3:57ed/64 scope link
       valid_lft forever preferred_lft forever
$ ip -d link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:2c:42:10 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 4 numrxqueues 4 gso_max_size 65536 gso_max_segs 65535
    altname enp0s6
    altname ens6
3: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 2e:da:ca:cc:de:22 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 0
    dummy addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:2c:7b:7a brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 4 numrxqueues 4 gso_max_size 65536 gso_max_segs 65535
    altname enp0s8
    altname ens8
5: caliaec797095c6@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 92:1a:36:d3:57:ed brd ff:ff:ff:ff:ff:ff link-netns cni-174631ed-0647-1e1b-5eef-6efabd9dd8b0 promiscuity 0 minmtu 68 maxmtu 65535
    veth addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

Pod 使用 veth pair，节点一端的网卡名称以 cali 开头，网卡上没绑 IP 地址。

路由

$ ip rule list
0:      from all lookup local
512:    from all to 10.0.5.103 lookup main
2048:   from 10.0.5.103 lookup 1004
32766:  from all lookup main
32767:  from all lookup default
$ ip route show table 1004
default via 10.0.5.253 dev eth1 onlink
$ ip route show table main
default via 10.0.5.253 dev eth0 proto dhcp src 10.0.5.102 metric 100
10.0.5.0/24 dev eth0 proto kernel scope link src 10.0.5.102 metric 100
10.0.5.103 dev caliaec797095c6 scope link

从路由表可以看到，访问 Pod IP 的包会被路由到指定的 cali 开头的网卡，而该网卡真实 Pod 的 veth pair 在节点上的一端。

从阿里云控制台可以看到，该 Pod IP 通过中继网卡插到了节点上（ENI Trunking）:

所以 Pod IP 跟 ECS 之类的云上 IP 一样，可直接在 VPC 底层路由转发。

Pod 出流量则被策略路由转到 eth1 出去，eth1 应该就是中继网卡了，节点上所有中继网卡共用 eth1 出入流量。

容器内

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether ca:b1:c0:0b:15:52 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.4.72/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::c8b1:c0ff:fe0b:1552/64 scope link
       valid_lft forever preferred_lft forever
$ ip route
default via 169.254.1.1 dev eth0 onlink

DataPath V2

网卡

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:1c:40:85 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 10.0.0.238/24 brd 10.0.0.255 scope global dynamic noprefixroute eth0
       valid_lft 1892159457sec preferred_lft 1892159457sec
    inet6 fe80::216:3eff:fe1c:4085/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:16:3e:1c:7d:35 brd ff:ff:ff:ff:ff:ff
    altname enp0s8
    altname ens8
    inet 10.0.0.238/32 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe1c:7d35/64 scope link
       valid_lft forever preferred_lft forever
4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:d7:33:dd:e0:59 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f4d7:33ff:fedd:e059/64 scope link
       valid_lft forever preferred_lft forever
5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 12:ff:43:ea:2d:1f brd ff:ff:ff:ff:ff:ff
    inet 169.254.10.1/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80:2400:3200:baba::1/128 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::10ff:43ff:feea:2d1f/64 scope link
       valid_lft forever preferred_lft forever
6: calid7425eb8b46@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether aa:25:d0:ea:1d:d7 brd ff:ff:ff:ff:ff:ff link-netns cni-214e294f-d376-6c3e-a76b-100bca475796
    inet6 fe80::a825:d0ff:feea:1dd7/64 scope link
       valid_lft forever preferred_lft forever
7: cali8b8630acb2b@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3e:3a:38:81:12:93 brd ff:ff:ff:ff:ff:ff link-netns cni-42638ac7-385b-76b9-71fd-0524a49ea8b6
    inet6 fe80::3c3a:38ff:fe81:1293/64 scope link
       valid_lft forever preferred_lft forever
8: cali6dc30143901@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:aa:e3:71:3d:b8 brd ff:ff:ff:ff:ff:ff link-netns cni-da7d277e-2f5d-19fd-7ca2-69b0451abde2
    inet6 fe80::88aa:e3ff:fe71:3db8/64 scope link
       valid_lft forever preferred_lft forever
10: cali8797a3843fa@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e2:48:51:1e:bd:1e brd ff:ff:ff:ff:ff:ff link-netns cni-f4a5b1f7-3a77-cc86-e31b-ed6b465bde7e
    inet6 fe80::e048:51ff:fe1e:bd1e/64 scope link
       valid_lft forever preferred_lft forever
11: calic699fed89dc@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 36:fa:e2:69:b6:e5 brd ff:ff:ff:ff:ff:ff link-netns cni-c10f3468-bbe9-3ec4-64d5-7f096b9ce496
    inet6 fe80::34fa:e2ff:fe69:b6e5/64 scope link
       valid_lft forever preferred_lft forever
12: cali5869d48a1d1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 46:8b:55:a8:b7:b1 brd ff:ff:ff:ff:ff:ff link-netns cni-3f125d63-d8db-73e5-df5a-bf329c860d0a
    inet6 fe80::448b:55ff:fea8:b7b1/64 scope link
       valid_lft forever preferred_lft forever
13: cali1a0b76096c7@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether c6:f9:99:bd:0f:c2 brd ff:ff:ff:ff:ff:ff link-netns cni-b8e9de81-33f7-bdb1-657f-46612a0b9841
    inet6 fe80::c4f9:99ff:febd:fc2/64 scope link
       valid_lft forever preferred_lft forever
14: calicb82c9f0082@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f2:6f:f5:cf:26:02 brd ff:ff:ff:ff:ff:ff link-netns cni-9b439f79-5535-3622-118a-8d6325c8156b
    inet6 fe80::f06f:f5ff:fecf:2602/64 scope link
       valid_lft forever preferred_lft forever

路由

$ ip rule list
     from all lookup local
   from all to 10.0.0.239 lookup main
   from all to 10.0.0.240 lookup main
   from all to 10.0.0.241 lookup main
   from all to 10.0.0.242 lookup main
   from all to 10.0.0.247 lookup main
   from all to 10.0.0.245 lookup main
   from all to 10.0.0.244 lookup main
   from all to 10.0.0.246 lookup main
  from 10.0.0.239 lookup 1003
  from 10.0.0.240 lookup 1003
  from 10.0.0.241 lookup 1003
  from 10.0.0.242 lookup 1003
  from 10.0.0.247 lookup 1003
  from 10.0.0.245 lookup 1003
  from 10.0.0.244 lookup 1003
  from 10.0.0.246 lookup 1003
 from all lookup main
 from all lookup default
$ ip route show table 1003
default via 10.0.0.253 dev eth1 onlink
$ ip route show table main
default via 10.0.0.253 dev eth0 proto dhcp src 10.0.0.238 metric 100
0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.238 metric 100
0.0.239 dev calid7425eb8b46 proto kernel scope link
0.0.240 dev cali8b8630acb2b proto kernel scope link
0.0.241 dev cali6dc30143901 proto kernel scope link
0.0.242 dev cali8797a3843fa proto kernel scope link
0.0.244 dev cali1a0b76096c7 proto kernel scope link
0.0.245 dev cali5869d48a1d1 proto kernel scope link
0.0.246 dev calicb82c9f0082 proto kernel scope link
0.0.247 dev calic699fed89dc proto kernel scope link

容器内

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether aa:52:e5:ce:7b:df brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.0.4.63/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a852:e5ff:fece:7bdf/64 scope link
       valid_lft forever preferred_lft forever
$ ip route show
default via 169.254.1.1 dev eth0 onlink

Terway 插件的三种模式​

组件部署 YAML​

CNI 配置​

CNI 二进制​

启动参数​

网络实现分析​

默认​

网卡​

路由​

容器内​

DataPath V2​

网卡​

路由​