Skip to main content

Using EnvoyGateway Traffic Gateway on TKE

Overview​

EnvoyGateway is a Kubernetes gateway based on Envoy that implements the Gateway API. You can manage Kubernetes north-south traffic by defining resources like Gateway, HTTPRoute from the Gateway API specification.

This article describes how to install EnvoyGateway on TKE and use the Gateway API to manage traffic forwarding.

Note

Kubernetes provides the Ingress API to handle layer 7 north-south traffic, but its functionality is limited. Each implementation adds different annotations to enhance Ingress capabilities, resulting in poor flexibility and extensibility. Driven by community efforts, the Gateway API was introduced as a better solution that addresses Ingress API pain points while unifying layer 4/7 north-south traffic, and also supports service mesh east-west traffic (refer to GAMMA). Various cloud providers and open-source proxy software are actively adapting to the Gateway API. Refer to the Gateway API Implementations List, where Envoy Gateway is one of the popular implementations.

Using EnvoyGateway on TKE has a clear advantage over the built-in CLB Ingress: multiple forwarding rule resources (like HTTPRoute) can reuse the same CLB and can cross namespaces. CLB Ingress requires all forwarding rules to be written in the same Ingress resource, which is inconvenient to manage, and if different backend Services are in different namespaces, they cannot be managed with the same Ingress (CLB).

Prerequisites​

  1. Ensure helm and kubectl are installed, and kubeconfig is configured to connect to the cluster (refer to Connecting to Cluster).
  2. Ensure EnvoyGateway supports your current cluster version. Refer to EnvoyGateway official documentation Compatibility Matrix (current latest v1.5 requires cluster version >= 1.30).

Installing EnvoyGateway​

You can install via TKE Application Market or directly using helm. For users with hands-on experience, we recommend installing directly with helm to use the latest community version (TKE Application Market versions are usually updated promptly but not guaranteed to be the latest).

Method 1: Install using helm​

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.5.4 \
  -n envoy-gateway-system \
  --create-namespace

Refer to EnvoyGateway official documentation Install with Helm.

Method 2: Install via Application Market​

Search for envoygateway in the TKE Application Market or find it in the Network category. Click 【Create Application】, select namespace envoy-gateway-system, create it if it doesn't exist, complete other configurations, then click 【Create】 to install envoygateway into the cluster.

  1. Create a namespace named envoy-gateway-system in the target TKE cluster.
  2. Search for and select envoygateway in the TKE Application Market.
  3. Click Create Application, select the target TKE cluster, suggested name eg, namespace select envoy-gateway-system.
  4. Configure parameters according to requirements, then click Create to install Envoy Gateway into the cluster.

Basic Usage​

Create GatewayClass​

Similar to how Ingress needs to specify IngressClass, each Gateway in Gateway API needs to reference a GatewayClass. GatewayClass essentially represents the gateway instance configuration excluding listeners (such as deployment method, gateway Pod template, replica count, associated Service, etc.). So first create a GatewayClass:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

GatewayClass is a non-namespaced resource, no need to specify namespace.

Create Gateway​

Each Gateway corresponds to a CLB. Declaring ports on a Gateway is equivalent to creating corresponding protocol listeners on the CLB:

Note

All Gateway fields refer to API Specification: Gateway

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test-gw
  namespace: test
spec:
  gatewayClassName: eg
  listeners:
  - name: http
    protocol: HTTP
    port: 8080
    allowedRoutes:
      namespaces:
        from: All

Gateway can specify namespace and can be referenced cross-namespace by routing rules like HTTPRoute.

After Gateway creation, EnvoyGateway automatically creates a LoadBalancer-type Service for it, which is a CLB. On TKE, LoadBalancer-type Services default to public CLBs. To customize, refer to the FAQ section on How to Customize CLB.

Note

Gateway exposes traffic externally through LoadBalancer-type Services, so CLB only uses layer 4 listeners (TCP/UDP). Layer 7 traffic first enters CLB layer 4 listeners, gets forwarded to EnvoyGateway Pods, then EnvoyGateway parses layer 4/7 traffic and forwards according to configuration rules.

How to get the CLB address corresponding to Gateway? Use kubectl get gtw:

$ kubectl get gtw test-gw -n test
NAME      CLASS   ADDRESS         PROGRAMMED   AGE
test-gw   eg      139.155.64.52   True         358d

Where ADDRESS is the CLB address (IP or domain).

Create HTTPRoute​

HTTPRoute is used to define HTTP forwarding rules (layer 7 traffic) and is the most commonly used forwarding rule in Gateway API, similar to the Ingress resource in Ingress API.

Example below:

Note

All HTTPRoute fields refer to API Specification: HTTPRoute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: nginx
  namespace: test
spec:
  parentRefs:
  - name: test-gw
    namespace: test
  hostnames:
  - "test.example.com"
  rules:
  - backendRefs:
    - name: nginx
      port: 80
Note
  1. parentRefs specifies the Gateway (CLB) to reference, applying this rule to that Gateway.
  2. hostnames defines the domains used by the forwarding rules. Ensure these domains resolve to the Gateway's corresponding CLB so you can access cluster services via domain names.
  3. backendRefs defines the backend Service corresponding to this forwarding rule.

Create TCPRoute and UDPRoute​

TCPRoute and UDPRoute are used to define TCP and UDP forwarding rules (layer 4 traffic), similar to LoadBalancer-type Service.

Note

All TCPRoute and UDPRoute fields refer to API Specification: TCPRoute and API Specification: UDPRoute.

First ensure Gateway has defined TCP and UDP ports:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test-gw
  namespace: test
spec:
  gatewayClassName: eg
  listeners:
  - name: foo
    protocol: TCP
    port: 6000
    allowedRoutes:
      namespaces:
        from: All
  - name: bar
    protocol: UDP
    port: 6000
    allowedRoutes:
      namespaces:
        from: All

Then TCPRoute and UDPRoute can reference this Gateway:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: foo
spec:
  parentRefs:
  - namespace: test
    name: test-gw
    sectionName: foo
  rules:
  - backendRefs:
    - name: foo
      port: 6000
Note
  1. parentRefs specifies the Gateway (CLB) to reference, indicating this TCP should listen on this Gateway. Usually only one port from the Gateway is used, so specify sectionName to indicate which listener to expose through.
  2. backendRefs defines the backend Service corresponding to this forwarding rule.

Usage Examples​

Customize CLB​

You can customize by creating EnvoyProxy custom resources. Example below:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 1
        container:
          resources:
            requests:
              tke.cloud.tencent.com/eni-ip: "1"
            limits:
              tke.cloud.tencent.com/eni-ip: "1"
        pod:
          annotations:
            tke.cloud.tencent.com/networks: tke-route-eni
      envoyService:
        annotations:
          service.kubernetes.io/tke-existed-lbid: lb-5nhlk3nr
          service.cloud.tencent.com/direct-access: "true"

In the above example:

  • Explicitly declare using VPC-CNI network mode and enable CLB direct-to-pod.
  • Use existing CLB, specifying CLB ID.

Correspondingly, Gateway needs to reference this EnvoyProxy configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: web
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: proxy-config
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    tls:
      certificateRefs:
      - kind: Secret
        group: ""
        name: website-crt-secret
    allowedRoutes:
      namespaces:
        from: All
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All

For more CLB-related customization, refer to Service Annotation Description.

Some common customization examples:

  1. Use service.cloud.tencent.com/specify-protocol annotation to modify listener protocol to HTTPS and correctly reference SSL certificates to enable CLB integration with Tencent Cloud WAF.
  2. Use service.kubernetes.io/qcloud-loadbalancer-internal-subnetid annotation to specify CLB private IP, enabling automatic creation of private CLB for traffic access.
  3. Use service.kubernetes.io/service.extensiveParameters annotation to customize more properties of automatically created CLBs, such as specifying carriers, bandwidth limits, instance specifications, network billing modes, etc.

Multiple HTTPRoutes Reusing Same CLB​

Usually one Gateway object corresponds to one CLB. As long as different HTTPRoutes reference the same Gateway object in their parentRefs, they will reuse the same CLB.

Note

If multiple HTTPRoutes reuse the same CLB, ensure their defined HTTP rules don't conflict, otherwise forwarding behavior may not match expectations.

Example below. First HTTPRoute, referencing Gateway test-gw, using domain test1.example.com:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test1
  namespace: test
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: test-gw
      namespace: test
  hostnames:
    - "test1.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: test1
          port: 80

Second HTTPRoute, also referencing Gateway test-gw, using domain test2.example.com:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test2
  namespace: test
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: test-gw
      namespace: test
  hostnames:
    - "test2.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: test2
          port: 80

Layer 4 and Layer 7 Sharing Same CLB​

Using TKE's built-in LoadBalancer-type Service, you can achieve multiple Services reusing the same CLB, meaning multiple layer 4 ports (TCP/UDP) reuse the same CLB. Using TKE's built-in Ingress (CLB Ingress), you cannot reuse the same CLB with any other Ingress or LoadBalancer-type Service. So if you need to achieve layer 4 and layer 7 sharing the same CLB, it's impossible with TKE's built-in CLB Service and CLB Ingress, but if you install EnvoyGateway, you can achieve it.

Example below. First Gateway listener declares layer 4 and layer 7 ports:

Note

Use name to give each listener a name for easy reference via sectionName later.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test-gw
  namespace: test
spec:
  gatewayClassName: eg
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All
  - name: https
    protocol: HTTPS
    port: 443
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        group: ""
        name: https-cert
  - name: tcp-6000
    protocol: TCP
    port: 6000
    allowedRoutes:
      namespaces:
        from: All
  - name: udp-6000
    protocol: UDP
    port: 6000
    allowedRoutes:
      namespaces:
        from: All

HTTPRoute uses layer 7 listeners (80 and 443) from Gateway:

Note

Use sectionName to specify the exact listener to bind to.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test
  namespace: test
spec:
  parentRefs:
  - name: test-gw
    namespace: test
    sectionName: http
  - name: test-gw
    namespace: test
    sectionName: https
  hostnames:
  - "test.example.com"
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: nginx
      port: 80

TCPRoute and UDPRoute use layer 4 listeners (TCP/6000 and UDP/6000) from Gateway:

Note

Same as HTTPRoute, use sectionName to specify the exact listener to bind to.

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: foo
spec:
  parentRefs:
  - namespace: test
    name: test-gw
    sectionName: tcp-6000
  rules:
  - backendRefs:
    - name: foo
      port: 6000
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
  name: foo
spec:
  parentRefs:
  - namespace: test
    name: test-gw
    sectionName: udp-6000
  rules:
  - backendRefs:
    - name: foo
      port: 6000

Automatic Redirect​

Configure HTTPRoute's filters to achieve automatic redirects. Examples below.

Replace path prefix /api/v1 with /apis/v1:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: redirect-api-v1
  namespace: test
spec:
  hostnames:
  - test.example.com
  parentRefs:
  - name: test-gw
    namespace: test
    sectionName: https
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api/v1
    filters:
    - type: RequestRedirect
      requestRedirect:
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /apis/v1
        statusCode: 301

http://test.example.com/api/v1/pods will be redirected to http://test.example.com/apis/v1/pods

Redirect all paths starting with /foo to /bar:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: redirect-api-v1
  namespace: test
spec:
  hostnames:
  - test.example.com
  parentRefs:
  - name: test-gw
    namespace: test
    sectionName: https
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /foo
    filters:
    - type: RequestRedirect
      requestRedirect:
        path:
          type: ReplaceFullPath
          replaceFullPath: /bar
        statusCode: 301

https://test.example.com/foo/cayenne and https://test.example.com/foo/paprika will both be redirected to https://test.example.com/bar

HTTP to HTTPS redirect:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: redirect-https
  namespace: test
spec:
  hostnames:
  - test.example.com
  parentRefs:
  - name: test-gw
    namespace: test
    sectionName: http
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /
    filters:
    - type: RequestRedirect
      requestRedirect:
        scheme: https
        statusCode: 301
        port: 443

http://test.example.com/foo will be redirected to https://test.example.com/foo

Configure HTTPS and TLS​

Store certificates and keys in Kubernetes Secrets:

Note

If you don't want to manually manage certificates and prefer automatic issuance, consider using cert-manager for automatic certificate issuance. Refer to Using cert-manager to Issue Free Certificates for dnspod Domains.

apiVersion: v1
kind: Secret
metadata:
  name: test-cert
  namespace: test
type: kubernetes.io/tls
data:
  tls.crt: ***
  tls.key: ***

Configure TLS in Gateway listeners (HTTPS or TLS protocol), reference certificate Secret in tls field:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test-gw
  namespace: test
spec:
  gatewayClassName: eg
  listeners:
  - name: https
    protocol: HTTPS
    port: 443
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        group: ""
        name: test-cert
  - name: tls
    protocol: TLS
    port: 9443
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - kind: Secret
        group: ""
        name: test-cert

Modify HTTP Headers​

Use the RequestHeaderModifier filter in HTTPRoute to modify HTTP request headers.

Examples of modifying request headers:

Note

Modify headers for requests with paths starting with /foo.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  namespace: test
  name: foo-add-header
spec:
  hostnames:
  - test.example.com
  parentRefs:
  - name: test-gw
    namespace: test
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /foo
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
        add:
        - name: my-header-name
          value: my-header-value
    backendRefs:
    - name: foo
      port: 8080

Modifying response headers is similar, just change RequestHeaderModifier to ResponseHeaderModifier:

    filters:
    - type: ResponseHeaderModifier
      responseHeaderModifier:
        add:
        - name: X-Header-Add-1
          value: header-add-1
        - name: X-Header-Add-2
          value: header-add-2
        - name: X-Header-Add-3
          value: header-add-3

Expose API Server​

Refer to the following configuration:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 3 # Specify the number of EnvoyGateway gateway Pods
      envoyService:
        annotations:
          service.cloud.tencent.com/direct-access: "true" # Enable CLB direct-to-EnvoyGateway gateway Pod

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: envoy-gateway-system
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: eg
  listeners: # Specify CLB external listening ports
  - name: apiserver
    protocol: TCP
    port: 443
    allowedRoutes:
      namespaces:
        from: All

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
  name: apiserver
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
    namespace: envoy-gateway-system
    sectionName: apiserver # Reference Gateway
  rules:
  - backendRefs: # Backend points to default/kubernetes apiserver service
    - group: ""
      kind: Service
      name: kubernetes
      port: 443
      weight: 1

Explore More Usage​

Gateway API is very powerful and can implement many complex functions, such as routing based on weights, headers, cookies; canary releases; traffic mirroring; URL redirects and rewrites; TLS routing; GRPC routing, etc. For more detailed usage, refer to Gateway API Official Documentation.

EnvoyGateway also supports some advanced capabilities beyond Gateway API. Refer to EnvoyGateway Official Documentation.