Extensibility

• 1: Build a Wasm image
• 2: Dynamic Modules
• 3: Envoy Patch Policy
• 4: Envoy Gateway Extension Server
• 5: External Processing
• 6: Lua Extensions
• 7: OPA Sidecar with Unix Domain Socket
• 8: Remote Infrastructure Provider
• 9: Wasm Extensions

1 - Build a Wasm image

Envoy Gateway supports two types of Wasm extensions within the EnvoyExtensionPolicy API: HTTP Wasm Extensions and Image Wasm Extensions. Packaging a Wasm extension as an OCI image is beneficial because it simplifies versioning and distribution for users. Additionally, users can leverage existing image toolchain to build and manage Wasm images.

This document describes how to build OCI images which are consumable by Envoy Gateway.

Wasm Image Formats

There are two types of images that are supported by Envoy Gateway. One is in the Docker format, and another is the standard OCI specification compliant format. Please note that both of them are supported by any OCI registries. You can choose either format depending on your preference, and both types of images are consumable by Envoy Gateway EnvoyExtensionPolicy API.

Build Wasm Docker image

We assume that you have a valid Wasm binary named plugin.wasm. Then you can build a Wasm Docker image with the Docker CLI.

1. First, we prepare the following Dockerfile:

$ cat Dockerfile
FROM scratch

COPY plugin.wasm ./

Note: you must have exactly one COPY instruction in the Dockerfile in order to end up having only one layer in produced images.

2. Then, build your image via docker build command

$ docker build . -t my-registry/mywasm:0.1.0

3. Finally, push the image to your registry via docker push command

$ docker push my-registry/mywasm:0.1.0

Build Wasm OCI image

We assume that you have a valid Wasm binary named plugin.wasm, and you have buildah installed on your machine. Then you can build a Wasm OCI image with the buildah CLI.

1. First, we create a working container from scratch base image with buildah from command.

$ buildah --name mywasm from scratch
mywasm

2. Then copy the Wasm binary into that base image by buildah copy command to create the layer.

$ buildah copy mywasm plugin.wasm ./
af82a227630327c24026d7c6d3057c3d5478b14426b74c547df011ca5f23d271

Note: you must execute buildah copy exactly once in order to end up having only one layer in produced images

4. Now, you can build an OCI image and push it to your registry via buildah commit command

$ buildah commit mywasm docker://my-remote-registry/mywasm:0.1.0

2 - Dynamic Modules

This task provides instructions for configuring Dynamic Modules.

Dynamic Modules are a critical extension mechanism that allows functionality to be loaded directly into the Envoy proxy process at runtime, typically as shared object files (.so). This approach enables customized filtering and request processing without requiring a full recompile of the core Envoy binary, streamlining deployments and upgrades.

Envoy Gateway is able to load dynamic modules from the local filesystem using EnvoyExtensionPolicy. This example demonstrates it’s working by loading the Coraza Web Application Firewall (WAF) using Built On Envoy development toolkit. The module’s full documentation can be found on the Coraza WAF extension page.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Coraza WAF Extension

Installation

Add the dynamic module to the Envoy proxy container’s filesystem and configure the DynamicModules spec to load it into Envoy.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: my-proxy
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          volumes:
            - name: dynamic-modules
              image:
                reference: ghcr.io/tetratelabs/built-on-envoy/composer:0.6.0-dev
                pullPolicy: IfNotPresent
        container:
          env:
            - name: GODEBUG
              value: "cgocheck=0"
          volumeMounts:
            - name: dynamic-modules
              mountPath: /etc/envoy/dynamic-modules
              readOnly: true
  dynamicModules:
    - name: composer
      source: 
        type: Local
        local:
          path: /etc/envoy/dynamic-modules/libcomposer.so
      doNotClose: true
      loadGlobally: false
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: my-proxy
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          volumes:
            - name: dynamic-modules
              image:
                reference: ghcr.io/tetratelabs/built-on-envoy/composer:0.6.0-dev
                pullPolicy: IfNotPresent
        container:
          env:
            - name: GODEBUG
              value: "cgocheck=0"
          volumeMounts:
            - name: dynamic-modules
              mountPath: /etc/envoy/dynamic-modules
              readOnly: true
  dynamicModules:
    - name: composer
      source: 
        type: Local
        local:
          path: /etc/envoy/dynamic-modules/libcomposer.so
      doNotClose: true
      loadGlobally: false

Note: verify version compatibility of the extension because of dynamic module forward compatibility.

Image Volumes are relatively new and only supported from Kubernetes 1.35 onwards. Alternative ways of loading the dynamic module are:

• Building a custom docker image
• Copying from InitContainer to a shared volume

Verify the EnvoyProxy status:

kubectl get envoyproxy/my-proxy -o yaml

Attach to Gateway via a GatewayClass with spec.parametersRef

kubectl patch gatewayclass eg --type=merge -p '{
  "spec": {
    "parametersRef": {
      "group": "gateway.envoyproxy.io",
      "kind": "EnvoyProxy",
      "name": "my-proxy",
      "namespace": "envoy-gateway-system"
    }
  }
}'

The entire configuration can also be specified directly on the Gateway instead by using spec.envoyProxy.

Configuration

Create a new EnvoyExtensionPolicy resource to configure the dynamic module for an entire Gateway or per HTTPRoute.

This EnvoyExtensionPolicy targets the Gateway “eg” created with the quickstart. It loads the Coraza WAF extension with its configuration.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: waf-extension
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  dynamicModule:
    - name: composer
      filterName: coraza-waf
      config:
        directives:
          - Include @coraza.conf
          - SecRuleEngine On
          - SecResponseBodyAccess Off
          - Include @crs-setup.conf
          - Include @owasp_crs/*.conf
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: waf-extension
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway # or e.g. HTTPRoute
      name: eg

  dynamicModule:
    - name: composer
      filterName: coraza-waf
      config:
        directives:
          - Include @coraza.conf
          - SecRuleEngine On
          - SecResponseBodyAccess Off
          - Include @crs-setup.conf
          - Include @owasp_crs/*.conf

Verify the Envoy Extension Policy configuration:

kubectl get envoyextensionpolicy/waf-extension -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a normal request to the backend service:

curl -v -H "Host: www.example.com" "http://${GATEWAY_HOST}/"

You should get a 200 OK response from the backend.

Now send a request with a SQL injection payload to trigger the WAF:

curl -v -H "Host: www.example.com" "http://${GATEWAY_HOST}/?id=1'+OR+'1'%3D'1"

The Coraza WAF should block the request and return a 403 Forbidden response:

> GET /?id=1'+OR+'1'='1 HTTP/1.1
> Host: www.example.com
[...]
< HTTP/1.1 403 Forbidden
< date: Sat, 03 May 2026 12:00:00 GMT
< content-length: 0
<

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the EnvoyProxy and EnvoyExtensionPolicy:

kubectl delete envoyproxy/my-proxy
kubectl delete envoyextensionpolicy/waf-extension

Next Steps

Checkout the Developer Guide to get involved in the project.

3 - Envoy Patch Policy

This task explains the usage of the EnvoyPatchPolicy API. Note: This API is meant for users extremely familiar with Envoy xDS semantics. Also before considering this API for production use cases, please be aware that this API is unstable and the outcome may change across versions. Use at your own risk.

Introduction

The EnvoyPatchPolicy API allows user to modify the output xDS configuration generated by Envoy Gateway intended for EnvoyProxy, using JSON Patch semantics.

Motivation

This API was introduced to allow advanced users to be able to leverage Envoy Proxy functionality not exposed by Envoy Gateway APIs today.

Security Warning

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Enable EnvoyPatchPolicy

• By default EnvoyPatchPolicy is disabled. Lets enable it in the EnvoyGateway startup configuration

• The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it using a ConfigMap. In the next step, we will update this resource to enable EnvoyPatchPolicy.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true

After updating the ConfigMap, you will need to wait the configuration kicks in.
You can force the configuration to be reloaded by restarting the envoy-gateway deployment.

kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system

XDS Name Scheme V2

Starting from v1.5, Envoy Gateway uses version 2 of the xDS name scheme when generating xDS resources. Because EnvoyPatchPolicy relies on specific xDS resource names, it’s important to use the correct naming format when authoring a patch policy.

This change is gated by the XDSNameSchemeV2 runtime flag. The flag is disabled by default in v1.5 and will be enabled by default starting in v1.10.

We recommend users begin migrating their EnvoyPatchPolicy resources to use the version 2 naming scheme before upgrading to v1.10.

To opt in to the new naming scheme early, add theXDSNameSchemeV2 runtime flag to the runtimeFlags.enabled field in your EnvoyGateway configuration.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
    runtimeFlags:
      enabled:
      - XDSNameSchemeV2
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
    runtimeFlags:
      enabled:
      - XDSNameSchemeV2

Testing

Customize Response

• Use EnvoyProxy’s Local Reply Modification feature to return a custom response back to the client when the status code is 404

• Apply the configuration

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"

When mergeGateways is enabled, there will be one Envoy deployment for all Gateways in the cluster. Then the EnvoyPatchPolicy should target a specific GatewayClass.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"

• Edit the HTTPRoute resource from the Quickstart to only match on paths with value /get

kubectl patch httproute backend --type=json --patch '
  - op: add
    path: /spec/rules/0/matches/0/path/value
    value: /get
  '

• Test it out by specifying a path apart from /get

$ curl --header "Host: www.example.com" http://$GATEWAY_HOST/find
Handling connection for 8888
could not find what you are looking for

Customize VirtualHost by name

• Use EnvoyProxy’s include_attempt_count_in_response feature to include the attempt count as header in the downstream response.
• Apply the configuration

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: include-attempts
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
      # The RouteConfiguration name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        # Every virtual_host that ends with 'www_example_com' (using RegEx Filter)
        jsonPath: "..virtual_hosts[?match(@.name, '.*www_example_com')]"
        # If the property does not exists, it can not be selected with jsonPath
        # Therefore the new property must be set in path
        path: "include_attempt_count_in_response"
        value: true
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: include-attempts
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
      # The RouteConfiguration name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        # Every virtual_host that ends with 'www_example_com' (using RegEx Filter)
        jsonPath: "..virtual_hosts[?match(@.name, '.*www_example_com')]"
        # If the property does not exists, it can not be selected with jsonPath
        # Therefore the new property must be set in path
        path: "include_attempt_count_in_response"
        value: true

• Test it out by looking at the response headers

$ curl -v --header "Host: www.example.com" http://localhost:8888/
...
< x-envoy-attempt-count: 1
...

Debugging

Runtime

• The Status subresource should have information about the status of the resource. Make sure Accepted=True and Programmed=True conditions are set to ensure that the policy has been applied to Envoy Proxy.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyPatchPolicy","metadata":{"annotations":{},"name":"custom-response-patch-policy","namespace":"default"},"spec":{"jsonPatches":[{"name":"default/eg/http","operation":{"op":"add","path":"/default_filter_chain/filters/0/typed_config/local_reply_config","value":{"mappers":[{"body":{"inline_string":"could not find what you are looking for"},"filter":{"status_code_filter":{"comparison":{"op":"EQ","value":{"default_value":404}}}}}]}},"type":"type.googleapis.com/envoy.config.listener.v3.Listener"}],"priority":0,"targetRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"eg","namespace":"default"},"type":"JSONPatch"}}
  creationTimestamp: "2023-07-31T21:47:53Z"
  generation: 1
  name: custom-response-patch-policy
  namespace: default
  resourceVersion: "10265"
  uid: a35bda6e-a0cc-46d7-a63a-cee765174bc3
spec:
  jsonPatches:
  - name: default/eg/http
    operation:
      op: add
      path: /default_filter_chain/filters/0/typed_config/local_reply_config
      value:
        mappers:
        - body:
            inline_string: could not find what you are looking for
          filter:
            status_code_filter:
              comparison:
                op: EQ
                value:
                  default_value: 404
    type: type.googleapis.com/envoy.config.listener.v3.Listener
  priority: 0
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
status:
  conditions:
  - lastTransitionTime: "2023-07-31T21:48:19Z"
    message: EnvoyPatchPolicy has been accepted.
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2023-07-31T21:48:19Z"
    message: successfully applied patches.
    reason: Programmed
    status: "True"
    type: Programmed

Offline

• You can use egctl x translate to validate the translated xds output.

Caveats

This API will always be an unstable API and the same outcome cannot be guaranteed across versions for these reasons

• The Envoy Proxy API might deprecate and remove API fields
• Envoy Gateway might alter the xDS translation creating a different xDS output such as changing the name field of resources.

4 - Envoy Gateway Extension Server

This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.

Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.

Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.

Introduction

One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.

Security Warning

Extension Hooks Overview

Envoy Gateway provides several extension hooks that are called at different stages of the xDS translation process. These hooks allow extensions to modify various aspects of the generated xDS configuration:

Available Hooks

Hook Execution Order

The hooks are executed in the following order during xDS translation:

1. Route Processing Phase

   • Route hook (PostRouteModifyHook): Called for each route that has extension filters attached
   • Cluster hook (PostClusterModifyHook): Called for each cluster generated from custom backend references

2. Virtual Host Processing Phase

   • VirtualHost hook (PostVirtualHostModifyHook): Called for each virtual host after all routes are processed

3. Listener Processing Phase

   • HTTPListener hook (PostHTTPListenerModifyHook): Called for each HTTP listener after virtual hosts are configured

4. Final Translation Phase

   • Translation hook (PostTranslateModifyHook): Called once with all generated clusters, secrets, listeners, and routes

Hook Details

Route Hook (PostRouteModifyHook)

• When called: After each individual route is generated from an HTTPRoute with extension filters
• Input: Single route, hostnames, and extension resources from the HTTPRoute
• Output: Modified route
• Use cases: Add route-specific filters, modify route configuration, add typed per-filter config

Cluster Hook (PostClusterModifyHook)

• When called: During cluster generation for custom backend references only
• Input: Single cluster and extension resources from the backend reference
• Output: Modified cluster
• Use cases: Configure custom backend clusters, add health checks, modify load balancing

VirtualHost Hook (PostVirtualHostModifyHook)

• When called: After all routes for a virtual host are processed
• Input: Complete virtual host with all its routes
• Output: Modified virtual host (can add/remove/modify routes)
• Use cases: Add virtual host-level configuration, inject additional routes

HTTPListener Hook (PostHTTPListenerModifyHook)

• When called: After listener is fully configured with all virtual hosts
• Input: Complete HTTP listener and associated extension policies
• Output: Modified listener
• Use cases: Add listener filters, modify listener configuration, add authentication

Translation Hook (PostTranslateModifyHook)

• When called: After all individual resources are generated and processed
• Input: All clusters, secrets, listeners, and routes, plus extension policies
• Output: Complete set of modified resources
• Use cases: Global resource injection, cross-resource modifications, cleanup operations

Configuration

Extensions must specify which hooks they want to use in the extensionManager.hooks configuration:

extensionManager:
  hooks:
    xdsTranslator:
      post:
        - Route          # Enable route modification hook
        - VirtualHost    # Enable virtual host modification hook
        - HTTPListener   # Enable HTTP listener modification hook
        - Cluster        # Enable cluster modification hook
        - Translation    # Enable global translation hook
      # Configure which resources to include in PostTranslateModifyHook
      # Default: true for clusters and secrets (for backward compatibility)
      # Default: false for listeners and routes (for backward compatibility)
      translation:
        listener:
          includeAll: true
        route:
          includeAll: true

This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.

Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Build and run the example Extension Server

Build and deploy the example extension server in the examples/extension-server folder into the cluster running Envoy Gateway.

• Build the extension server image

  Note: The provided Makefile builds an image with the name extension-server:latest. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.

  make image
  

• Publish the extension server image in your docker repository

  • local kind server
  • other Kubernetes server

  kind load docker-image --name envoy-gateway extension-server:latest
  

  docker tag extension-server:latest $YOUR_DOCKER_REPO
  docker push $YOUR_DOCKER_REPO
  

• Deploy the extension server in your cluster

  If you are using your own docker image repository, make sure to update the values.yaml with the correct image name and tag.

  helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
  

Configure Envoy Gateway

• Grant Envoy Gateway’s ServiceAccount permission to access the extension server’s CRD

  kubectl create clusterrole listener-context-example-status-update \
             --verb=update  \
             --resource=ListenerContextExample/status
  
  kubectl create clusterrole listener-context-example-viewer \
             --verb=get,list,watch  \
             --resource=ListenerContextExample
  
  kubectl create clusterrolebinding envoy-gateway-listener-context \
             --clusterrole=listener-context-example-viewer \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  
  kubectl create clusterrolebinding envoy-gateway-listener-context-status \
             --clusterrole=listener-context-example-status-update \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  

• Configure Envoy Gateway to use the Extension Server

  Add the following fragment to Envoy Gateway’s configmap:

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: envoy-gateway-config
    namespace: envoy-gateway-system
  data:
    envoy-gateway.yaml: |
      apiVersion: gateway.envoyproxy.io/v1alpha1
      kind: EnvoyGateway
      provider:
        type: Kubernetes
      gateway:
        controllerName: gateway.envoyproxy.io/gatewayclass-controller
      extensionManager:
        # Envoy Gateway will watch these resource kinds and use them as extension policies
        # which can be attached to Gateway resources.
        policyResources:
        - group: example.extensions.io
          version: v1alpha1
          kind: ListenerContextExample
        hooks:
          # The type of hooks that should be invoked
          xdsTranslator:
            post:
            - HTTPListener
        service:
          # The service that is hosting the extension server
          fqdn:
            hostname: extension-server.envoy-gateway-system.svc.cluster.local
            port: 5005
  EOF
  

  After updating Envoy Gateway’s configmap, restart Envoy Gateway.

Multiple extension servers

Envoy Gateway also supports chaining multiple extension servers. Use the extensionManagers field (plural) instead of extensionManager:

extensionManagers:
- name: ext-a
  resources:
  - group: a.example.io
    version: v1
    kind: FooFilter
  hooks:
    xdsTranslator:
      post:
      - Route
  service:
    fqdn:
      hostname: ext-server-a.envoy-gateway-system.svc.cluster.local
      port: 5005
- name: ext-b
  resources:
  - group: b.example.io
    version: v1
    kind: BarFilter
  hooks:
    xdsTranslator:
      post:
      - Route
  service:
    fqdn:
      hostname: ext-server-b.envoy-gateway-system.svc.cluster.local
      port: 5005

A few nuances apply when multiple extension managers are configured:

• Sequential chaining. Hooks are invoked in the order the servers are declared. Each server sees the output of the previous one, so mutations compose.
• Resource isolation. Each server only receives the extension resources whose GVK it declares under resources, policyResources, or backendResources. In the example above a FooFilter is sent to ext-a but not to ext-b.
• failOpen is per server. Setting failOpen: true on a server causes Envoy Gateway to skip that server in the chain when its hook returns an error (or when its gRPC connection cannot be established), while still invoking the remaining servers.
• Unique name required. Every entry in extensionManagers must have a distinct name. Names surface in error messages as extension "<name>": <error>.
• Mutually exclusive with the singular form. extensionManager and extensionManagers cannot both be set.

Testing

Get the Gateway’s address:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:

$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
< 
...
User authentication failed. Missing username and password.
...

Add a new Username/Password combination using the example extension server’s CRD:

kubectl apply -f - << EOF 
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
  name: listeneruser
spec:
  targetRefs:
  - kind: Gateway
    name: eg
    group: gateway.networking.k8s.io
  username: user
  password: p@ssw0rd
EOF

Authenticating with this user/password combination will now work.

$ curl -v http://${GATEWAY_HOST}/example  -H "Host: www.example.com"   --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
< 
...
 "headers": {
  "Authorization": [
   "Basic dXNlcm5hbWU6cEBzc3cwcmQ="
  ],
  "X-Example-Ext": [
   "user"
  ],
...

5 - External Processing

This task provides instructions for configuring external processing.

External processing calls an external gRPC service to process HTTP requests and responses. The external processing service can inspect and mutate requests and responses.

Envoy Gateway introduces a new CRD called EnvoyExtensionPolicy that allows the user to configure external processing. This instantiated resource can be linked to a Gateway and HTTPRoute resource.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

GRPC External Processing Service

Installation

Install a demo GRPC service that will be used as the external processing service:

kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-proc-grpc-service.yaml

Create a new HTTPRoute resource to route traffic on the path /myapp to the backend service.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
  - name: eg
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /myapp
    backendRefs:
    - name: backend
      port: 3000   
EOF

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
  - name: eg
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /myapp
    backendRefs:
    - name: backend
      port: 3000   

Verify the HTTPRoute status:

kubectl get httproute/myapp -o yaml

Configuration

Create a new EnvoyExtensionPolicy resource to configure the external processing service. This EnvoyExtensionPolicy targets the HTTPRoute “myApp” created in the previous step. It calls the GRPC external processing service “grpc-ext-proc” on port 9002 for processing.

By default, requests and responses are not sent to the external processor. The processingMode struct is used to define what should be sent to the external processor. In this example, we configure the following processing modes:

• The empty request field configures envoy to send request headers to the external processor.
• The response field includes configuration for body processing. As a result, response headers are sent to the external processor. Additionally, the response body is streamed to the external processor.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: ext-proc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: myapp
  extProc:
  - backendRefs:
    - name: grpc-ext-proc
      port: 9002
    processingMode:
      request: {}
      response: 
        body: Streamed 
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: ext-proc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: myapp
  extProc:
    - backendRefs:
        - name: grpc-ext-proc
          port: 9002
      processingMode:
        request: {}
        response: 
          body: Streamed

Verify the Envoy Extension Policy configuration:

kubectl get envoyextensionpolicy/ext-proc-example -o yaml

Because the gRPC external processing service is enabled with TLS, a BackendTLSPolicy needs to be created to configure the communication between the Envoy proxy and the gRPC auth service.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: grpc-ext-proc-btls
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: grpc-ext-proc
  validation:
    caCertificateRefs:
    - name: grpc-ext-proc-ca
      group: ''
      kind: ConfigMap
    hostname: grpc-ext-proc.envoygateway
EOF

---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: grpc-ext-proc-btls
spec:
  targetRefs:
    - group: ''
      kind: Service
      name: grpc-ext-proc
  validation:
    caCertificateRefs:
      - name: grpc-ext-proc-ca
        group: ''
        kind: ConfigMap
    hostname: grpc-ext-proc.envoygateway

Verify the BackendTLSPolicy configuration:

kubectl get backendtlspolicy/grpc-ext-proc-btls -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service without Authentication header:

curl -v -H "Host: www.example.com" "http://${GATEWAY_HOST}/myapp"

You should see that the external processor added headers:

• x-request-ext-processed - this header was added before the request was forwarded to the backend
• x-response-ext-processed- this header was added before the response was returned to the client

curl -v -H "Host: www.example.com"  http://localhost:10080/myapp
[...]
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 14 Jun 2024 19:30:40 GMT
< content-length: 502
< x-response-ext-processed: true
<
{
 "path": "/myapp",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
[...] 
  "X-Request-Ext-Processed": [
   "true"
  ],
[...]
 }

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the demo auth services, HTTPRoute, EnvoyExtensionPolicy and BackendTLSPolicy:

kubectl delete -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-proc-grpc-service.yaml
kubectl delete httproute/myapp
kubectl delete envoyextensionpolicy/ext-proc-example
kubectl delete backendtlspolicy/grpc-ext-proc-btls

Next Steps

Checkout the Developer Guide to get involved in the project.

6 - Lua Extensions

This task provides instructions for extending Envoy Gateway with Lua extensions.

Lua extensions allow you to extend the functionality of Envoy Gateway by running custom code against HTTP requests and responses, without modifying the Envoy Gateway binary. These comparatively light-weight extensions are written in the Lua scripting language using APIs defined here.

Envoy Gateway allows the user to configure Lua extensions using the EnvoyExtensionPolicy CRD. This instantiated resource can be linked to a Gateway or HTTPRoute resource. If linked to both, the resource linked to the route takes precedence over those linked to Gateway.

Enable Lua

Lua extensions are disabled by default. To enable Lua, set enableLua: true in the EnvoyGateway configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableLua: true

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Configuration

Envoy Gateway supports Lua in EnvoyExtensionPolicy in two modes:

• Inline Lua: The extension defines the Lua script as an inline string.
• ValueRef Lua: The extension points to an in-cluster ConfigMap resource that contains the Lua script in it’s data.

The following example demonstrates how to configure an EnvoyExtensionPolicy to attach a Lua extension to a HTTPRoute. This Lua extension adds a custom header x-lua-custom: FOO to the response.

Lua Extension - Inline

This EnvoyExtensionPolicy configuration defines the Lua script as an inline string.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-inline-test -o yaml

Lua Extension - ValueRef

This EnvoyExtensionPolicy configuration defines the Lua extension in a ConfigMap resource.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-valueref-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-valueref
        kind: ConfigMap
        group: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-valueref
data:
  lua: |
    function envoy_on_response(response_handle)
      response_handle:headers():add("x-lua-custom", "FOO")
    end
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-valueref-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-valueref
        kind: ConfigMap
        group: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-valueref
data:
  lua: |
    function envoy_on_response(response_handle)
      response_handle:headers():add("x-lua-custom", "FOO")
    end

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-valueref-test -o yaml

Lua Extension - FilterContext

The filterContext field allows you to pass key/value pairs to a shared Lua script, so it can be parameterized differently per route. The Lua script accesses these values via the filterContext() stream handle API.

This example uses a shared ConfigMap Lua script that reads a configurable header name from filter context and copies its value to the authorization header.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-shared-auth
data:
  lua: |
    function envoy_on_request(request_handle)
      local ctx = request_handle:filterContext()
      local token = request_handle:headers():get(ctx:get("token_header"))
      if token and token ~= "" then
        request_handle:headers():replace("authorization", "Bearer " .. token)
      end
    end
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-filter-context-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-shared-auth
        kind: ConfigMap
        group: v1
      filterContext:
        token_header: x-api-key
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-shared-auth
data:
  lua: |
    function envoy_on_request(request_handle)
      local ctx = request_handle:filterContext()
      local token = request_handle:headers():get(ctx:get("token_header"))
      if token and token ~= "" then
        request_handle:headers():replace("authorization", "Bearer " .. token)
      end
    end
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-filter-context-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-shared-auth
        kind: ConfigMap
        group: v1
      filterContext:
        token_header: x-api-key

A different route can reuse the same ConfigMap with different filter context values:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-filter-context-route-b
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend-b
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-shared-auth
        kind: ConfigMap
        group: v1
      filterContext:
        token_header: x-session-token

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-filter-context-test -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service:

curl -i -H "Host: www.example.com" "http://${GATEWAY_HOST}"

You should see that the lua extension has added this header to the response:

x-lua-custom: FOO

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the EnvoyExtensionPolicy:

kubectl delete envoyextensionpolicy/lua-inline-test
kubectl delete envoyextensionpolicy/lua-valueref-test
kubectl delete envoyextensionpolicy/lua-filter-context-test
kubectl delete configmap/cm-lua-valueref
kubectl delete configmap/cm-lua-shared-auth

Next Steps

Checkout the Developer Guide to get involved in the project.

7 - OPA Sidecar with Unix Domain Socket

This task demonstrates how to deploy an Open Policy Agent (OPA) sidecar inside Envoy Proxy pods created by Envoy Gateway, with Envoy Proxy communicating with OPA over a Unix Domain Socket.

The setup provides these advantages over running OPA as its own Kubernetes service:

• Using a sidecar ensures that the OPA is always on the same worker node as Envoy Proxy. This reduces latency and simplifies access control.
• Using a Unix Domain Socket avoids TCP/TLS overhead while improving security by avoiding network exposure and reduces latency.

This production-ready approach uses Envoy Gateway’s extension APIs to configure Envoy Proxy’s external authorization and standard Kubernetes API objects to configure OPA’s Envoy External Authorization gRPC plugin.

graph TB
    subgraph K8sPod ["⎈ Envoy Proxy Pod"]
        subgraph EnvoyContainer ["Envoy Proxy Container"]
            ExtAuthz("🔐 ext_authz Filter")
        end

        UnixSocket[("Unix Domain Socket<br/>/shared/opa/opa.sock")]

        subgraph OPAContainer ["OPA Container"]
            RegoPolicy("⚙️ Rego Policy")
        end
    end

    %% Flow Definitions
    ExtAuthz <-->|"Authz check"| UnixSocket
    UnixSocket <-->|"Authz check"| RegoPolicy

    style UnixSocket fill:#D0D0D0
    style EnvoyContainer fill:#2962FF,color:#FFFFFF
    style OPAContainer fill:#2962FF,color:#FFFFFF

Note: In this diagram, the Unix Domain Socket is shown outside of the Proxy and OPA containers, but it is actually mounted to both containers.

The example flow on this page will use Envoy to authenticate JWTs and OPA to validate the ISS claim on the JWT. If the ISS matches a configured env var value then requests will be allowed.

Note that the OPA sidecar with Unix Domain Sockets pattern does not require that JWTs be used. Also, simple ISS claim validation could be done more easily using the “authentication” property of the SecurityPolicy spec, but using OPA allows for more complex logic to be applied. See JWT Claim-Based Authorization to see how to use the SecurityPolicy “authentication” property.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Steps overview

1. Enable the Backend extension API so Envoy can reference a Unix Domain Socket.
2. Create an OPA policy in a ConfigMap and mount it into an OPA sidecar in the Envoy pods.
3. Configure the Envoy Proxy pods to have an OPA sidecar
4. Configure a SecurityPolicy that delegates authorization to OPA via the Unix Domain Socket Backend. For demonstration purposes we’ll also configure JWT authentication.
5. Create an HTTPRoute that requires protection
6. Test the configuration

Step 1: Enable the Backend extension API

Enable the Backend API in your Envoy Gateway ConfigMap to allow Envoy to reference Unix Domain Sockets by setting the “enableBackend” property to “true” in the “envoy-gateway-config” ConfigMap:

data:
  envoy-gateway.yaml: |
    ...
    extensionApis:
      enableBackend: true

Run these commands to update the ConfigMap:

OLD_CONFIG=$(kubectl get cm envoy-gateway-config -n envoy-gateway-system -o jsonpath='{.data.envoy-gateway\.yaml}')
NEW_CONFIG=$(echo "$OLD_CONFIG" | sed "s/extensionApis: {}/extensionApis:\n  enableBackend: true/")
kubectl create configmap envoy-gateway-config -n envoy-gateway-system \
  --from-literal=envoy-gateway.yaml="$NEW_CONFIG" \
  --dry-run=client -o yaml | kubectl apply -f -

Ignore the kubectl warning. It happens because kubectl apply expects a “receipt” (the last-applied-configuration annotation) to be present on the resource so it can track changes over time. If the ConfigMap was originally created using kubectl create (without --save-config) or by a Helm chart, that receipt is missing. kubectl is simply telling you that it’s adding that receipt now.

Restart the Envoy Gateway deployment to apply the updated configuration:

kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway

The Envoy Proxy pod will be restarted later in this guide.

Step 2: Create the OPA policy

Create a ConfigMap with a Rego policy that OPA will load. This example reads the “ALLOWED_ISS” environment variable and compares it to the JWT ISS claim, allowing the request only if they match.

apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-policy
  namespace: envoy-gateway-system
data:
  policy.rego: |
    package envoy.authz

    import rego.v1

    default allow := false

    opa_env := opa.runtime().env
    allowed_iss := opa_env.ALLOWED_ISS
    jwt_iss := input.attributes.request.http.headers["x-jwt-iss"]

    allow if {
        jwt_iss == allowed_iss
    }

On MacOS you can apply that by copying it and running pbpaste | kubectl apply -f -.

In this policy we extract a x-jwt-iss header from the request. That header is not automatically available on ext_authz filter requests. We will configure Envoy to add that header from the decoded JWT. That configuration happens in the SecurityPolicy in step 4.

Step 3: Configure the Envoy Proxy pods to have an OPA sidecar

Create an EnvoyProxy resource that mounts the OPA policy ConfigMap, shares a Unix Domain Socket between Envoy and OPA, and configures the “ext_authz” filter order so OPA receives JWT headers from the JWT authentication filter.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: envoy-gateway-system
spec:
  # Ensure ext_authz runs after jwt_authn so that OPA sees the JWT-derived headers
  # configured in the SecurityPolicy in step 4
  filterOrder:
    - name: envoy.filters.http.ext_authz
      after: envoy.filters.http.jwt_authn
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  volumes:
                    # Shared Unix Domain Socket volume for communication between Envoy
                    # and OPA. This will be mounted in both containers.
                    - name: opa-unix-socket
                      emptyDir: {}
                    # OPA Rego policy file from ConfigMap. This will be mounted
                    # in the OPA container.
                    - name: opa-policy
                      configMap:
                        name: opa-policy
                  containers:
                    # Add the Unix Domain Socket volume to the existing envoy
                    # proxy container named "envoy".
                    - name: envoy
                      volumeMounts:
                        # Mount the Unix Domain Socket volume in the Envoy container
                        - name: opa-unix-socket
                          mountPath: /shared/opa
                    # Create the OPA sidecar container
                    - name: opa
                      # OPA image tag format:
                      # - "x.y.z": OPA version
                      # - "envoy": includes OPA's Envoy External Authorization gRPC plugin
                      # - "5": plugin version
                      # - "static": static build (no glibc dependency, smaller image)
                      image: openpolicyagent/opa:1.10.1-envoy-5-static
                      args:
                        - "run"
                        - "--server"
                        # In addition to listening on the Unix Domain Socket, we'll listen
                        # on port 8181 for k8s probes
                        - "--addr=0.0.0.0:8181"
                        # Configure OPA's Envoy external authorization plugin to listen on a Unix Domain Socket
                        - "--set=plugins.envoy_ext_authz_grpc.addr=unix:///shared/opa/opa.sock"
                        # In the OPA policy we defined the policy as being in a
                        # "envoy.authz" package and set the "allow" variable
                        # based on the policy decision. Here we tell the plugin
                        # to check that value.
                        - "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
                        - "--ignore=.*"
                        # You can set log-level to "debug" for detailed request logging
                        - "--log-level=error"
                        # Tell OPA where to find the policy file
                        - "/policy/policy.rego"
                      env:
                        # Environment variable used by the OPA policy to validate the JWT issuer
                        - name: ALLOWED_ISS
                          value: "https://example.com"
                      ports:
                        # Port for health checks only; gRPC server listens on the Unix Domain Socket
                        - name: http
                          containerPort: 8181
                          protocol: TCP
                      volumeMounts:
                        # Mount the Unix Domain Socket volume in the OPA container
                        - name: opa-unix-socket
                          mountPath: /shared/opa
                        # Mount the OPA Rego policy from ConfigMap
                        - name: opa-policy
                          mountPath: /policy
                          readOnly: true
                      livenessProbe:
                        httpGet:
                          path: /health
                          port: 8181
                        initialDelaySeconds: 5
                        periodSeconds: 10
                        timeoutSeconds: 3
                        failureThreshold: 3
                      readinessProbe:
                        httpGet:
                          path: /health?bundle=true
                          port: 8181
                        initialDelaySeconds: 5
                        periodSeconds: 5
                        timeoutSeconds: 3
                        failureThreshold: 3
                      securityContext:
                        # Use the same user ID as the Envoy container (65532) so both containers can access the socket
                        # The user ID was found by running "docker pull docker.io/envoyproxy/gateway:v<version> && docker inspect docker.io/envoyproxy/gateway:v<version>"
                        runAsUser: 65532
                        allowPrivilegeEscalation: false
                      resources:
                        # Adjust as necessary
                        requests:
                          cpu: 100m
                          memory: 128Mi
                        limits:
                          memory: 512Mi

This patch adds an OPA sidecar container to the Envoy pods, creates a shared Unix Domain Socket at /shared/opa/opa.sock, and configures OPA to listen only on this socket instead of TCP. It sets an ALLOWED_ISS property that OPA will use to validate the request. We purposefully set the wrong value here so we can see the response when the wrong ISS is used. We’ll set it to the correct value later.

Tell Envoy Gateway to use this EnvoyProxy spec by patching the GatewayClass.

kubectl patch gatewayclass eg --type merge -p '
{
  "spec": {
    "parametersRef": {
      "group": "gateway.envoyproxy.io",
      "kind": "EnvoyProxy",
      "name": "eg",
      "namespace": "envoy-gateway-system"
    }
  }
}'

This will cause the Envoy Proxy pod to be replaced with the new config.

Step 4: Connect Envoy to OPA with a SecurityPolicy and Backend

Create a Unix Domain Socket SecurityPolicy and a Backend that authenticates JWTs, maps the ISS claims to the x-jwt-iss header, and delegates authorization to OPA.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: opa-jwt-authz
  # The SecurityPolicy must be in the same namespace as the HTTPRoutes that it
  # will apply to
  namespace: default
spec:
  targetSelectors:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      matchLabels:
        auth: opa-jwt
  jwt:
    providers:
      - name: my-jwt-validator
        # Verify that the JWT is signed with the correct key
        remoteJWKS:
          uri: https://raw.githubusercontent.com/envoyproxy/gateway/refs/heads/main/examples/kubernetes/jwt/jwks.json
        # Add the JWT ISS claim to request to the headers sent to OPA. Note: this only works
        # for string claims, not arrays. For example, the "aud" claim cannot be passed
        # this way because the JWT spec defines it as an array.
        claimToHeaders:
          - header: x-jwt-iss
            claim: iss
  # Delegate authorization to an external program after JWT verification
  extAuth:
    grpc:
      backendRefs:
        # Reference the OPA Unix Domain Socket backend, which is the next API object in this code block
        - group: gateway.envoyproxy.io
          kind: Backend
          name: opa-unix-socket
    # Headers to send to OPA for authorization decisions. This should match the
    # "claimToHeaders" values above.
    headersToExtAuth:
      - x-jwt-iss
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: opa-unix-socket
  # The Backend must be in the same namespace as the SecurityPolicy
  namespace: default
spec:
  endpoints:
    # This configures Envoy proxy to send external auth requests to the Unix Domain Socket
    - unix:
        path: /shared/opa/opa.sock

Step 5: Configure route to use the OPA authorization

Attach the policy by labeling routes that require authorization. This HTTPRoute uses the backend service installed in the Quick Start:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-opa
  namespace: default
  labels:
    auth: opa-jwt
spec:
  hostnames:
  - foo.bar.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /test-opa
    backendRefs:
    - name: backend
      port: 3000

Step 6: Test the configuration

Prepare some env vars:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
export VALID_TOKEN="eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImI1MjBiM2MyYzRiZDc1YTEwZTljZWJjOTU3NjkzM2RjIn0.eyJpc3MiOiJodHRwczovL2Zvby5iYXIuY29tIiwic3ViIjoiMTIzNDU2Nzg5MCIsInVzZXIiOnsibmFtZSI6IkpvaG4gRG9lIiwiZW1haWwiOiJqb2huLmRvZUBleGFtcGxlLmNvbSIsInJvbGVzIjpbImFkbWluIiwiZWRpdG9yIl19LCJwcmVtaXVtX3VzZXIiOnRydWUsImlhdCI6MTUxNjIzOTAyMiwic2NvcGUiOiJyZWFkIGFkZCBkZWxldGUgbW9kaWZ5In0.P36iAlmiRCC79OiB3vstF5Q_9OqUYAMGF3a3H492GlojbV6DcuOz8YIEYGsRSWc-BNJaBKlyvUKsKsGVPtYbbF8ajwZTs64wyO-zhd2R8riPkg_HsW7iwGswV12f5iVRpfQ4AG2owmdOToIaoch0aym89He1ZzEjcShr9olgqlAbbmhnk-namd1rP-xpzPnWhhIVI3mCz5hYYgDTMcM7qbokM5FzFttTRXAn5_Luor23U1062Ct_K53QArwxBvwJ-QYiqcBycHf-hh6sMx_941cUswrZucCpa-EwA3piATf9PKAyeeWHfHV9X-y8ipGOFg3mYMMVBuUZ1lBkJCik9f9kboRY6QzpOISARQj9PKMXfxZdIPNuGmA7msSNAXQgqkvbx04jMwb9U7eCEdGZztH4C8LhlRjgj0ZdD7eNbRjeH2F6zrWyMUpGWaWyq6rMuP98W2DWM5ZflK6qvT1c7FuFsWPvWLkgxQwTWQKrHdKwdbsu32Sj8VtUBJ0-ddEb"

If Envoy is not running behind a load balancer, then you’ll have to start port forwarding whenever the Envoy Proxy pod is reconfigured and the proxy pods are replaced:

kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &

Test without a JWT

Requests without a valid JWT are denied:

curl -v -H "Host: foo.bar.com" "http://${GATEWAY_HOST}/test-opa/get"

Expected result: HTTP 401 Unauthorized

Test with a valid JWT that has the wrong ISS claim

Earlier, we configured the “ALLOWED_ISS” env var on the proxy pods to be “https://example.com”, but the JWT in the “VALID_TOKEN” env var has an ISS claim of “https://foo.bar.com”.

Requests with an incorrect ISS are denied:

curl -v -H "Host: foo.bar.com" -H "Authorization: Bearer $VALID_TOKEN" "http://${GATEWAY_HOST}/test-opa/get"

Expected result: HTTP 403 Forbidden

Test with a valid JWT and ISS claim

We must update the EnvoyProxy spec to use “https://foo.bar.com” for the “ALLOWED_ISS” so that it matches the ISS in “$VALID_TOKEN”.

...
                  containers:
                    - name: opa
                      env:
                        - name: ALLOWED_ISS
                          value: "https://foo.bar.com" # Change to this
...

The quickest way to do that for this tutorial is via kubectl -n envoy-gateway-system edit envoyproxy eg.

Wait for the Envoy Proxy pod to be replaced.

When the JWT issuer matches your configured provider and the OPA policy allows it, the request is forwarded to the backend:

curl -v -H "Host: foo.bar.com" -H "Authorization: Bearer $VALID_TOKEN" "http://${GATEWAY_HOST}/test-opa/get"

Expected result: HTTP 200 and a response from the backend service.

Clean up

Delete all resources created by this task:

kubectl delete httproute/test-opa securitypolicy/opa-jwt-authz backend/opa-unix-socket
kubectl delete envoyproxy/eg configmap/opa-policy -n envoy-gateway-system
kubectl delete deploy/$(kubectl get deploy -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}') -n envoy-gateway-system

8 - Remote Infrastructure Provider

This task explains how to configure Envoy Gateway to defer Envoy proxy and rate limit infrastructure management to a remote service of your own. With the Remote infrastructure provider, Envoy Gateway translates Gateway API resources into its into an infrastructure IR format and forwards it over gRPC to a provider you operate. The provider is then responsible for reconciling the data plane.

This is the right choice when the built-in Kubernetes infrastructure provider doesn’t fit your environment. Common cases include running proxies in a different cluster from the control plane, deploying onto a non-Kubernetes substrate (VMs, an internal scheduler, etc.), or imposing custom organizational rules on how proxy workloads are shaped.

Note: Most users should use the built-in Kubernetes infrastructure provider. The Remote provider is an extension point; running it means owning the lifecycle of every proxy and rate limit deployment yourself.

Security Warning

How it works

When the Remote provider is enabled, Envoy Gateway defers infrastructure management using an gRPC client to send updates to your provider. For every reconcile that mutates a proxy or rate limit deployment, Envoy Gateway issues a corresponding RPC carrying the IR as structured protobuf data.

The service contract is defined in proto/remoteinfra/service.proto:

service EnvoyGatewayRemoteInfrastructureProvider {
    rpc CreateOrUpdateProxyInfra(CreateOrUpdateProxyInfraRequest) returns (CreateOrUpdateProxyInfraResponse);
    rpc DeleteProxyInfra(DeleteProxyInfraRequest) returns (DeleteProxyInfraResponse);
    rpc CreateOrUpdateRateLimitInfra(CreateOrUpdateRateLimitInfraRequest) returns (CreateOrUpdateRateLimitInfraResponse);
    rpc DeleteRateLimitInfra(DeleteRateLimitInfraRequest) returns (DeleteRateLimitInfraResponse);
}

The proxy RPCs carry the IR as a structured Infra message that mirrors internal/ir/infra.go, including proxy metadata (labels, annotations, and the owner reference) and each metric sink’s destination (endpoints and upstream TLS). Most fields are typed protobuf fields; the proxy config (an EnvoyProxy resource) is the only field carried as JSON-encoded bytes, because that CRD schema is large and evolves independently of this contract. The rate limit RPCs are parameterless. The provider is expected to be idempotent — Envoy Gateway calls the CreateOrUpdate RPCs every time the desired state changes and may retry on transient errors.

Configuration

The Remote provider is selected under provider.custom.infrastructure in the EnvoyGateway config. The service field is required and uses the standard [ExtensionService] shape, so you can reach the provider over a Unix socket, a hostname, or an IP.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
provider:
  type: Custom
  custom:
    resource:
      type: Kubernetes
    infrastructure:
      type: Remote
      remote:
        service:
          unix:
            path: /var/run/remote-infra/server.sock

To dial a remote service over the network instead, use fqdn or ip:

service:
  fqdn:
    hostname: remote-infra.envoy-gateway-system.svc.cluster.local
    port: 5005
  tls:
    certificateRef:
      name: remote-infra-ca
      namespace: envoy-gateway-system

When TLS is configured, Envoy Gateway verifies the server certificate against the CA in the referenced Secret. Add clientCertificateRef if your provider requires mTLS.

Rate limit service address

The Remote provider tells Envoy Gateway what rate limit infrastructure to stand up via the CreateOrUpdateRateLimitInfra RPC, but Envoy proxies still need to know where to send rate limit checks. Set rateLimit.url on the EnvoyGateway config to the address your provider exposes. The value must be a grpc:// URL with an explicit host and port:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
rateLimit:
  url: grpc://my-hosted-rl-service.com:443

If rateLimit.url is unset, Envoy Gateway falls back to the address of the Service the Kubernetes infrastructure manager would have created.

Quickstart

This walks through deploying the example provider in examples/remote-infra as a native sidecar of the envoy-gateway Deployment, communicating over a Unix domain socket. The example provider reconciles a Deployment and a Service per Gateway and is intended as a starting point — not a production-ready provider.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Build and load the example provider image

The example lives in examples/remote-infra/ and ships with a Makefile target that builds a multi-arch image tagged envoyproxy/gateway-remote-infra:latest.

make -C examples/remote-infra docker-buildx

If you’re running on kind, load the image into the cluster so the sidecar can pull it:

kind load docker-image --name envoy-gateway envoyproxy/gateway-remote-infra:latest

Inject the provider as a sidecar

The provider needs to share a volume with envoy-gateway so they can talk over a UDS, and it needs to start before envoy-gateway does. Native sidecars give us both for free on Kubernetes 1.29+.

cat <<'EOF' | kubectl patch deployment envoy-gateway -n envoy-gateway-system --patch-file /dev/stdin
apiVersion: apps/v1
kind: Deployment
metadata:
  name: envoy-gateway
  namespace: envoy-gateway-system
spec:
  template:
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 65532
        runAsGroup: 65532
        # fsGroup ensures the shared emptyDir is group-owned by 65532 so the
        # non-root sidecar can create the UDS socket file in it.
        fsGroup: 65532
      volumes:
        - name: remote-infra-socket
          emptyDir: {}
      containers:
        - name: envoy-gateway
          volumeMounts:
            - name: remote-infra-socket
              mountPath: /var/run/remote-infra
      initContainers:
        - name: remote-infra
          image: envoyproxy/gateway-remote-infra:latest
          imagePullPolicy: IfNotPresent
          restartPolicy: Always
          args:
            - --socket-path=/var/run/remote-infra/server.sock
          env:
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          # Gates startup of envoy-gateway: it will not start until the UDS
          # socket exists.
          startupProbe:
            exec:
              command: ["test", "-S", "/var/run/remote-infra/server.sock"]
            periodSeconds: 1
            failureThreshold: 30
          volumeMounts:
            - name: remote-infra-socket
              mountPath: /var/run/remote-infra
EOF