EnvoyProxy Primer

Cornelio Hopmann 2018-08-29

Prerequisites

  • Basic knowledge of networking concepts.
  • YAML

Introduction

EnvoyProxy is a powerful reverse proxy software commonly used in Kubernetes and hosted by Cloud Native. Although not everybody has the pleasure to work with Kubernetes and enjoy some of the tooling and software around it, that does not mean we cannot use some of the great parts outside of Kubernetes. This lesson will cover that basics of EnvoyProxy and give an example of how to port another common reverse proxy software in this case HAProxy to EnvoyProxy and then extend in functionality.

What is EnvoyProxy?

Tip: google-fu EnvoyProxy (envoy proxy wont work that well)

EnvoyProxy is the "new shiny thing" that came out of Lyft around 2016. It was already being used on production when it came out and is battle tested. It's written in C++, has an interesting threading model, has a vibrant developer community and it's a project in the cloud native computing foundation. It's used at Netflix, Google et al.

Supported features

L3/L4 filter architecture, HTTP L7 filter architecture, First class HTTP/2 support, HTTP L7 routing, gRPC support, MongoDB L7 support, DynamoDB L7 support, Service discovery, Health checking, Advanced load balancing, ability to install plugins (such as support for mqtt) and a lot more.

Something simple first

Let's start by looking at simple haproxy.cfg file and then go from there.

# https://linuxacademy.com/howtoguides/posts/show/topic/14569-configuration-of-haproxy
global
    daemon
    maxconn 4096
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-  SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 ssl-min-ver TLSv1.2 no-tls-tickets
    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-   SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    mode http
    timeout connect 25000ms
    timeout client 60000ms
    timeout server 60000ms
    timeout queue 60000ms
    timeout http-request 15000ms
    timeout http-keep-alive 15000ms
    option redispatch
    option forwardfor
    option http-server-close

frontend http-in
    bind 10.0.1.50:80
    rspadd X-Forwarded-Host:\ http:\\\example.com
    redirect scheme https code 301 if !{ ssl_fc }

frontend https-in
    bind 10.0.1.50:443 ssl crt /etc/ssl/bundle.pem
    rspadd X-Forwarded-Host:\ https:\\\\example.com
    rspadd Strict-Transport-Security:\ max-age=31536000

    acl url_image path_beg -i /image/
    use_backend image-backend if url_image

    acl url_map path_beg -i /map/
    use_backend map-backend if url_map

    acl url_geocode path_beg -i /geocode/
    use_backend geocode-backend if url_geocode

    acl url_stream path_beg -i /stream/
    use_backend stream-backend if url_stream

    default_backend image-backend

backend image-backend
    server image01 10.0.5.10:443 check ssl verify none
    server image02 10.0.5.11:443 check ssl verify none
    server image03 10.0.5.12:443 check ssl verify none

backend map-backend
    server map01 10.0.5.13:6443 check ssl verify none
    server map02 10.0.5.14:6443 check ssl verify none
    server map03 10.0.5.15:6443 check backup ssl verify none

backend geocode-backend
    server geocode01 10.0.5.16:6080 check verify none
    server geocode02 10.0.5.17:6080 check verify none backup

backend stream-backend
    server stream01 10.0.5.18:6443 check ssl verify none
    server stream02 10.0.5.19:6443 check backup ssl verify none
    server stream03 10.0.5.20:6443 check backup ssl verify none
    server stream04 10.0.5.21:6443 check backup ssl verify none

If you want to learn more about HAProxy check out this guide from Linux Academy.

We will port the feature set listed below to an EnvoyProxy configuration file and on the way learn a bit about the EnvoyProxy.

  • Listen on tcp 80 (HTTP) and 443 (HTTPS)
  • On HTTP(80) redirect to HTTPS(443)
  • ACL rules to direct traffic to backend pools
  • Default backend for traffic that does not match any ACL
  • Set the X-Forward headers to example.com
  • Enable HSTS (no downgrade to HTTP)

EnvoyProxy configuration

Envoy's configuration feels a lot more bulky due the fact that you write everything as json or yaml but this comes with many advantages.

Below I show how to manually add the features seen in the HAProxy, but if you would like to programmatically generate the configuration it isn't that hard because almost everything in EnvoyProxy is defined as a Protobuf.

More info:

Listen on tcp 80 (HTTP) and 443 (HTTPS)

---
admin:
  access_log_path: "/dev/stdout"
  address:
    socket_address: { address: 127.0.0.1, port_value: 9901 }

static_resources:
  listeners:
  - name: http_listener
    address:
      socket_address: { address: "10.0.1.50", port_value: 80 }

  - name: https_listener
    address:
      socket_address: { address: "10.0.1.50", port_value: 443 }

The admin interface

---
admin:
  access_log_path: "/dev/stdout"
  address:
    socket_address: { address: 127.0.0.1, port_value: 9901 }

With the admin interface you can control how the EnvoyProxy behaves. You can shutdown the server, set logging levels, activate circuit breakers, get some Stats and Metrics and perform potential dangerous operations. Due to the fact that this interface lacks proper security, it is highly recommended to expose the interface only to localhost or a properly secured network.

More info:

Static resources

static_resources:
  listeners:
  - name: http_listener
    address:
      socket_address:
        address: "10.0.1.50"
        port_value: 80

EnvoyProxy supports multiple configuration mechanisms. To keep things simple for this lesson, we are using a fully static configuration file. This may sound as a limiting factor, but EnvoyProxy supports graceful hot restarts and as we learned, is not that difficult to generate the right json/yaml file and trigger a configuration update.

More Info:

Listeners

EnvoyProxy supports any number of listeners on a single process. In HAProxy you can group multiple listeners under a single frontend, thus effectively sharing the configuration, in EnvoyProxy you have to configure every listener independently. When using a dynamic configuration mechanism, EnvoyProxy is able to open and close listeners on demand.

EnvoyProxy allows you to configure filters directly on the listener giving you the possibility to react and manipulate the connection and connection metadata before any more advance filtering has taken place. You could have for example a listener that accepts tls and non-tls traffic and routes accordingly.

As of this writing EnvoyProxy support only TCP listeners, but a feature request and a tracking ticket has been opened to support UDP.

More Info:

On HTTP(80) redirect to HTTPS(443)

static_resources:
  listeners:
    - name: http_listener
      address:
        socket_address: { address: "10.0.1.50", port_value: 80 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                http_filters:
                  - name: envoy.router
                    config: {}
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: backend
                      domains:
                        - "example.com"
                      routes:
                        - match:
                            prefix: /
                          redirect:
                            https_redirect: true # Swap http with https and 301
                stat_prefix: ingress_http

    - name: https_listener
      address:
        socket_address: { address: "10.0.1.50", port_value: 443 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                http_filters:
                  - name: envoy.router
                    config: {}
                route_config:
                  name: local_route
                  require_ssl: all # Enforce tls, send https 302 if no tls
                  virtual_hosts:
                    - name: backend
                      domains:
                        - "example.com"
                      routes: "{...}"
                stat_prefix: ingress_https
          tls_context:
            common_tls_context:
              tls_certificates:
                - certificate_chain:
                    filename: ""/etc/ssl/server.crt""
                  private_key:
                    filename: "/etc/ssl/server.key"

Filter chains and Filters

Here is one of the points where EnvoyProxy really shines. It comes with a set of pre built Network Filters (client tls authentication, rate limit, rbac and more) and the really powerful HttpConnectionManager that allows us to plug in filters that are specially designed for working on HTTP connections.

You can have multiple filter chains per listener and have them selected by a matching criteria. Filter chains themselves can have multiple filters that a apply in order of appearance. You would normally put the client tls authentication, before doing any routing.

More Info:

HttpConnectionManager

"HTTP is such a critical component of modern service oriented architectures that Envoy implements a large amount of HTTP specific functionality. Envoy has a built in network level filter called the HTTP connection manager. This filter translates raw bytes into HTTP level messages and events (e.g., headers received, body data received, trailers received, etc.). It also handles functionality common to all HTTP connections and requests such as access logging, request ID generation and tracing, request/response header manipulation, route table management, and statistics." - EnvoyProxy Docs

With the help of filters like route.CorsPolicy you can easily centralize CORS handling for Example or add an external auth system with envoy.ext_authz.

More Info:

Stats and metrics

EnvoyProxy allows you through the admin interface to view the current stats in real time. It provides a number of useful abstractions: counters, gauges, and histograms. For a developer this hides away the complexity of the objects and for the operator it provides easy way to plug in and collect stats. Throughout out this lesson yo might have noticed the "key" "stat_prefix".

This will help organize different stats under the same label. You can connect external systems to collect and process the data generated. One of the supported software is Prometheus.

TLS

You can use EnvoyProxy as an TLS termination and origination endpoint. It comes with a plethora of options for configuring, authenticating and validating certificates. It also enables you to use client side certificates in addition to only server side certificates. Filters like 'clientsslauth" allow you to poll an http endpoint for a list of valid client certificates fingerprints, with that is not much work in getting an easy "vpn".

More Info:

ACL rules to direct traffic to a pool of servers

static_resources:
  listeners:
    - name: https_listener
      address:
        socket_address: { address: "10.0.1.50", port_value: 443 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                http_filters:
                  - name: envoy.router # Sends traffic to clusters
                    config: {}
                route_config:
                  "{...}"
                  virtual_hosts:
                    - "{...}"
                      routes:
                        - match:  # Rule
                            prefix: /image/ # /image/thisWorksAlso
                          route: # route to cluster ...
                            cluster: image_backend
                stat_prefix: ingress_https
          tls_context: "{...}"
  clusters: "{...}"

With a route you specify how to match an a request and what action to take. In this example we are matching on a path, but you could also match on header, methods and more. If that's not enough, you can swap the envoy.Router and use Lua instead.

Clusters

static_resources:
  listeners: {...}
  clusters:
    - name: image_backend
      connect_timeout: 0.25s
      type: strict_dns
      dns_lookup_family: v4_only
      lb_policy: random
      outlier_detection: { consecutive_5xx: 4 } # Passive Health
      hosts:
        - socket_address: { address: "10.0.5.10", port_value: 443 }
        - socket_address: { address: "10.0.5.11", port_value: 443 }
        - socket_address: { address: "10.0.5.12", port_value: 443 }
      ssl_context:
        ca_cert_file: { filename: "/etc/ssl/ca.crt" }

As we have learned, EnvoyProxy supports multiple configuration options. In our case we are using a fully static configuration. Using the dynamic properties of the proxy, we can have a more advanced configuration. Letting for example the service discovery service, update and add members to a cluster, will allow you to react to failures or an increase in load.

We have set up the load-balancing algorithm to random. You are free to use other algorithms or even do weighted balancing. This comes handy if you are trying to roll out an update, with minimal disruption or just wanting to be able to compare both systems. You can configure a cluster to use client side certificates and also validate server side certificates.

More Info:

Outlier detection

For this lesson we will only be using a passive method for health checking. We will rely on the answers giving back by the used backend to tell if that backend is operating as intended. For simplicity sakes, we assume that a server answering with 5xx to requests is faulty. EnvoyProxy also supports different types of active health checking and passive health checking parameters ie success rate.

More Info:

Default backend for traffic that does not match any ACL

static_resources:
  listeners:
    - name: https_listener
      address:
        socket_address: { address: "10.0.1.50", port_value: 443 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                http_filters: "{...}"
                route_config:
                  "{...}"
                  virtual_hosts:
                    - "{...}"
                      routes:
                        - match:
                            prefix: /image/
                          route:
                            cluster: image_backend
                        - "{...}"
                        - match: # Should be last
                            path: /
                          route:
                            cluster: image_backend
                stat_prefix: ingress_https
          tls_context: "{...}"
  clusters: "{...}"

Set the X-Forward headers to example.com and HSTS

static_resources:
  listeners:
    - name: https_listener
      address:
        socket_address: { address: "10.0.1.50", port_value: 443 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                http_filters:
                  - name: envoy.router
                    config: {}
                route_config:
                  name: example_service
                  require_ssl: all # http on 443 send 302 with HTTPS
                  response_headers_to_add:
                    - header:
                        key: "x-forwarded-host"
                        value: "https://example.com"
                      append: true
                    - header:
                        key: "Strict-Transport-Security"
                        value: "max-age=31536000"
                      append: true
                  virtual_hosts: "{...}"

Modifying a request or response with EnvoyProxy is super easy and clear.

Summary

We have learned how to port a sample HAProxy configuration to a working EnvoyProxy configuration and got to know some of the very useful Filters and configurations options. EnvoyProxy is so massive in feature set and plug-ability that I'd recommend taking a look at the documentation for a couple of hours just to find what's in there. Don't be afraid to jump in directly to the Project repositories to have a better understanding of the technology. I hope this lessons makes you hungry to learn more about this cool tech that is EnvoyProxy.

Learn more

In the next lessons learn how easy is to add service discovery and more use advanced features of EnvoyProxy.

Copyright © 2019 Cybus GmbH

Cybus GmbH
Data Privacy
Imprint