EnvoyProxy is a powerful reverse proxy software commonly used in Kubernetes and hosted by Cloud Native. Although not everybody has the pleasure to work with Kubernetes and enjoy some of the tooling and software around it, that does not mean we cannot use some of the great parts outside of Kubernetes. This lesson will cover that basics of EnvoyProxy and give an example of how to port another common reverse proxy software in this case HAProxy to EnvoyProxy and then extend in functionality.
Tip: google-fu EnvoyProxy (envoy proxy wont work that well)
EnvoyProxy is the "new shiny thing" that came out of Lyft around 2016. It was already being used on production when it came out and is battle tested. It's written in C++, has an interesting threading model, has a vibrant developer community and it's a project in the cloud native computing foundation. It's used at Netflix, Google et al.
L3/L4 filter architecture, HTTP L7 filter architecture, First class HTTP/2 support, HTTP L7 routing, gRPC support, MongoDB L7 support, DynamoDB L7 support, Service discovery, Health checking, Advanced load balancing, ability to install plugins (such as support for mqtt) and a lot more.
Let's start by looking at simple haproxy.cfg file and then go from there.
# https://linuxacademy.com/howtoguides/posts/show/topic/14569-configuration-of-haproxy
global
daemon
maxconn 4096
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM- SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM- SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 ssl-min-ver TLSv1.2 no-tls-tickets
defaults
mode http
timeout connect 25000ms
timeout client 60000ms
timeout server 60000ms
timeout queue 60000ms
timeout http-request 15000ms
timeout http-keep-alive 15000ms
option redispatch
option forwardfor
option http-server-close
frontend http-in
bind 10.0.1.50:80
rspadd X-Forwarded-Host:\ http:\\\example.com
redirect scheme https code 301 if !{ ssl_fc }
frontend https-in
bind 10.0.1.50:443 ssl crt /etc/ssl/bundle.pem
rspadd X-Forwarded-Host:\ https:\\\\example.com
rspadd Strict-Transport-Security:\ max-age=31536000
acl url_image path_beg -i /image/
use_backend image-backend if url_image
acl url_map path_beg -i /map/
use_backend map-backend if url_map
acl url_geocode path_beg -i /geocode/
use_backend geocode-backend if url_geocode
acl url_stream path_beg -i /stream/
use_backend stream-backend if url_stream
default_backend image-backend
backend image-backend
server image01 10.0.5.10:443 check ssl verify none
server image02 10.0.5.11:443 check ssl verify none
server image03 10.0.5.12:443 check ssl verify none
backend map-backend
server map01 10.0.5.13:6443 check ssl verify none
server map02 10.0.5.14:6443 check ssl verify none
server map03 10.0.5.15:6443 check backup ssl verify none
backend geocode-backend
server geocode01 10.0.5.16:6080 check verify none
server geocode02 10.0.5.17:6080 check verify none backup
backend stream-backend
server stream01 10.0.5.18:6443 check ssl verify none
server stream02 10.0.5.19:6443 check backup ssl verify none
server stream03 10.0.5.20:6443 check backup ssl verify none
server stream04 10.0.5.21:6443 check backup ssl verify none
If you want to learn more about HAProxy check out this guide from Linux Academy.
We will port the feature set listed below to an EnvoyProxy configuration file and on the way learn a bit about the EnvoyProxy.
Envoy's configuration feels a lot more bulky due the fact that you write everything as json or yaml but this comes with many advantages.
Below I show how to manually add the features seen in the HAProxy, but if you would like to programmatically generate the configuration it isn't that hard because almost everything in EnvoyProxy is defined as a Protobuf.
More info:
---
admin:
access_log_path: "/dev/stdout"
address:
socket_address: { address: 127.0.0.1, port_value: 9901 }
static_resources:
listeners:
- name: http_listener
address:
socket_address: { address: "10.0.1.50", port_value: 80 }
- name: https_listener
address:
socket_address: { address: "10.0.1.50", port_value: 443 }
---
admin:
access_log_path: "/dev/stdout"
address:
socket_address: { address: 127.0.0.1, port_value: 9901 }
With the admin interface you can control how the EnvoyProxy behaves. You can shutdown the server, set logging levels, activate circuit breakers, get some Stats and Metrics and perform potential dangerous operations. Due to the fact that this interface lacks proper security, it is highly recommended to expose the interface only to localhost or a properly secured network.
More info:
static_resources:
listeners:
- name: http_listener
address:
socket_address:
address: "10.0.1.50"
port_value: 80
EnvoyProxy supports multiple configuration mechanisms. To keep things simple for this lesson, we are using a fully static configuration file. This may sound as a limiting factor, but EnvoyProxy supports graceful hot restarts and as we learned, is not that difficult to generate the right json/yaml file and trigger a configuration update.
More Info:
EnvoyProxy supports any number of listeners on a single process. In HAProxy you can group multiple listeners under a single frontend, thus effectively sharing the configuration, in EnvoyProxy you have to configure every listener independently. When using a dynamic configuration mechanism, EnvoyProxy is able to open and close listeners on demand.
EnvoyProxy allows you to configure filters directly on the listener giving you the possibility to react and manipulate the connection and connection metadata before any more advance filtering has taken place. You could have for example a listener that accepts tls and non-tls traffic and routes accordingly.
As of this writing EnvoyProxy support only TCP listeners, but a feature request and a tracking ticket has been opened to support UDP.
More Info:
static_resources:
listeners:
- name: http_listener
address:
socket_address: { address: "10.0.1.50", port_value: 80 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
http_filters:
- name: envoy.router
config: {}
route_config:
name: local_route
virtual_hosts:
- name: backend
domains:
- "example.com"
routes:
- match:
prefix: /
redirect:
https_redirect: true # Swap http with https and 301
stat_prefix: ingress_http
- name: https_listener
address:
socket_address: { address: "10.0.1.50", port_value: 443 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
http_filters:
- name: envoy.router
config: {}
route_config:
name: local_route
require_ssl: all # Enforce tls, send https 302 if no tls
virtual_hosts:
- name: backend
domains:
- "example.com"
routes: "{...}"
stat_prefix: ingress_https
tls_context:
common_tls_context:
tls_certificates:
- certificate_chain:
filename: ""/etc/ssl/server.crt""
private_key:
filename: "/etc/ssl/server.key"
Here is one of the points where EnvoyProxy really shines. It comes with a set of pre built Network Filters (client tls authentication, rate limit, rbac and more) and the really powerful HttpConnectionManager that allows us to plug in filters that are specially designed for working on HTTP connections.
You can have multiple filter chains per listener and have them selected by a matching criteria. Filter chains themselves can have multiple filters that a apply in order of appearance. You would normally put the client tls authentication, before doing any routing.
More Info:
"HTTP is such a critical component of modern service oriented architectures that Envoy implements a large amount of HTTP specific functionality. Envoy has a built in network level filter called the HTTP connection manager. This filter translates raw bytes into HTTP level messages and events (e.g., headers received, body data received, trailers received, etc.). It also handles functionality common to all HTTP connections and requests such as access logging, request ID generation and tracing, request/response header manipulation, route table management, and statistics." - EnvoyProxy Docs
With the help of filters like route.CorsPolicy you can easily centralize CORS handling for Example or add an external auth system with envoy.ext_authz.
More Info:
EnvoyProxy allows you through the admin interface to view the current stats in real time. It provides a number of useful abstractions: counters, gauges, and histograms. For a developer this hides away the complexity of the objects and for the operator it provides easy way to plug in and collect stats. Throughout out this lesson yo might have noticed the "key" "stat_prefix".
This will help organize different stats under the same label. You can connect external systems to collect and process the data generated. One of the supported software is Prometheus.
You can use EnvoyProxy as an TLS termination and origination endpoint. It comes with a plethora of options for configuring, authenticating and validating certificates. It also enables you to use client side certificates in addition to only server side certificates. Filters like 'clientsslauth" allow you to poll an http endpoint for a list of valid client certificates fingerprints, with that is not much work in getting an easy "vpn".
More Info:
static_resources:
listeners:
- name: https_listener
address:
socket_address: { address: "10.0.1.50", port_value: 443 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
http_filters:
- name: envoy.router # Sends traffic to clusters
config: {}
route_config:
"{...}"
virtual_hosts:
- "{...}"
routes:
- match: # Rule
prefix: /image/ # /image/thisWorksAlso
route: # route to cluster ...
cluster: image_backend
stat_prefix: ingress_https
tls_context: "{...}"
clusters: "{...}"
With a route you specify how to match an a request and what action to take. In this example we are matching on a path, but you could also match on header, methods and more. If that's not enough, you can swap the envoy.Router and use Lua instead.
static_resources:
listeners: {...}
clusters:
- name: image_backend
connect_timeout: 0.25s
type: strict_dns
dns_lookup_family: v4_only
lb_policy: random
outlier_detection: { consecutive_5xx: 4 } # Passive Health
hosts:
- socket_address: { address: "10.0.5.10", port_value: 443 }
- socket_address: { address: "10.0.5.11", port_value: 443 }
- socket_address: { address: "10.0.5.12", port_value: 443 }
ssl_context:
ca_cert_file: { filename: "/etc/ssl/ca.crt" }
As we have learned, EnvoyProxy supports multiple configuration options. In our case we are using a fully static configuration. Using the dynamic properties of the proxy, we can have a more advanced configuration. Letting for example the service discovery service, update and add members to a cluster, will allow you to react to failures or an increase in load.
We have set up the load-balancing algorithm to random. You are free to use other algorithms or even do weighted balancing. This comes handy if you are trying to roll out an update, with minimal disruption or just wanting to be able to compare both systems. You can configure a cluster to use client side certificates and also validate server side certificates.
More Info:
For this lesson we will only be using a passive method for health checking. We will rely on the answers giving back by the used backend to tell if that backend is operating as intended. For simplicity sakes, we assume that a server answering with 5xx to requests is faulty. EnvoyProxy also supports different types of active health checking and passive health checking parameters ie success rate.
More Info:
static_resources:
listeners:
- name: https_listener
address:
socket_address: { address: "10.0.1.50", port_value: 443 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
http_filters: "{...}"
route_config:
"{...}"
virtual_hosts:
- "{...}"
routes:
- match:
prefix: /image/
route:
cluster: image_backend
- "{...}"
- match: # Should be last
path: /
route:
cluster: image_backend
stat_prefix: ingress_https
tls_context: "{...}"
clusters: "{...}"
static_resources:
listeners:
- name: https_listener
address:
socket_address: { address: "10.0.1.50", port_value: 443 }
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
http_filters:
- name: envoy.router
config: {}
route_config:
name: example_service
require_ssl: all # http on 443 send 302 with HTTPS
response_headers_to_add:
- header:
key: "x-forwarded-host"
value: "https://example.com"
append: true
- header:
key: "Strict-Transport-Security"
value: "max-age=31536000"
append: true
virtual_hosts: "{...}"
Modifying a request or response with EnvoyProxy is super easy and clear.
We have learned how to port a sample HAProxy configuration to a working EnvoyProxy configuration and got to know some of the very useful Filters and configurations options. EnvoyProxy is so massive in feature set and plug-ability that I'd recommend taking a look at the documentation for a couple of hours just to find what's in there. Don't be afraid to jump in directly to the Project repositories to have a better understanding of the technology. I hope this lessons makes you hungry to learn more about this cool tech that is EnvoyProxy.
In the next lessons learn how easy is to add service discovery and more use advanced features of EnvoyProxy.