Using Envoy for Egress Traffic

In this blog post, Palantir’s Network Infrastructure team will share our recent experience transitioning to Envoy to enable granular egress traffic filtering for the forward proxy in Rubix, Palantir’s Kubernetes infrastructure. Envoy is an open-source, high-performance edge and service proxy with built-in features for L4/L7 filtering, service discovery, dynamic configuration, and more. In the article, we will:

• Explain the motivation to switch from Squid to Envoy
• Describe the new Envoy forward proxy configuration inside Rubix
• Cover how we monitor Envoy metrics

Our old Squid-based forward proxy

We initially used Squid as the forward proxy in Rubix. This forward proxy had a basic but reliable setup: when we performed Kubernetes cluster stand up in Rubix, the Squid proxy configurations were statically defined and initiated as a systemed unit. We configured Rubix egress traffic such that all outbound traffic would be directed to the router instance and redirected via iptables to Squid for traffic filtering on a global level.

Over time, this solution failed to adapt to our evolving deployment strategies. Fundamentally, we wanted to shift control of egress and ingress traffic filter configuration management away from Rubix and into the Foundry platform to reduce the manual burden on infrastructure engineers. This transfer of control into the customer’s hands would also greatly empower customer IT organizations to control network traffic in Foundry themselves without requiring support from our teams.

To support the transfer of control of egress and ingress traffic filter configuration management, it was critically important for us to add first-class networking support for our Foundry platform. This included incorporating secure, granular traffic rules that we and our customers could dynamically create as needed. Since Squid was statically configured and required entire cluster restarts for rule updates, we investigated new forward proxy solutions.

Using Envoy to control egress traffic

As a modern, high-performance, small-footprint edge and service proxy, Envoy was a natural choice for us. We also have high confidence in the benefits of Envoy after deploying it for the service mesh and ingress in Rubix. The following two features offered particular benefit to us:

• Envoy can discover a variety of dynamic resources, including listeners, clusters, and route configuration. These resources are requested via subscriptions to a management server that provides discovery services and corresponding API updates via gRPC. This approach would enables us to make automatic updates to the proxy configuration without cluster restarts whenever a traffic rule change is made on the Foundry side.
• Another benefit of Envoy is it produces an extensive in-class collection of observability data via distributed tracing, logging, and time-series metrics. This capability would allow our engineers to maintain clear network transparency and quickly root cause problems.

These two promising features combined with the extensive catalog of configuration possibilities made us eager to replace Squid with Envoy.

Envoy proxy setup inside Rubix

Inside Rubix, we created our implementation of an xDS control plane that leverages the Envoy go-control-plane library to implement the discovery service API server and proxy configuration cache. The xDS control plane is responsible for watching for egress traffic filter rule changes, making corresponding proxy configuration updates, and pushing them to subscribed proxies.

The final egress traffic flow through Envoy inside Rubix works as follows:

Traffic leaving the environment is routed to per-availability-zone egress hosts based on routing table entries. When it reaches these hosts, it is redirected to an Envoy proxy via iptables. A listener accepts this downstream TCP connection and creates a new listener filter chain, which holds a set of pre-processing filters and matching criteria.

After processing, Envoy uses the newly extracted packet information to match the packet to a filter chain that defines outbound destinations allowed on a global level or a per-pod or per-workflow basis. Matching criteria can be set using any combination of CIDR block ranges, domain names, source address, etc. If there is a match, the traffic will pass through a TCP proxy filter that performs a single network connection proxy to an upstream cluster. Since the iptables had previously redirected the packet and rewrote the destination to be the Envoy listener address, we rely on the original destination listener and cluster to recover the original outbound destination address and send the traffic to the correct endpoint.

While setting up the definition of these components in our proxy configuration, we used Envoy’s Configuring Envoy as an edge proxy article for helpful recommendations on edge proxy-specific settings.

Monitoring Envoy metrics

As noted earlier, Envoy emits a large variety of downstream, upstream, and server statistics and supports custom, pluggable sinks that can aggregate and forward these statistics to a common destination. We have deployed a sidecar to each Envoy proxy to listen for metrics coming from a configured StatsD sink to take advantage of this. The emitted metrics are consumed for display on Datadog dashboards. We also have several of the Datadog monitors set up to track critical metrics and alert the team when some metric statistic surpasses a preset threshold. As the migration to Envoy occurs, there are a few key metrics that we plan on monitoring to provide a useful signal on the health of the migration:

• listener.no_filter_chain_match: The upstream allowlist is an Envoy filter chain, so denied requests will show up in this metric. Monitoring this as the migration occurs gives visibility for any configuration issues.
• listener.downstream_cx_destroy/listener.downstream_cx_total: Monitoring this metric gives signal for the health of the connections between Rubix pods and the Envoy instances.
• control_plane.connected_state: This metric is monitored across all our Envoy instances to ensure they stay connected to our control plane and receive config updates.

Complementary security features

At Palantir, we use Cilium for our Kubernetes-based firewalls, as discussed in this blog post. These firewalls also allow/deny egress traffic, so it is worth discussing how Cilium fits in with Envoy within the context of egress traffic.

Cilium policies are application driven, which means a service may declare that it would like to egress the cluster to a specific destination. Once the Cilium policy is created, traffic from a pod can then egress through the forward proxy. Should the pod try to egress without a Cilium policy for that destination, then traffic will be blocked by Cilium.

Our egress proxy also contains another allowlist of destinations. The allowlist at the egress proxy contains the union of all legitimate Cilium policies for a cluster. Should a rogue Cilium policy be created to egress to a destination, traffic will be still be stopped at the egress proxy as that destination won’t be defined in the egress proxy’s allowlist. This dual layer approach allows for very granular control over egress traffic with cluster level controls defined at the egress proxy and pod-level controls defined at the firewall level.

Closing Thoughts

Transitioning to Envoy as the new forward proxy has been crucial to enabling granular egress traffic filtering in Rubix and removing the complex chain of manual actions that infrastructure engineers had to perform to maintain customer allowlist requests. The large set of observability metrics and logging is a huge bonus, allowing us to gain insights about the state of networking within Rubix efficiently.

We still have a lot of work to do on the Network Infrastructure side! Some egress-specific projects we’re working on include: adding advanced connectivity features in Rubix for Palantir’s customers, and creating Envoy sidecars to Spark compute pods for tighter egress traffic control.

Authors

Liam Hazell, Software Engineer, Network Infrastructure

Jannie Yu, Software Engineer Intern, Network Infrastructure