istio / ztunnel

The `ztunnel` component of ambient mesh

Ztunnel

Ztunnel provides an implementation of the ztunnel component of ambient mesh.

Note: istio/ztunnel is currently in beta status.

Feature Scope

Ztunnel is intended to be a purpose built implementation of the node proxy in ambient mesh. Part of the goals of this included keeping a narrow feature set, implementing only the bare minimum requirements for ambient. This ensures the project remains simple and high performance.

Explicitly out of scope for ztunnel include:

• Terminating user HTTP traffic
• Terminating user HTTP traffic (its worth repeating)
• Generic extensibility such as ext_authz, WASM, linked-in extensions, Lua, etc.

In general, ztunnel does not aim to be a generic extensible proxy; Envoy is better suited for that task. If a feature is not directly used to implement the node proxy component in ambient mesh, it is unlikely to be accepted.

The details of architecture is here.

Building

Please use the same Rust version as the build-tools image. You can determine the version that the build-tools image uses by running the below command:

$ BUILD_WITH_CONTAINER=1 make rust-version

TLS/Crypto provider

Ztunnel's TLS is built on rustls.

Rustls has support for plugging in various crypto providers to meet various needs (compliance, performance, etc).

Name	How To Enable
ring	Default (or --features tls-ring)
boring	--features tls-boring --no-default-features

In all options, only TLS 1.3 with cipher suites TLS13_AES_256_GCM_SHA384 and TLS13_AES_128_GCM_SHA256 is used.

boring FIPS

With the boring option, the FIPS version is used. Please note this only implies the specific version of the library is used; FIPS compliance requires more than just using a specific library.

FIPS has strict requirements to ensure that compliance is granted only to the exact binary tested. FIPS compliance was granted to an old version of BoringSSL that was tested with Clang 12.0.0.

Given that FIPS support will always have special environmental build requirements, we currently we work around this by vendoring OS/arch specific FIPS-compliant binary builds of boringssl in

We vendor FIPS boringssl binaries for

• linux/x86_64
• linux/arm64

To use these vendored libraries and build ztunnel for either of these OS/arch combos, for the moment you must manually edit .cargo/config.toml and change the values of BORING_BSSL_PATH and BORING_BSSL_INCLUDE_PATH under the [env] key to match the path to the vendored libraries for your platform, e.g:

For linux/x86_64

BORING_BSSL_PATH = { value = "vendor/boringssl-fips/linux_x86_64", force = true, relative = true }
BORING_BSSL_INCLUDE_PATH = { value = "vendor/boringssl-fips/include/", force = true, relative = true }

For linux/arm64

BORING_BSSL_PATH = { value = "vendor/boringssl-fips/linux_arm64", force = true, relative = true }
BORING_BSSL_INCLUDE_PATH = { value = "vendor/boringssl-fips/include/", force = true, relative = true }

Once that's done, you should be able to build:

cargo build

This manual twiddling of environment vars is not ideal but given that the alternative is prefixing cargo build with these envs on every cargo build/run, for now we have chosen to hardcode these in config.toml - that may be revisited in the future depending on local pain and/or evolving boring upstream build flows.

Note that the Dockerfiles used to build these vendored boringssl builds may be found in the respective vendor directories, and can serve as a reference for the build environment needed to generate FIPS-compliant ztunnel builds.

Development

Please refer to this.

Metrics

Ztunnel exposes a variety of metrics, at varying levels of stability. They are accessible by making an HTTP request to either "/stats/prometheus" or "/metrics" on port 15020.

Core metrics are considered stable APIs.

Unstable metrics may be changed. This includes removal, semantic changes, and label changes.

Core metrics

Traffic metrics

• Tcp Bytes Sent (istio_tcp_sent_bytes_total): This is a COUNTER which measures the size of total bytes sent during response in case of a TCP connection.
• Tcp Bytes Received (istio_tcp_received_bytes_total): This is a COUNTER which measures the size of total bytes received during request in case of a TCP connection.
• Tcp Connections Opened (istio_tcp_connections_opened_total): This is a COUNTER incremented for every opened connection.
• Tcp Connections Closed (istio_tcp_connections_closed_total): This is a COUNTER incremented for every closed connection.

Meta metrics

• Istio build information (istio_build)

Unstable metrics

DNS metrics

• DNS Requests (istio_dns_requests_total)
• DNS Upstream Requests (istio_dns_upstream_requests_total)
• DNS Upstream Failures (istio_dns_upstream_failures_total)
• DNS Upstream Request Duration (istio_dns_upstream_request_duration_seconds)
• On Demand DNS Requests (istio_on_demand_dns_total)

In-Pod metrics

• Active proxy count (istio_active_proxy_count_total)
• Pending proxy count (istio_pending_proxy_count_total)
• Proxies started (istio_proxies_started_total)
• Proxies stopped (istio_proxies_stopped_total)

XDS metrics

• XDS Connection terminations (istio_xds_connection_terminations_total)

Logging

Ztunnel exposes a variety of logs, both operational and "access logs".

Logs are controlled by the RUST_LOG variable. This can set all levels, or a specific target. For instance, RUST_LOG=error,ztunnel::proxy=warn. Access logs are under the access target.

An example access log looks like (with newlines for readability; the real logs are on one line):

2024-04-11T15:38:42.182974Z  INFO access: connection complete
    src.addr=10.244.0.24:46238 src.workload="shell-6d8bcd654d-t88gp" src.namespace="default" src.identity="spiffe://cluster.local/ns/default/sa/default"
    dst.addr=10.244.0.42:15008 dst.hbone_addr=10.96.108.116:80 dst.service="echo.default.svc.cluster.local"
    direction="outbound" bytes_sent=67 bytes_recv=490 duration="13ms"

Access logs are emitted upon completion of each connection. Logs for connect establishment are also logged (with less information) at debug level.

Currently, the access log format is considered unstable and subject to changes.

ztunnel's Issues

Access Logging

Should we have access logs similar to envoy?
How configurable should they be?

Connection Logging

Cheap, in memory, stores some list of past events, similar to Envoy access logs.

There should be a cheap in-memory connection log that's accessible for debugging purposes. The runtime cost of maintaining this log should be extremely low. This could be achieved by a per-thread in-memory list of events including the 5-tuple of the connection, what happened (open, close, error, pkt/bytes sent) and a timestamp. Whenever something happens on a connection, we can add to this list a note at very low cost. There could be a configurable limit to the size of the thing.
With this, when debugging a user could run a ztctl command to explore this data. They should be able to pass in an IP or 5-tuple and see all connections that have occurred recently with that configuration. There should be a watch command that shows the connections currently flowing through the system as they come and go. The debug dump tool should dump every thread's connection log so that it can be analyzed off-box by a support engineer.

Either add rust support to build-tools, build-tools-proxy, or make a build-tools-rust

FIPS compliant TLS

This is already done.

testnewlines

a\nb

Question: any consideration for HA(high availability) of ztunnel?

since ztunnel is running in standalone mode per node, any plan to support HA for it?

Audit existing library usage for best practices.

Implement (L4) RBAC

This was an initial attempt in Go:

unc (p *Proxy) AssertRBAC(r *http.Request) error {
	ip, dport, err := net.SplitHostPort(r.Host)
	if err != nil {
		return err
	}
	pip, err := netip.ParseAddr(ip)
	if err != nil {
		return err
	}
	wl := p.ConnectionManager.FindWorkloadByAddr(pip)
	var identity string
	if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
		n, err := util.ExtractIDs(r.TLS.PeerCertificates[0].Extensions)
		if err != nil {
			return err
		}
		if len(n) > 0 {
			identity = n[0]
		}
	}

	if wl.RBAC == nil {
		return nil
	}
	var namespace string
	if identity != "" {
		s, err := spiffe.ParseIdentity(identity)
		if err != nil {
			return err
		}
		namespace = s.Namespace
	}

	log := log.WithLabels("ip", pip.String(), "ident", identity, "port", dport)

	// This is from the PEP, which handles this already
	// TODO: make this check more robust
	if wl.RemoteProxy != (netip.Addr{}) && identity == wl.Identity {
		if len(wl.RBAC.Allow) == 0 {
			log.Debugf("allow (no policies)")
		} else {
			log.Infof("allow (from remote)")
		}
		return nil
	}
	deny := rbacMatch(wl.RBAC.Deny, namespace, identity, dport)
	if deny {
		// TODO context
		return fmt.Errorf("RBAC: deny")
	}
	if len(wl.RBAC.Allow) == 0 {
		log.Debug("allow (no policies)")
		return nil
	}
	allow := rbacMatch(wl.RBAC.Allow, namespace, identity, dport)
	if allow {
		log.Info("allow")
		return nil
	}
	return fmt.Errorf("RBAC: no allow matched")
}

func rbacMatch(pol []*uproxyapi.Policy, namespace string, identity string, dports string) bool {
	dport, _ := strconv.Atoi(dports)
	for _, pol := range pol {
		// TODO pol.When
		ruleMatch := false
		for _, rule := range pol.Rule {
			rmatch := true
			if rule.Namespace != "" && rule.Namespace != namespace {
				rmatch = false
			}
			if rule.Identity != "" && "spiffe://"+rule.Identity != identity {
				rmatch = false
			}
			if rule.Invert {
				rmatch = !rmatch
			}
			ruleMatch = ruleMatch || rmatch
		}
		whenMatch := false
		if len(pol.When) == 0 {
			whenMatch = true
		}
		for _, when := range pol.When {
			rmatch := true
			if when.Port != 0 && when.Port != uint32(dport) {
				rmatch = false
			}
			if when.Invert {
				rmatch = !rmatch
			}
			whenMatch = whenMatch || rmatch
		}
		if ruleMatch && whenMatch {
			return true
		}
	}
	return false
}

Likely needs a lot of work though.

Add tests for pathological cases

slow loris attacks, spiky workloads, etc)

Setup cargo audit (or similar) to report dependency CVEs and validate license

Developer-Facing Simple Counters

Can be cheaply incremented (no locking, etc). Used
primarily for devs to see how many times some
codepath is hit. Not “prometheus-y” metrics (e.g.
byte counts, etc)

It should be incredibly easy to create a counter in the code and update it.

• Number that can be incremented cheaply at runtime.
• istioctl should include a tool that shows value of all counters. Should also show the rate of change.
• It should not be possible to disable counters, these aren't like log levels, they should be on all the time.
• Debug tool should dump the current value of all counters.
• Should be able to watch counters.
• Counters should include other major events
  • Errors
  • Cleanup work or other debugging stuff.
  • Control Plane Changes

Vulnerability Testing

Automated tests of basic attacks:
SYN flood

Ping/ICMP flood

Slow loris (are we vulnerable to L7 things here?)

The libraries we use may have sufficient testing depending on the layer each attack targets.

[Investigation] Identify and test unhappy paths.

We should identify all of the “bad” cases we can possibly take and make sure to have tests for them and we can verify sane behavior.

Examples:

Sending to a closed port

Note: I tested this and we hang for ~60-90 seconds before timing out.

Sending to an unroutable address

Sending to a unresolvable VIP

… others to be investigated

Setup prow to cache rustc outputs

Enable building on arm64

Currently, the build fails since we use the FIPS boringssl which only works on amd64.

Even with the FIPS disabled though, still getting issues around poly_rq_mul.S that we will need to resolve

Setup ProxyConfig (and other config) reading, and pass this config to various components.

Implement connection pooling

Use pilot-agent

Using a custom-build proxy is great - but I don't think we need to lose all the integrations and support we built in pilot-agent.

• We should continue to use SDS to get the certs - and let pilot-agent handle the integrations, including the special call to Citadel,
  but other CAs or Spire may also support the same mechanisms, no need to hardcode citadel and the k8s token.
• Istiod address should be the UDS socket, as before - let agent deal with finding the credentials and root and address of istiod
• DNS and other things should continue to work if needed.

This preserve integrations - and also simplifies the code in ztunnel.

In time we may decide to rewrite all that in rust as well - but one component at a time.

Various traffic issues in Kubernetes

Trying to run our existing integration tests on my linux machine in KinD (istio/istio#41633)

• Traffic to a DNS shortname doesn't resolve

I have no name!@captured-v1-54f5cc6f5c-7rl45:/$ curl captured:80     
curl: (6) Could not resolve host: captured

• Traffic to a VIP with a bad port (e.g. 1234 instead of 80) hangs instead of fails (vip + good port succeeds). It does eventually timeout (~60s?), giving the client a "connection reset by peer".

2022-10-26T18:22:29.926979Z  INFO ztunnel::proxy::outbound: accepted outbound connection from [::ffff:10.244.1.12]:34202
2022-10-26T18:22:29.927015Z  INFO ztunnel::proxy::outbound: Proxying to 10.96.18.21:123 using TCP via 10.96.18.21:123 type Passthrough

• Semi-frequent xds stream breakages. Possibly related to the above (I've only seen it immediately after one of these bad attempts)

2022-10-26T18:23:38.889583Z  WARN ztunnel::xds::client: XDS client error: gRPC error (Unknown error): error reading a body from connection: broken pipe, retrying
2022-10-26T18:23:38.889606Z  INFO ztunnel::xds::client: Starting ADS client with 21 workloads

Minimal support for SOCKS5

SOCKS is a very simple protocol, easy to implement (probably already has rust impl). This can be optionally compiled in.

It will make it very easy to do local development without eBPF/interception/root - and using common tools (virtually all
apps support an option to use a socks proxy).

Happy to work on it.

Implement Cert Verification

Currently, we present certs and encrypt, but we
do not verify the root cert or spiffe://. We must have
this on both client/server.

Audit existing unit test coverage for all components

Tests should just be isolated ztunnel tests; full Kubernetes e2e tests will be handled by the existing (Go) integ test suite.

Implement proper load balancing

Is there any option/config to support upstream selection while VIP upstream has many wordload endpoint?

Setup prow tests for ztunnel repo

Admin API improvements

Split readiness probe and admin interface; admin interface should only be over localhost or UDS.

Figure out how to ensure fairness between clients

Support for TUN interception in ztunnel

TUN generally requires an IP stack - in go I've used lwIP and gvisor, AFAIK lwIP is also supported in Rust ( at least Rust embedded).

It is possible to use Tun without root - by setting the owner of the tun device. This is intended for running zTunnel on VMs or
on Android, as well as in cases where Tproxy + eBPF are not available.

Connections being pooled per-zTunnel pair

HBONE Connections between zTunnels from the same source and destination Identities should be reusable across multiple underlying client connections (streams inside HBONE).

Basic Traffic Tests Pass

Combinations of captured/uncaptured all work. All of our existing integration tests in ambient/ that don’t use waypoints should pass with the new zTunnel.

Profiling

The admin interface should expose endpoints for:
CPU profiling

Heap profiling (#19)

Allocation profiling

Additionally, we should have tools/scripts to easily generate flame graphs from the output.

Support for pprof format will allow piggybacking on some of Envoy’s tooling.

Setup heap profiling support

And make sure existing CPU profiling still works

Implement Hairpinning

Traffic sent from outside ambient to a Pod in ambient should use the Waypoint if available.

There are some cases guarded with Skip in existing tests, but that might not be comprehensive.

Example:
Traffic arrives on the inbound-plaintext port of a uProxy, but the destination is part of an ambient mesh; we should either deny the traffic, or initiate a tunnel.

For this milestone, we likely just want to deny it. Well-behaved traffic should use proper ingress.

Implement graceful shutdown

Audit .unwrap and .expect usages; replace with proper error handling where needed.

Audit .clone() usages (especially for String); replace with references where possible

testnewlines

a\nb

Tune HTTP2 settings

Develop certificate provider module that can reach out to Istiod CA over gRPC and fetch and rotate certificates

Either add rust support to build-tools, build-tools-proxy, or make a build-tools-rust

Add microbenchmarks

Fail-open vs Fail-closed without a control-plane

If you boot up and have no connection to the control
plane, do you forward everything in plain text, or do
you drop all traffic?

Decision: fail-closed only.

Support IPv6 and Dualstack

Research optimal threading configuration, and implement it.

Likely a dedicated admin thread, and 1-2 worker threads.

Tracing

Trace should give extreme detail on every decision made for every single connection. Essentially super verbose connection logging. Follow prior art in Envoy.

There should be a mechanism to enable TRACE level logging for traffic to/from a single IP, or a single service account.

Trace should have no cost when not enabled

Expose stats handler on admin server for prometheus

Depends on #12; they may be 1 task depending on the impl

Allow providing static YAML config

Command line flags can be provided as part of “bootstrap” config (flags still take precedence)

Any xDS objects can be specified here

It can be used in conjunction with xDS

Tune TLS settings

Should use a strict cipher set and TLS 1.3 only

Audit tracing usage to ensure we have enough logs, at the right level.

Only compiles on Linux?

I tried compiling on mac m1 and no go because of linux only configuration present.

On Ubuntu I had to install librust-clang-sys-dev and protobuf-compiler packages to get things working there.

Implement stats object that can store stats from each request