istio / ztunnel Goto Github PK

SOCKS is a very simple protocol, easy to implement (probably already has rust impl). This can be optionally compiled in.

It will make it very easy to do local development without eBPF/interception/root - and using common tools (virtually all
apps support an option to use a socks proxy).

Happy to work on it.

I tried compiling on mac m1 and no go because of linux only configuration present.

On Ubuntu I had to install librust-clang-sys-dev and protobuf-compiler packages to get things working there.

Split readiness probe and admin interface; admin interface should only be over localhost or UDS.

If you boot up and have no connection to the control
plane, do you forward everything in plain text, or do
you drop all traffic?

Decision: fail-closed only.

Currently, the build fails since we use the FIPS boringssl which only works on amd64.

Even with the FIPS disabled though, still getting issues around poly_rq_mul.S that we will need to resolve

slow loris attacks, spiky workloads, etc)

a\nb

This was an initial attempt in Go:

unc (p *Proxy) AssertRBAC(r *http.Request) error {
	ip, dport, err := net.SplitHostPort(r.Host)
	if err != nil {
		return err
	}
	pip, err := netip.ParseAddr(ip)
	if err != nil {
		return err
	}
	wl := p.ConnectionManager.FindWorkloadByAddr(pip)
	var identity string
	if r.TLS != nil && len(r.TLS.PeerCertificates) > 0 {
		n, err := util.ExtractIDs(r.TLS.PeerCertificates[0].Extensions)
		if err != nil {
			return err
		}
		if len(n) > 0 {
			identity = n[0]
		}
	}

	if wl.RBAC == nil {
		return nil
	}
	var namespace string
	if identity != "" {
		s, err := spiffe.ParseIdentity(identity)
		if err != nil {
			return err
		}
		namespace = s.Namespace
	}

	log := log.WithLabels("ip", pip.String(), "ident", identity, "port", dport)

	// This is from the PEP, which handles this already
	// TODO: make this check more robust
	if wl.RemoteProxy != (netip.Addr{}) && identity == wl.Identity {
		if len(wl.RBAC.Allow) == 0 {
			log.Debugf("allow (no policies)")
		} else {
			log.Infof("allow (from remote)")
		}
		return nil
	}
	deny := rbacMatch(wl.RBAC.Deny, namespace, identity, dport)
	if deny {
		// TODO context
		return fmt.Errorf("RBAC: deny")
	}
	if len(wl.RBAC.Allow) == 0 {
		log.Debug("allow (no policies)")
		return nil
	}
	allow := rbacMatch(wl.RBAC.Allow, namespace, identity, dport)
	if allow {
		log.Info("allow")
		return nil
	}
	return fmt.Errorf("RBAC: no allow matched")
}

func rbacMatch(pol []*uproxyapi.Policy, namespace string, identity string, dports string) bool {
	dport, _ := strconv.Atoi(dports)
	for _, pol := range pol {
		// TODO pol.When
		ruleMatch := false
		for _, rule := range pol.Rule {
			rmatch := true
			if rule.Namespace != "" && rule.Namespace != namespace {
				rmatch = false
			}
			if rule.Identity != "" && "spiffe://"+rule.Identity != identity {
				rmatch = false
			}
			if rule.Invert {
				rmatch = !rmatch
			}
			ruleMatch = ruleMatch || rmatch
		}
		whenMatch := false
		if len(pol.When) == 0 {
			whenMatch = true
		}
		for _, when := range pol.When {
			rmatch := true
			if when.Port != 0 && when.Port != uint32(dport) {
				rmatch = false
			}
			if when.Invert {
				rmatch = !rmatch
			}
			whenMatch = whenMatch || rmatch
		}
		if ruleMatch && whenMatch {
			return true
		}
	}
	return false
}

Likely needs a lot of work though.

Cheap, in memory, stores some list of past events, similar to Envoy access logs.

There should be a cheap in-memory connection log that's accessible for debugging purposes. The runtime cost of maintaining this log should be extremely low. This could be achieved by a per-thread in-memory list of events including the 5-tuple of the connection, what happened (open, close, error, pkt/bytes sent) and a timestamp. Whenever something happens on a connection, we can add to this list a note at very low cost. There could be a configurable limit to the size of the thing.
With this, when debugging a user could run a ztctl command to explore this data. They should be able to pass in an IP or 5-tuple and see all connections that have occurred recently with that configuration. There should be a watch command that shows the connections currently flowing through the system as they come and go. The debug dump tool should dump every thread's connection log so that it can be analyzed off-box by a support engineer.

Trace should give extreme detail on every decision made for every single connection. Essentially super verbose connection logging. Follow prior art in Envoy.

There should be a mechanism to enable TRACE level logging for traffic to/from a single IP, or a single service account.

Trace should have no cost when not enabled

We should identify all of the “bad” cases we can possibly take and make sure to have tests for them and we can verify sane behavior.

Examples:

Sending to a closed port

Note: I tested this and we hang for ~60-90 seconds before timing out.

Sending to an unroutable address

Sending to a unresolvable VIP

… others to be investigated

Likely a dedicated admin thread, and 1-2 worker threads.

This is already done.

Traffic sent from outside ambient to a Pod in ambient should use the Waypoint if available.

There are some cases guarded with Skip in existing tests, but that might not be comprehensive.

Example:
Traffic arrives on the inbound-plaintext port of a uProxy, but the destination is part of an ambient mesh; we should either deny the traffic, or initiate a tunnel.

For this milestone, we likely just want to deny it. Well-behaved traffic should use proper ingress.

Currently, we present certs and encrypt, but we
do not verify the root cert or spiffe://. We must have
this on both client/server.

Should we have access logs similar to envoy?
How configurable should they be?

TUN generally requires an IP stack - in go I've used lwIP and gvisor, AFAIK lwIP is also supported in Rust ( at least Rust embedded).

It is possible to use Tun without root - by setting the owner of the tun device. This is intended for running zTunnel on VMs or
on Android, as well as in cases where Tproxy + eBPF are not available.

Tests should just be isolated ztunnel tests; full Kubernetes e2e tests will be handled by the existing (Go) integ test suite.

Is there any option/config to support upstream selection while VIP upstream has many wordload endpoint?

HBONE Connections between zTunnels from the same source and destination Identities should be reusable across multiple underlying client connections (streams inside HBONE).

Depends on #12; they may be 1 task depending on the impl

since ztunnel is running in standalone mode per node, any plan to support HA for it?

Command line flags can be provided as part of “bootstrap” config (flags still take precedence)

Any xDS objects can be specified here

It can be used in conjunction with xDS

Should use a strict cipher set and TLS 1.3 only

Automated tests of basic attacks:
SYN flood

Ping/ICMP flood

Slow loris (are we vulnerable to L7 things here?)

The libraries we use may have sufficient testing depending on the layer each attack targets.

a\nb

Using a custom-build proxy is great - but I don't think we need to lose all the integrations and support we built in pilot-agent.

• We should continue to use SDS to get the certs - and let pilot-agent handle the integrations, including the special call to Citadel,
  but other CAs or Spire may also support the same mechanisms, no need to hardcode citadel and the k8s token.
• Istiod address should be the UDS socket, as before - let agent deal with finding the credentials and root and address of istiod
• DNS and other things should continue to work if needed.

This preserve integrations - and also simplifies the code in ztunnel.

In time we may decide to rewrite all that in rust as well - but one component at a time.

The admin interface should expose endpoints for:
CPU profiling

Heap profiling (#19)

Allocation profiling

Additionally, we should have tools/scripts to easily generate flame graphs from the output.

Support for pprof format will allow piggybacking on some of Envoy’s tooling.

Trying to run our existing integration tests on my linux machine in KinD (istio/istio#41633)

• Traffic to a DNS shortname doesn't resolve

I have no name!@captured-v1-54f5cc6f5c-7rl45:/$ curl captured:80     
curl: (6) Could not resolve host: captured

• Traffic to a VIP with a bad port (e.g. 1234 instead of 80) hangs instead of fails (vip + good port succeeds). It does eventually timeout (~60s?), giving the client a "connection reset by peer".

2022-10-26T18:22:29.926979Z  INFO ztunnel::proxy::outbound: accepted outbound connection from [::ffff:10.244.1.12]:34202
2022-10-26T18:22:29.927015Z  INFO ztunnel::proxy::outbound: Proxying to 10.96.18.21:123 using TCP via 10.96.18.21:123 type Passthrough

• Semi-frequent xds stream breakages. Possibly related to the above (I've only seen it immediately after one of these bad attempts)

2022-10-26T18:23:38.889583Z  WARN ztunnel::xds::client: XDS client error: gRPC error (Unknown error): error reading a body from connection: broken pipe, retrying
2022-10-26T18:23:38.889606Z  INFO ztunnel::xds::client: Starting ADS client with 21 workloads

Combinations of captured/uncaptured all work. All of our existing integration tests in ambient/ that don’t use waypoints should pass with the new zTunnel.

Can be cheaply incremented (no locking, etc). Used
primarily for devs to see how many times some
codepath is hit. Not “prometheus-y” metrics (e.g.
byte counts, etc)

It should be incredibly easy to create a counter in the code and update it.

• Number that can be incremented cheaply at runtime.
• istioctl should include a tool that shows value of all counters. Should also show the rate of change.
• It should not be possible to disable counters, these aren't like log levels, they should be on all the time.
• Debug tool should dump the current value of all counters.
• Should be able to watch counters.
• Counters should include other major events
  • Errors
  • Cleanup work or other debugging stuff.
  • Control Plane Changes

And make sure existing CPU profiling still works