zio / zio-keeper

A ZIO library for building distributed systems

ZIO Keeper

ZIO Keeper is a purely-functional, type-safe library for building distributed systems.

Introduction

It provides numerous primitives for tackling the common problems in distributed computing (e.g. leader election, cluster forming etc.).

Under the hood, the library is backed by ZIO and ZIO NIO, profiting from their performant, type and resource-safe APIs:

• Composable. Design complex systems by composing the available building blocks.
• Resilient. Build apps with automated failure recovery.
• Secure. Benefit from security guarantees built into the library core.

From the high-level perspective, the library can be separated into the following "modules":

• transport
• membership
• consensus

Installation

In order to use this library, we need to add the following line in our build.sbt file:

libraryDependencies += "dev.zio" %% "zio-keeper" % "<version>"

resolvers += Resolver.sonatypeRepo("snapshots")

Documentation

Learn more on the ZIO Keeper homepage!

Contributing

For the general guidelines, see ZIO contributor's guide.

Code of Conduct

See the Code of Conduct

Support

Come chat with us on .

License

License

zio-keeper's Issues

TLS transport

We should add TLS version of transport that could be mixin with TCP or UDP transports.

Add correlationId to message

All messages in the cluster should have identifier. When message are part of the same "transaction" they should have the same prefix in order to group them together.

Automate the release process

Every merge to develop or master should trigger build and publishing of the snapshot and stable releases, respectively.

Implement Gossip

Before obtaining an access to metadata API, one has to join the cluster and obtain the public "access protocol" (see #2). Gossip protocol should be used to exchange the data necessary for successful bootstrap.

Note: Due to the size of this issue, the preferable approach is to split the implementation in multiple pull requests.

Define CRDTs

The decision has been made to use conflict-free replicated data types (CRDTs) for providing consistency guarantees. The purpose of this issue is to make an agreement upon the data structures and the strategy (state vs operational) that we will provide in the first release.

Prerequisites

In case you are not familiar with the concept, the following videos provide a (relatively) gentle introduction:

• CRDTs illustrated
• Strong Eventual Consistency and Conflict-free Replicated Data Types
• CRDTs in Practice

For an in-depth understanding of CRDTs, please make sure to check out this paper.

HyParView: e2e tests

Ensure HyParView is working e2e using real infrastructure and if possible add a separate test suite for it.

Explore plumtree for broadcasts on top of hyparview

Create a microsite

Document and publish a microsite using mdoc. Use ZIO's own documentation as reference.

Add UDP transport

At the moment we support reliable transport via TCP. Ideally, we should support the best effort, UDP, transport.

Security concerns

The current version of library API ignores any potential security issues (e.g. unauthorized access to cluster / data). This issue should be used to enumerate major vulnerabilities, exchange relevant papers and videos, and to come up with the draft of the security policy. Once that's in place, the follow-up implementation tickets will be created.

Complete the SWIM implementation

The current implementation of SWIM is missing handling of suspicious part of the protocol.

Upgrade to ZIO 2.0

ZIO is at Milestone 4, with an RC expected in the next few weeks.
https://github.com/zio/zio/releases/tag/v2.0.0-M4

The API is nearly stable at this point, so any early migration work against this version should pay off towards the official 2.0 release.

The progress is being tracked here:
zio/zio#5470

The Stream Encoding work in progress is the only area where the API might still change before the RC.

To assist with the migration, follow this guide:
https://zio.dev/howto/migrate/zio-2.x-migration-guide/

We are actively working on a ScalaFix rule that will cover the bulk of the simple API changes:
https://github.com/zio/zio/blob/series/2.x/scalafix/rules/src/main/scala/fix/Zio2Upgrade.scala

If you would like assistance with the migration from myself or other ZIO contributors, please let us know!

Clean the code

The current implementation can be polished in terms of cleanliness. The things that I'd propose doing in the first run are:

• ensure return type is specified for all functions
• order functions alphabetically
• extract versions of ZIO to variables in build.sbt

try ScalaPB for serialization

Error when running TcpServer example

When I try to run TcpServer example with the command:

sbt "examples/runMain zio.keeper.example.TcpServer"

I got an error:

Failed to instantiate SLF4J LoggerFactory
Reported exception:
java.lang.NoClassDefFoundError: ch/qos/logback/core/joran/spi/JoranException
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
	at zio.logging.slf4j.Slf4jLogger.$anonfun$logger$1(Slf4jLogger.scala:18)
	at zio.internal.FiberContext.evaluateNow(FiberContext.scala:333)
	at zio.internal.FiberContext.$anonfun$fork$2(FiberContext.scala:655)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: ch.qos.logback.core.joran.spi.JoranException
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:98)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 10 more

Looks like logger lib is missing, but on the pther hand I can start Node1 without any problems and it logs.

README Update

our README is outdated.

Membership notification

When nodes are joining, leaving and become unreachable each of node should be notify.
We defined those events in https://github.com/zio/zio-keeper/blob/develop/src/main/scala/zio/keeper/MembershipEvent.scala.

Integrate PeerSim into the project

In order to ensure that the implementations are reliable we should use something like PeerSim to run simulations. Scope of this ticket is to figure out a way to test Membership layer implementations using it

Implement InMemoryTransport

Implementation of Transport that works completely in memory. It additionally support a percentMessageDrop parameter that causes a certain percentage of all messages to be dropped.
Furthermore it should have an api that allows to simulate network partitions between hosts.

Document the library

Write a clear statement of purpose and provide a high-level architecture overview. Enlist all of the resources (videos, papers, books) as part of the "getting started" guide.

shards and partitions

shards and partitions will be pretty useful when it comes to build any distributed storage mechanism.

Upgrade ZIO to 1.0.0-RC17

Fix all errors present in the current codebase that prevent automatic migration.

Client library

This is topic that we should discuss. Do we need client library?

Failing tests

Hi, I just cloned this project and was trying to execute tests by running:

sbt
test

But it looks some of them fail on my computer:

Failed tests:
[error] 	zio.keeper.membership.SwimSpec
[error] 	zio.keeper.protocol.InternalProtocolSpec
[error] 	zio.keeper.transport.TransportSpec
[error] 	zio.keeper.membership.GossipStateSpec
[error] (keeper / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] (membership / Test / test) sbt.TestsFailedException: Tests unsuccessful

Should I run them in different way?

Adding metrics

one of most important part of distributed system is observability that's why we should build our library with this mindset.

Probably we should try to use https://github.com/zio/zio-metrics if it is not ready we should join them to help them release it.

@jdegoes @mijicd Let me know if you see other options.

Access metadata using optics

Library design assumes usage of Lens and Prism for metadata access (see #2). Since one of the goals of scalaz-distributed is to minimize the number of external dependencies, it is not an option to use Monocle.

Provide an implementation of Lens and Prism. In case additional typeclasses are introduced, briefly explain their purpose in the context of scalaz-distributed.

Implement Membership layer using HyParView protocol

Remove dependency to scalaz-sbt

Instead of relying on common scalaz build configuration, encode the config explicitly. Side effect of this change is re-enabling the cross build that was disabled due to 2.11 flags incompatibility.

HyParView: Missing Tests

Tests need to be expanded for:

• active Protocol
• PlumTree module

Replace Travis with CircleCI

Travis has become quite clunky, and the builds last very long. Migrate the CI pipeline to CircleCI, as it offers more resources for open source projects, has a decent configuration syntax and a proven track record.

Align the project structure with other scalaz projects

Project structure needs to be aligned with other projects. To do that, the following criteria must be met:

• no dependencies other than zio
• no unused plugins
• same formatting rules (.scalafmt.conf)
• CI pipeline aligned with ZIO

If possible, use SBT plugin.

Data serialization

Nodes will exchange a lot of operational and metadata over network. We should try to minimize the traffic and reduce the serialization / deserialization footprint as much as possible. The focus of this issue is to design an (close to) optimal message representation, and decide on the serialization format.

In my opinion, some of the strong candidates are Protobuf, Avro, and Thrift, but there are many more available. We should keep an eye on scalaz-schema as it will eventually provide all necessary "tools".

Upgrade ZIO to 1.0.0-RC18

Due to the large amount of new features, and delay of RC18 (or 1.0.0 :)), we will depend on snapshot version in order to align our APIs with the latest changes and prepare for an official release.

Design docs

Structure the notes gathered over Berlin hackathon.

OOM when running example

If I start Node1 only everything is ok.
But if I also start Node2, then after some time (5-10 minutes) Node1 will crash with:

java.lang.OutOfMemoryError: Java heap space

Implement caching for transport

Transport should keep connection open for a certain time when sending and reuse them if sending to the same node again. One way of implementing this is to have a Ref[Map[Address, Channel]] that is created on construction of the transport and periodically garbage collected.

Leader election

we should support leader election to be able to assign role to particular node in the cluster.

Design errors hierarchy

API design (see #2) introduced sealed errors hierarchy. Expand the draft idea by providing additional members of error hierarchy.

Plumtree tree optimization

I'm leaving this out of #180 for now.

Should be quite easy to do, but we need to decide what policy to adopt. There is one recommended in the paper, but I've seen projects use other, more sophisticated ones

Upgrade zio-nio version

The current implementation of zio-keeper uses zio-nio:0.1.2. To be able to upgrade the version, and utilise the latest features shipped in zio-nio, zio-keeper internals needs to be updated to work with ZManaged.

Adding logging

currently we log to console using zio.console.Console which obviously is not production solution. We may use some of existing libs for that.

Potential candidates are:

1. https://github.com/NeQuissimus/zio-slf4j - it's depending on scalaz and since we are trying to minimize dependency we probable don't want to use it.
2. https://izumi.7mind.io/latest/release/doc/logstage/ - it seems to work and it's documented

other option would be to create zio-logger project with very simple interface that would be easy to plug to slf4j or whatever.

@jdegoes @mijicd Thoughts?

Publish 2.13 release

Align the build configuration with other projects in the organization and enable 2.13 artifact publishing.

Upgrade to zio-nio 0.4.0

Modules introduced in #96 depend on different versions of zio-nio. To facilitate release of Scala 2.13 (#108), we need to use the latest version of zio-nio.

local development environment automation

we should have docker compose or terraform script to setup cluster locally for testing

Use STM.foreach_ wherever possible

Can only be resolved once we have upgraded ZIO version

Update package hierarchy

Due to the recent organization changes, all artifacts should be moved from scalaz.ziokeeper to zio.keeper.