zio / zio-connect

Sources, Sinks and Pipelines for channeling data

Project Stage	CI	Release	Snapshot	Issues

Scaladoc	Scaladex	Discord	Twitter	Gitpod
Scaladoc

Welcome to ZIO

ZIO is a zero-dependency Scala library for asynchronous and concurrent programming.

Powered by highly-scalable, non-blocking fibers that never waste or leak resources, ZIO lets you build scalable, resilient, and reactive applications that meet the needs of your business.

• High-performance. Build scalable applications with minimal runtime overhead.
• Type-safe. Use the full power of the Scala compiler to catch bugs at compile time.
• Concurrent. Easily build concurrent apps without deadlocks, race conditions, or complexity.
• Asynchronous. Write sequential code that looks the same whether it's asynchronous or synchronous.
• Resource-safe. Build apps that never leak resources (including threads!), even when they fail.
• Testable. Inject test services into your app for fast, deterministic, and type-safe testing.
• Resilient. Build apps that never lose errors, and which respond to failure locally and flexibly.
• Functional. Rapidly compose solutions to complex problems from simple building blocks.

To learn more about ZIO, see the following references:

• Homepage
• Contributor's Guide
• License
• Issues
• Pull Requests

---

Adopters

Following is a partial list of companies happily using ZIO in production to craft concurrent applications.

Want to see your company here? Submit a PR!

• AdGear / Samsung Ads
• Adidas
• adpulse.io
• adsquare
• Amazon
• Anduin Transactions
• AnyMind Group
• Ayolab
• Asana
• Aurinko
• auto.ru
• AutoScout24
• Avast
• Bank of America
• Believe
• Blue Insight Digital
• BlueLabs
• Bpp
• Broad
• Caesars Digital
• CalcBank
• Call Handling
• Carvana
• Cellular
• ChannelPilot
• Chartboost
• Cloudfarms
• CodeComprehension
• Codept
• Colisweb
• Collibra
• Compellon
• Complicated Robot
• Conduktor
• Contramap
• Coralogix
• Credit Karma
• CurrencyCloud
• D.E.Solution
• DaSCH
• DataChef
• Demandbase
• Demyst
• Devsisters
• DHL Parcel The Netherlands
• Disney+ Streaming
• Doomoolmori
• Dow Jones
• DPG recruitment
• Dream11
• Deutsche Telekom IoT GmbH
• eBay
• Eaglescience
• Electricité de France (EDF)
• EnelX
• Evolution
• Evo
• Flipp
• Fugo.ai
• Garner Distributed Workflow
• Glean
• GrandParade
• greyflower.media GmbH
• Hunters.AI
• Hypefactors
• iHeartRadio
• IHS Markit
• Investsuite
• Invia
• Kaizen Solutions
• Kamon APM
• Kodmagi
• Kensu
• LambdaWorks
• LeadIQ
• Lernkunst
• LiveIntent Inc.
• Lottoland
• MATECHS
• Megogo
• MOIA
• Mylivn
• NewMotion
• Nexxchange
• Nike
• NsLookup
• Ocado Technology
• Olyro GmbH
• Optrak
• Performance Immo
• Pirum
• PITS Global Data Recovery Services
• Playtika
• PPC Samurai
• Prezi
• Radix Labs
• Railroad19
• Randstad Groep Nederland
• Rapidor
• PIM Solutions
• Remember The Milk
• REWE Digital
• LKWfee
• Risk Ident
• Rocker
• Rudder
• Samsung Food
• Sanjagh
• Scalac
• SecurityScorecard
• SentinelOne
• Signicat
• Société Générale Corporate and Investment Banking
• SoftwareMill
• StreamWeaver
• Stuart
• Teads
• The Pokemon Company International
• TomTom
• Tinka
• Tinkoff
• Trackabus
• Trainor
• Tranzzo
• TreuTech
• Tweddle Group
• Undo
• Unit
• Univalence
• Unzer
• Vakantiediscounter
• Verbund AG
• Waylay
• Wehkamp
• Wefunder
• Wolt
• Yandex.Classifieds
• Audela
• Valamis Group
• Valsea
• VirtusLab
• Vish
• Vivid Money
• Zalando
• Zooz

Sponsors

Ziverge is a leading contributor to ZIO.

Scalac sponsors ZIO Hackathons and contributes work to multiple projects in ZIO ecosystem.

Septimal Mind sponsors work on ZIO Tracing and continuous maintenance.

YourKit generously provides use of their monitoring and profiling tools to maximize the performance of ZIO applications.

---

Learn More on the ZIO Homepage

---

Code of Conduct

See the Code of Conduct

---

Support

Come chat with us on .

---

Legal

Copyright 2017 - 2024 John A. De Goes and the ZIO Contributors. All rights reserved.

zio-connect's Issues

FS2Connector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.fs2
  

• Create the connector

   sealed trait FS2Connector
   
   case class LiveFS2Connector() extends FS2Connector { ... }
   
   object LiveFS2Connector {
       val layer: ZLayer[???, ???, FS2Connector] = {
           ...
           LiveFS2Connector()
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the FS2ConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a LiveFS2ConnectorSpec that runs the CassandraConnectorSpec files for the LiveCassandraConnector layer
  Example with testcontainers and S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestFS2Connector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

FTPConnector - new connector and documentation

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.ftp
  

• Create the connector

   sealed trait FTPConnector
   
   case class LiveFTPConnector() extends FTPConnector { ... }
   
   object LiveFTPConnector {
       val layer: ZLayer[???, ???, FTPConnector] = {
           ...
           LiveFTPConnector()
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the FTPConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a LiveFTPConnectorSpec that runs the FTPConnectorSpec files for the LiveFTPConnector layer
  You could use the https://hub.docker.com/r/atmoz/sftp/ container with testcontainers.
  Example with testcontainers and Localstack S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestFTPConnector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

CouchbaseConnector - documentation

Setup scalafix

We would like to have the config for scalafix. It should be the same as in the other zio projects.

MongoConnector - documentation

Maintenance: Replace the old Dotty dependencies with proper Scala 3 ones

S3Connector - support for multiple regions

Context - #83 (review)

FileConnector.deleteFile

Update to zio2

MongoConnector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.mongo
  

• Create the connector

   sealed trait MongoConnector
   
   case class LiveMongoConnector() extends MongoConnector { ... }
   
   object LiveMongoConnector {
       val layer: ZLayer[???, ???, MongoConnector] = {
           ...
           LiveMongoConnector()
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the MongoConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/MongoConnectorSpec.scala

• There will be a LiveMongoConnectorSpec that runs the MongoConnectorSpec files for the LiveMongoConnector layer
  You can use testcontainers - https://www.testcontainers.org/modules/databases/mongodb/
  Example with testcontainers and localstack S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestMongoConnector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestMongoConnector.scala

CouchbaseConnector - new connector

Xml Connector

@jypma started with zio-xml

Alpakka also has a connector like this, would be cool to add it as well

https://github.com/jypma/zio-xml

Also a XPath-like combinator would be a cool idea, to stream a certain subset (where you know each node set is bounded) of scala.xml.Elem from a large xml file

S3Connector - use subtype instead of newtype and refactor out redundant _.toString calls

The existing implementation uses subtype for the connector domain types. This leads to a lot of use of _.toString. We could achieve the same level of typesafety with Subtype and without having to call _.toString everythere.

The new implementation would look like this

  object BucketName extends Subtype[String] //previously was Newtype[String]
  type BucketName = BucketName.Type

  object ObjectKey extends Subtype[String] //previously was Newtype[String]
  type ObjectKey = ObjectKey.Type

FileConnector.moveFile

FirehoseConnector - documentation

couchbase-connector - add a version of get that also returns cas and a version of replace that uses cas

Context

• We want to add a versions of get that also return cas and a version of replace that uses cas. That way we can do smth like 'getWithCas >>> process >>> replaceWithCas'.
• https://docs.couchbase.com/java-sdk/current/howtos/concurrent-document-mutations.html

Upgrade to ZIO 2.0.3

KafkaConnector - new connector

DynamoDBConnector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.dynamobd
  

• Create the connector

   sealed trait DynamoDBConnector
  
   case class AWSDynamoDBConnector() extends DynamoDBConnector { ... }
   
   object AWSDynamoDBConnector {
       val layer: ZLayer[???, ???, DynamoDBConnector] = {
           ...
           AWSDynamoDBConnector()
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the DynamoDBConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a AWSDynamoDBConnectorSpec that runs the DynamoDBConnectorSpec files for the AWSDynamoDBConnector layer
  You can use it with localstack testcontainers.
  Example with testcontainers and S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestDynamoDBConnector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

AWSLambdaConnector - new connector

JMSConnector - documentation

CassandraConnector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.cassandra
  

• Create the connector

   sealed trait CassandraConnector
   
   case class LiveCassandraConnector() extends CassandraConnector { ... }
   
   object LiveCassandraConnector {
       val layer: ZLayer[???, ???, CassandraConnector] = {
           ...
           LiveCassandraConnector()
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the CassandraConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a LiveCassandraConnectorSpec that runs the CassandraConnectorSpec files for the LiveCassandraConnector layer
  You can use cassandra with testcontainers - https://www.testcontainers.org/modules/databases/cassandra/
  Example with testcontainers and S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestCassandraConnector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

Multi artifact publishing

We want to be able to publish a separate artifact for each connector. This will limit imports to only the necessary dependencies.
(we can use https://zio.github.io/zio-aws/docs/overview/overview_artifacts as a model)

Add a comprehensive example for FileConnector

The tests are a good example but it would be better for new users if they see an explicit examples module

Add support for jdk8 in CI

There are still a lot of users of jdk8 and we should support them also.

Get the non aggregated connectors to a point where they can be included in the aggregate

• kafka-connector
• kinesis-datastreams-connector

Use us-east-2 in the S3Connector tests

We noticed that using us-east-1 region with tests does not trigger BucketAlreadyExists exception with localstack. Could not find an explanation for this bad behaviour.

S3 connector's putObject is wrong

It seems to repeatedly call putObject for each chunk, which just overwrites the same object every time. It should use the multipart upload API: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

Investigate couchbase, s3 connector test flakyness surrounding the live spec test

I noticed that in some PR runs it's flaky, the container is not started or there is a timeout.

Setup formatting to the zio libs standard

S3Connector - documentation

Create TestFileConnector

zio.connect.testkit.file.TestFileConnector - in memory representation of a fileConnector

We want a layer that provides a in memory representation of a FileConnector version.
Users testing code that needs a FileConnector in R will be able to use this instead and avoid touching the filesystem.

AWS SQS - new connector

Using https://github.com/softwaremill/elasticmq for local testing.

FileConnect - Inotify for directory watching

The context is:

"
Does it manages inotify for directory watching? It's such a mess and jdk impl with its OVERFLOW is not helping, no more than not knowing when file is closed, I needed to implement smthg far too much complicated for a rather simple "watch for pair file+signature"

I mean, it remains understandable thanks to zio queue / cancellable effects / etc, but it horribly looks like a sausage factory internals in need for a good and robust api :)
"

PravegaConnector - new connector & documentation

Run tests as part of PR checks

Right now when a PR is created tests are not ran so the PR can be merged even if tests would fail. We want tests to be ran for PR's and not allow merging unless tests pass.

Couchbase Connector docs correct minor details

titles, name in side panel, ...

Update ci to only publish the website on special tags and releases

Now the website is published every time a new PR is merged in master.

RedisConnector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.redis
  

• Create the connector

   sealed trait RedisConnector
   
   case class LiveRedisConnector() extends RedisConnector { ... }
   
   object LiveRedisConnector {
       val layer: ZLayer[???, ???, RedisConnector] = {
           ...
           LiveRedisConnector(...)
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the RedisConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a LiveRedisConnectorSpec that runs the RedisConnectorSpec files for the LiveRedisConnector layer
  You can use testcontainers - maybe this one https://github.com/redis-developer/testcontainers-redis
  Example with testcontainers and localstack S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestRedisConnector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

KinesisDatastreamsConnector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.firehose
  

• Create the connector

   sealed trait FirehoseConnector
   
   case class LiveFirehoseConnector() extends FirehoseConnector { ... }
   
   object LiveFirehoseConnector {
       val layer: ZLayer[???, ???, FirehoseConnector] = {
           ...
           LiveFirehoseConnector()
       }
  
   }
  

• Provide accessors in the package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the FirehoseConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a LiveFirehoseConnectorSpec that runs the FirehoseConnectorSpec files for the LiveFirehoseConnector layer
  You can use firehose with localstack testcontainers.
  Example with testcontainers and S3 here - https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestFirehoseConnector layer that allows testing with a stub.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

JMSConnector - new connector

#Steps

• Add a new project
  -- Example:

  zio-connect/build.sbt

  Line 38 in 2507925

  lazy val fileConnector = project

• Create the following directory structure

     zio.connect.jms
  

• Create the connector

   sealed trait JMSConnector
   
   case class LiveJMSConnector() extends JMSConnector { ... }
   
   object LiveJMSConnector {
       val layer: ZLayer[???, ???, JMSConnector] = {
           ...
           LiveJMSConnector()
       }
  
   }
  

• Provide accessors in the jms package object
  Example: https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/package.scala

• Tests belong in the JMSConnectorSpec file.
  They use the accessors.
  Example:- https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/test/scala/zio/connect/file/FileConnectorSpec.scala

• There will be a LiveJMSConnectorSpec that runs the JMSConnectorSpec files for the LiveJMSConnector layer
  Use https://www.testcontainers.org/modules/rabbitmq/ to set up jms
  Example with testcontainers https://github.com/zio/zio-connect/pull/83/files#diff-e7e0899e455a741a644a05c0b26879d176a9126951477cd7c62dc19ade98b5cc

• Provide a TestJMSConnector layer that allows testing with a stubbed jms.
  Example - https://github.com/zio/zio-connect/blob/master/connectors/file-connector/src/main/scala/zio/connect/file/TestFileConnector.scala

S3Connector - new connector

Implement FileConnector.writeFile

CassandraConnector - documentation

Write tests for LiveFileConnector operations that are untested