7mind / izumi

Productivity-oriented collection of lightweight fancy stuff for Scala toolchain

---

---

What is it?

Izumi (jp. 泉水, spring) is an ecosystem of independent libraries and frameworks allowing you to significantly increase productivity of your Scala development.

including the following components:

1. distage – Compile-time safe, transparent and debuggable multi-modal Dependency Injection framework for pure FP Scala,
2. distage-testkit – Hyper-pragmatic pure FP Test framework. Shares heavy resources globally across all test suites; lets you easily swap implementations of component; uses your effect type for parallelism.
3. distage-framework-docker – A distage extension for using docker containers in tests or for local application runs, comes with example Postgres, Cassandra, Kafka & DynamoDB containers.
4. LogStage – Automatic structural logs from Scala string interpolations,
5. BIO - A typeclass hierarchy for tagless final style with Bifunctor effect types. Focused on ergonomics and ease of use with zero boilerplate.
6. izumi-reflect (moved to zio/izumi-reflect) - Portable, lightweight and kind-polymorphic alternative to scala-reflect's Typetag for Scala, Scala.js, Scala Native and Scala 3
7. IdeaLingua (moved to 7mind/idealingua-v1) – API Definition, Data Modeling and RPC language, optimized for fast prototyping – like gRPC or Swagger, but with a human face. Generates RPC servers and clients for Go, TypeScript, C# and Scala,
8. Opinionated SBT plugins (moved to 7mind/sbtgen) – Reduces verbosity of SBT builds and introduces new features – inter-project shared test scopes and BOM plugins (from Maven)
9. Percept-Plan-Execute-Repeat (PPER) – A pattern that enables modeling very complex domains and orchestrate deadly complex processes a lot easier than you're used to.

Docs

• Documentation
• Scaladoc

Example projects:

• DIStage Example Project
• Idealingua Example Project with TypeScript and Scala

Support Chats:

• Izumi User Group [RU] on Telegram
• Izumi User Group [EN] on Telegram
• Discussions on Github
• Izumi on Gitter

Videos:

• Izumi 1.0: Your Next Scala Stack
• Scala, Functional Programming and Team Productivity
• Hyper-pragmatic Pure FP Testing with distage-testkit
• Livecoding: DIStage & Bifunctor Tagless Final
• DevInsideYou — Tagless Final with BIO
• Source Talks — Pragmatic Pure FP approach to application design and testing with distage
• (Talk + Livecoding) Izumi Framework: reinvent your productivity with modern pure functional Scala
• (Talk only) Izumi Framework: reinvent your productivity with modern pure functional Scala

Slides:

• Izumi 1.0: Your Next Scala Stack
• Scala, Functional Programming and Team Productivity
• Hyper-pragmatic Pure FP Testing with distage-testkit
• distage: Staged Dependency Injection
• LogStage: Zero-cost Structured Logging
• More slides

Key goals

We aim to provide tools that:

1. Boost productivity and reduce code bloat
2. Are as non-invasive as possible
3. Are introspectable
4. Are better than anything else out there :3

Current state and future plans

We are looking for early adopters, contributors and sponsors.

This project is currently a work in progress.

In the future we are going to (or may) implement more tools based on PPER approach:

1. Best in the world build system
2. Best in the world cluster orchestration tool
3. Best in the world load testing/macro-benchmark tool

Adopters

Are you using Izumi? Please consider opening a pull request to list your organization here:

Projects powered by Izumi

• d4s - "Dynamo DB Database done Scala way". A library that allows accessing the DynamoDB in a purely-functional way.

Credits

YourKit supports open source projects with innovative and intelligent tools for monitoring and profiling Java and .NET applications. YourKit is the creator of YourKit Java Profiler, YourKit .NET Profiler and YourKit YouMonitor.

Triplequote Hydra is the world’s only parallel compiler for the Scala language. Hydra works by parallelizing all of the Scala compiler phases, taking full advantage of the many cores available in modern hardware.

Contributors

• Run ./sbtgen.sc to generate a JVM-only sbt project, run ./sbtgen.sc --js to generate a JVM+JS sbt crossproject

See:

• Build notes
• Project flow

izumi's Issues

Provide an abstraction on reflection

We wish to be as independent of jvm specifics as possible

Post-provisioning access to provisioner for factories/proxies

Think about private inheritance in intermediate language

Finish PPER paper

Metadata support

Logstash sink

Support Abstract Product definitions

slf4j backend

We should be able to accept messages coming from slf4j-enabled libraries

Artifact stub: logstage-adapter-slf4j

Allow shared settings to be defined at "In" level

val inDirectory = In("directory").settings(LibSettings)

Parent context values are enumerated by Locator#enumerate

Include type signatures into type info

Support default package for newModule command

In-code config support module

Implement import support at translation level

We need to correctly resolve types referenced from other domains within our scope

@ExposedTestScope analyzer supporting transitive dependencies

Reintroduce @ExposedTestScope annotation

RFC: Compile-time wiring checker

The only meaningful way to check a plan is to just run the planner inside a macro.

That means the check has to be done at top-level, when all modules have been merged into one module. This also means that module merges have to be defined at type level or macro/compiler-plugin level.

What meaningful checks are there?

• ensure all dependencies are wired (there are no imports)
• when checking a single module, ensure that exactly the provided imports are there
• statically check configs: Ensure that all required keys are defined in application-reference.conf and that there are no redundant keys.

Type checking dynamic modules:

Let's say there is a module that we change at runtime, depending on config, that means we have 2 possible static configurations: mod1 ++ mods and mod2 ++ mods. If we naively check both configurations, we'd need to run the macro twice. If we add more modules, the number of possible configurations grows exponentially. At 16 dynamic modules, we'd have to run the macro 65536 times, which would definitely slow down our compilation!

We may do it another way though:

define top-level module as a set of Variants of modules (TargetPoints in old izumitk)

// Modules with `||` have dynamic variants, modules without are fully static
new ProgramDef {
  val moduleVariants = Seq(
    (customerRepoImpl1 || customerRepoImpl2)
    ,(userRepoImpl1 || userRepoImpl2)
    ,(facebookNotifications || twitterNotifications)
    , staticModules // no `||` operator here, this part is static and doesn't change
  )  
}

Now extract the full static part of the program:

• Extract all the produced bindings in static modules
• Extract the intersection of produced bindings in dynamic modules (take produced bindings that are common in all the variants) // You can also say that we are extracting interfaces from modules. We can do that too. Instead of calculating the intersection between all variants = just require them to define and inherit an interface and check the interface. We'd have to use explicit interfaces to support plugins

Now check the dynamic part of the program:

• For each variant in a dynamic module, take the dependencies and check that every dependency is always fulfilled by the full static part of the program or by the module itself.

That means that instead of running the macro 65536 times, we only need to run it once, and check each variant once.

This typing mechanism is somewhat more restrictive, but it rules out invalid cases such as for example, there are 2 target points: Impl, Logger and 4 modules DummyLogger, DummyImpl, RealLogger, RealImpl.

DummyImpl can only work with DummyLogger and RealLogger can only work with RealLogger:

class RealImpl(logger: RealLogger) extends Impl

So there are 2 valid configurations possible:

Logger = DummyLogger
Impl = DummyImpl

and

Logger = RealLogger
Impl = RealImpl

But there are also 2 invalid configurations:

Logger = DummyLogger
Impl = RealImpl

and

Logger = RealLogger
Impl = DummyImpl

In our typing scheme the user will get errors such as:

In variant RealImpl of Impl: RealLogger is not part of static context. Provided dynamically by RealLogger variant of Logger. Logger has variant DummyLogger, that does not proviode RealLogger.
In variant DummyImpl of Impl: DummyLogger is not part of static context. Provided dynamically by DummyLogger of Logger.  Logger has variant RealLogger, that does not proviode DummyLogger.

To remove the error, user will have to remove the Logger target point and put the logger inside the *Impl modules:

object RealImpl extends ModuleDef {
  make[RealLogger]
  make[Impl].from[RealImpl]
}

Now there are only 2 possible configurations, and both are valid:

Impl = RealImpl | DummyImpl

IMHO this is a good thing, even though the user can't split the Logger inside into its own target point, the split was nonsensical to begin with and only added invalid configurations to the program.

Depends on #102

NB type level impl: HList Cons on single bindings could slow down the compiler a lot. should instead append HLists of HLists

NB performance: Running the planner at every recompile may not be fast at all! the user would have to be provided with performance diagnostics and a way to disable the checks or enable them selectively (i.e. disable for dev, enable in ci)

Scala.js support: replace scala-reflect runtime dependency in distage-core, only use in Provided scope

scala.js does not support any runtime scala-reflect symbols, including TypeTags, Tree and Universe: https://groups.google.com/forum/#!topic/scala-js/xY4Iuq8PVvA
We have runtime instances of TypeTags, Types, Annotations and Universe in distage-core.

Note that we can fully use all of scala-reflect, but only in macros.

We can replace all of these and provide working a impl on scala.js (only distage-static module, but right now it's on par in features and flexibility) by following these steps:

1. We already use Tag and TagK instead of TypeTag in APIs.
2. Provide alternative implementation of SafeType on scala.js. It may be as simple as a macro that .toStrings a TypeTag.
3. Provide alternative implementation of Annotation on scala.js. That means a wrapper API that doesn't (publicly) expose underlying Tree on JVM.
4. Provide alternative RuntimeDIUniverse on scala-js that doesn't reference mirror and Universe

Alternatively, there's a project underway to get cross-platform scala-reflect... https://github.com/portable-scala/reflect ; However, it's likely to not get anywhere, so we shouldn't rely on them.

Think about non-singleton binding semantic

At one hand we resolve at with factories, at the other hand it still may be convenient. Some ideas:

1. @Prototype annotation at usage point
2. Non-singleton binding and non-singleton key (prototype key)

Note: factories are different. Also I don't have any sane usage scenario for this at the moment, but let's keep it.

Accumulating sink for testing purposes

Shade CGLIB

Include full domain schema into rendered code for reflection purposes

Async sink

Decorator for an arbitrary sink. Should maintain a queue and perform the actual flushing in a separate thread (or a shared Executor)

Add logger to distage (including for macro phase)

Garbage collector for plans to support overrides (remove unpinned nodes when their dependees have been overriden)

Extract model artifact out of distage-macro

Write a manifesto explaining WTF is this and the philosophy behind

"WhoDepends" and "DependencyTree" context queries

Just for debug purposes

Add tests for @IDs

「If you don't have a test for it – it doesn't exist.」 – much wise philosof

Generic support at Final stage

Pattern support for StringRenderingPolicy

We should be able to provide a logback-styled pattern, then iterate over it and append message elements according to placeholders

DependencyKeyProviderDefaultImpl: support identified bindings

Full coverage for platform helpers

Better aliases model

Automatic converters for types compatible by signature

Implement Circular Dependencies proxy generator

scala-asm only, classloaders should be provided by a policy

Parser/renderer for intermediate representation ast

Support fields in dynamic trait instances

Technically, we have enough data for that.

A trait:

// class version 52.0 (52)
// access flags 0x601
public abstract interface org/bitbucket/pshirshov/izumi/distage/Case2$ATrait {

  // compiled from: Fixtures.scala

  ATTRIBUTE Scala : unknown

  ATTRIBUTE ScalaInlineInfo : unknown
  // access flags 0x609
  public static abstract INNERCLASS org/bitbucket/pshirshov/izumi/distage/Case2$ATrait org/bitbucket/pshirshov/izumi/distage/Case2 ATrait

  // access flags 0x401
  public abstract org$bitbucket$pshirshov$izumi$distage$Case2$ATrait$_setter_$x_$eq(I)V
    // parameter final  x$1

  // access flags 0x401
  public abstract x()I

  // access flags 0x9
  public static $init$(Lorg/bitbucket/pshirshov/izumi/distage/Case2$ATrait;)V
    // parameter final synthetic  $this
   L0
    LINENUMBER 85 L0
    ALOAD 0
    ICONST_1
    INVOKEINTERFACE org/bitbucket/pshirshov/izumi/distage/Case2$ATrait.org$bitbucket$pshirshov$izumi$distage$Case2$ATrait$_setter_$x_$eq (I)V
   L1
    LINENUMBER 84 L1
    RETURN
   L2
    LOCALVARIABLE $this Lorg/bitbucket/pshirshov/izumi/distage/Case2$ATrait; L0 L2 0
    MAXSTACK = 2
    MAXLOCALS = 1
}

And corresponding class:

  // class version 52.0 (52)
// access flags 0x21
public class org/bitbucket/pshirshov/izumi/distage/Case2$AClass implements org/bitbucket/pshirshov/izumi/distage/Case2$ATrait  {

  // compiled from: Fixtures.scala

  ATTRIBUTE Scala : unknown

  ATTRIBUTE ScalaInlineInfo : unknown
  // access flags 0x9
  public static INNERCLASS org/bitbucket/pshirshov/izumi/distage/Case2$AClass org/bitbucket/pshirshov/izumi/distage/Case2 AClass
  // access flags 0x609
  public static abstract INNERCLASS org/bitbucket/pshirshov/izumi/distage/Case2$ATrait org/bitbucket/pshirshov/izumi/distage/Case2 ATrait

  // access flags 0x12
  private final I x

  // access flags 0x1
  public x()I
   L0
    LINENUMBER 88 L0
    ALOAD 0
    GETFIELD org/bitbucket/pshirshov/izumi/distage/Case2$AClass.x : I
    IRETURN
   L1
    LOCALVARIABLE this Lorg/bitbucket/pshirshov/izumi/distage/Case2$AClass; L0 L1 0
    MAXSTACK = 1
    MAXLOCALS = 1

  // access flags 0x1
  public org$bitbucket$pshirshov$izumi$distage$Case2$ATrait$_setter_$x_$eq(I)V
    // parameter final  x$1
   L0
    LINENUMBER 88 L0
    ALOAD 0
    ILOAD 1
    PUTFIELD org/bitbucket/pshirshov/izumi/distage/Case2$AClass.x : I
    RETURN
   L1
    LOCALVARIABLE this Lorg/bitbucket/pshirshov/izumi/distage/Case2$AClass; L0 L1 0
    LOCALVARIABLE x$1 I L0 L1 1
    MAXSTACK = 2
    MAXLOCALS = 2

  // access flags 0x1
  public <init>()V
   L0
    LINENUMBER 88 L0
    ALOAD 0
    INVOKESPECIAL java/lang/Object.<init> ()V
    ALOAD 0
    INVOKESTATIC org/bitbucket/pshirshov/izumi/distage/Case2$ATrait.$init$ (Lorg/bitbucket/pshirshov/izumi/distage/Case2$ATrait;)V
    RETURN
   L1
    LOCALVARIABLE this Lorg/bitbucket/pshirshov/izumi/distage/Case2$AClass; L0 L1 0
    MAXSTACK = 1
    MAXLOCALS = 1
}

So, what we need to do:

1. Enumerate vals. This is the most problematic part, seems like we can't distinguish between method and val
2. Add field
3. Generate implementations for both setter and getter

Runtime plugins and modules/overrides support (with declarative overrides API because we want support for static checks)

Also see grafter API for inspiration (?)

Make DIException abstract

Allow arbitrary objects as a context for logger

def hello(thing: String) {
  val context = new {
    val x = 1
    val arg = thing
  }
  context.debug("Printing...")
}

def hello(thing: String) {
  object context {
    val x = 1
    val arg = thing
  }
  context.debug("Printing...")
}

def hello(thing: String) {
  val context = logger.withContext (
    "x" -> 1
    , "y" -> thing
  )

  context.debug("Printing...")
}

Implement Factory generator

Library descriptor artifact generator

JSON formatting policy

Declarative log config

The proposal:

1. Add neccessary case classes into the base model
2. Extract IDL object mapper into fundamentals/izumi-config or smth like that
3. Add logstage-typesafe-config and reuse the mapper

Should be implemented after #38

DI benchmark

How long would it take to instantiate a context with 10K instances?

Re-implement TimeUUID generator

Multiplexing abstraction

Implement trait implementation generator

Provide full test coverage for DI