toa-io / toa

Microservice operations distributed runtime

Toa

Runtime for low code eventually consistent distributed systems.

Status

The project is under heavy development, while being used in production by its authors. Non-scattered documentation is coming this fall.

Features

Runtime

• Interservice communications
  • Reliable RPC and Events (ComQ)
  • Flow control
  • Eventual consistency guarantee (not yet)
  • Uniform interface
    • Input/query segregation
    • Successful rejections
    • Distributed exception handling
  • Transparent service discovery
  • Multi-protocol transmission
    • built-in support for AMQP and HTTP
  • In-memory communications
  • Messages validation
• Persistent state management
  • Concurrency control
  • Batching
  • Data validation
  • Built-in implementations
    • MongoDB
    • SQL (Knex)
• Configuration with secrets
• API Gateway
  • Resource discovery
  • Identity
  • Access control
  • Cache control
  • File uploads, downloads, and processing.
• Realtime events
• BLOB storage
  • MIME type detection and validation
  • Deduplication
  • Arbitrary orderings
  • Providers: file system, Amazon S3
• Transient state
• External communications governance
  • HTTP
  • AMQP

Development

• Language interoperability
  • Node.js
  • Bash
• Service prototyping (inheritance)
  • Generic prototype
• Data and operations contracts
• Declarative API Gateway endpoints with authorization policies
• Integration tests SDK
• Runtime bootloader API
• Development environment configuration
• CLI
• Extensibility:
  • Persistent storages
  • Communication protocols
  • Runtime core abstractions
  • Arbitrary services

Operations

• Standalone infrastructure configuration
• Multi-environment configuration
• Components distribution across containers
• One-command fully automated deployment
  • Preset multi-arch docker images
  • Helm deployment to kubernetes
• CLI utilities
  • Secrets management
  • Remote Container Shell

toa's Issues

Operations HTTP context

Use cases

• client ip
• custom headers

Questions

• How operation access http context?
• Should operation be accessible without http context (interservice call)?
• Is context always optional?
• Is operation bounded to HTTP calls only?

REST (text/html) API

• runtime
• component
• gateway

Set length

https://github.com/quantumducks/kookaburra/blob/b4ed9171e5e6eaeec538dededeade82a3deb8c6b/runtime/prototype/manifest.yaml#L8

RPC

• error handling
• exception handling

State Projections

• use cases?

Algorithm declaration must override manifest

Implement pub/sub

Collection updates

• use cases?
• writeable cursor?

HashMap state

• storage?

Add pluggable runners

• javascript runner
• add Bridge (IPC)

Calculations

/sum?a=1&b=2 -> 3

Add component generator

Circuit Breaker

• strategy
• client behavior/fallback

Refactor http

• single-/multi- runtime resources?

Create new object

This changes original object which is later being sent to events.
https://github.com/quantumducks/kookaburra/blob/eb556b8bef49dffd0fada079260f914e7f4a3d7d/connectors/storages.mongodb/src/entry.js#L3

Public package

Scoped?
kbr
kbctl
kb
@kookaburra/cli

Make identification pluggable

kookaburra-identifier-uuid

Runtime composition

kbr host a b

Refactor runtime

• runtime
• call
• endpoint
• invocation
• locator
• operation
• proxy
• query
• schema (standalone repo with schemas)
• state (predefined properties)

Refactor state.mongodb

• standalone repo

Test

Transactional outbox

• events
• ecRPC

Refactor cli

• rename to kbr
• subcommands

RBAC for properties

Blobs

• schema type?
• creating?
• transferring?
• http gateway?
• standalone blobs gateway?

Split manifest validation into normalization and validation

• normalize with checks
• validate with schema
  • additionalProperties: false

Fix http binding

• restore integration
• add discovery integration

Standard errors

API Gateway Sockets

• event routing
• operation calls
• reply routing
• scaling (user shards / connection shards)
• events/replies buffer
• GET/events consistency problem (version?)
• STOMP?

Runtime proxy

• discovery
• transport

Add Sampling

• io
• state

create sample
apply sample

HTTP Client library

• swagger

Changelog

Add changelog generator

Refactor templates

• should it be standalone repo?
• rethink templates

Eventually consistent RPC

Fix create operation

• add domain resolution (promt?)
• add operation manifest
• fix: name is set to path instead of basename

Implement blobs

Add schema references

• common component-level definitions

Manifest validation

Refactor boot

• clean the mess
• manifest validation

HTTP API Gateway

• operation discovery
• operation binding
• authentication
• RBAC

Response caching

• vary
• storage?

Add JSDoc checker

Add examples

Refactor transport.amqp

• [] standalone repo

Rename to origin

https://github.com/quantumducks/kookaburra/blob/eb556b8bef49dffd0fada079260f914e7f4a3d7d/runtime/prototype/events/created.js#L3

Generic Runtime

Runtime inheritance

State operators

• create collection
• update collection
• indexes

Is this framework responsibility?

Make IO properties assignable

https://github.com/quantumducks/kookaburra/blob/d4c2fd1830a4ac6fff066c29ec4e403ab249182c/packages/runtime/src/io.js#L31

Replace with Object.assign()

Fix console

Do not create new function on each call