sinclairzx81 / sidewinder

Type Safe Micro Services for Node

Sidewinder

Type Safe Micro Services for Node

Overview

Sidewinder is a strictly typed and runtime validated micro service framework built for Node and Browser environments. It is designed for web service architectures where each service needs to communicate with other services in complex ways and where challenges often arise verifying each service is communicating using strict communication contracts.

Sidewinder is developed primarily around a runtime type system based on JSON Schema. It encodes runtime type information into JavaScript directly then leverages the TypeScript language to statically infer associated static types at compile time. This approach enables distributed services to be statically checked with the TypeScript compiler, with the same runtime data assertions handled automatically by Sidewinder packages using standard JSON Schema validation.

License MIT

Contents

• Overview
• Install
• Static and Runtime Safe
• Services and Clients
• Services and Metadata
• Build Local

Packages

• Async
• Buffer
• Channel
• Client
• Config
• Contract
• Event
• Hash
• Mime
• Mongo
• Path
• Platform
• Query
• Redis
• Server
• Token
• Type
• Validator
• Value
• Web

Install

Sidewinder consists of a number of packages that target various facets of micro service development. Each package is orientated towards type safe interactions with services and common Node infrastructure.

# Runtime Type System
$ npm install @sidewinder/type       # Json Schema Runtime Type Builder
$ npm install @sidewinder/validator  # Json Schema Validator

# Service Packages
$ npm install @sidewinder/contract   # Service Descriptions Contracts
$ npm install @sidewinder/client     # Http and Web Socket Clients
$ npm install @sidewinder/server     # Http and Web Socket Services and Hosting

# Database and Infrastructure
$ npm install @sidewinder/query      # Query Filter Syntax for Mongo
$ npm install @sidewinder/mongo      # Type Safe Mongo
$ npm install @sidewinder/redis      # Type Safe Redis

# Application Messaging
$ npm install @sidewinder/async      # Asynchronous Primitives
$ npm install @sidewinder/channel    # Asynchronous Channels
$ npm install @sidewinder/events     # Portable Event Emitter

# Hashing and Signing
$ npm install @sidewinder/hash       # Hashing Functions
$ npm install @sidewinder/token      # Type Safe Json Web Token

# Environment
$ npm install @sidewinder/buffer     # Operations on type Uint8Array
$ npm install @sidewinder/config     # Type Safe Configurations
$ npm install @sidewinder/path       # File System Pathing Utility
$ npm install @sidewinder/platform   # Runtime Environment Checks

Static and Runtime Safe

TypeScript Example Link

Sidewinder provides both runtime and static type safety derived from Contract definitions encoded in JavaScript. It makes heavy use of TypeScript's type inference capabilities to statically infer Client and Service method signatures for defined Contracts; with data received over the network runtime checked to ensure it matches the expected parameter and return types defined for each method.

// ---------------------------------------------------------------------------
// Contract
// ---------------------------------------------------------------------------

const Contract = Type.Contract({
    server: {
        'add': Type.Function([Type.Number(), Type.Number()], Type.Number())
    }
})

// ---------------------------------------------------------------------------
// Service
// ---------------------------------------------------------------------------

const service = new WebService(Contract)

service.method('add', (clientId, a, b) => {
    //        │           │      │
    //        │           │      └─── params (a: number, b: number)
    //        │           │
    //        │           └─── unique client identifier
    //        │
    //        └─── method name inferred from contract
    //
    //
    //     ┌─── return inferred as `number`
    //     │
    return a + b 
})

// ---------------------------------------------------------------------------
// Client
// ---------------------------------------------------------------------------

const client = new WebClient(Contract, 'http://....')

const result = await client.call('add', 1, 1)
//    │                         │         │
//    │                         │         └─── arguments as (method: string, a: number, b: number)
//    │                         │ 
//    │                         └─── method name inferred from contract
//    │
//    └─── result is `number`

Services and Clients

TypeScript Example Link

Sidewinder services consist of three main components, a Contract, Service and Client. A Contract defines a set of callable RPC methods and is shared between both Client and Server. A Service provides an implementation for a Contract; and a Client calls methods implemented on the Service. Contracts are used to infer type safe functions on Services and Clients, well as validate method calls made over the network.

The following shows general usage.

import { Type }             from '@sidewinder/contract'
import { Host, WebService } from '@sidewinder/server'
import { WebClient }        from '@sidewinder/client'

// ---------------------------------------------------------------------------
// Contract
//
// Contracts are service interface descriptions that describe a set of callable
// functions and an optional encoding format. Sidewinder uses Contracts to 
// validate data sent over a network as well as to statically infer Client 
// and Server methods in TypeScript. Try changing the parameters and return 
// types for the functions below to invalidate Client and Server methods.
//
// ---------------------------------------------------------------------------

const Contract = Type.Contract({
    format: 'json',
    server: {
        'add': Type.Function([Type.Number(), Type.Number()], Type.Number()),
        'sub': Type.Function([Type.Number(), Type.Number()], Type.Number()),
        'mul': Type.Function([Type.Number(), Type.Number()], Type.Number()),
        'div': Type.Function([Type.Number(), Type.Number()], Type.Number()),
    }
})

// ---------------------------------------------------------------------------
// Service
//
// Services provide concrete implementations for Contracts. Service methods
// receive a context along with typed parameters defined the the method. The
// static type information is derived from the Contract.
//
// ---------------------------------------------------------------------------

const service = new WebService(Contract)
service.method('add', (context, a, b) => a + b)
service.method('sub', (context, a, b) => a - b)
service.method('mul', (context, a, b) => a * b)
service.method('div', (context, a, b) => a / b)

const host = new Host()
host.use(service)
host.listen(5000)

// ---------------------------------------------------------------------------
// Client
//
// Clients connect to Services. Sidewinder provides two client types. The 
// first is a WebClient which connects to WebService implementations over 
// Http, and the second is a WebSocketClient which connects to WebSocketService 
// implementations over a Web Socket. The following creates a WebClient to 
// consume the service above.
//
// ---------------------------------------------------------------------------

const client = new WebClient(Contract, 'http://localhost:5000/')
const add = await client.call('add', 1, 2)
const sub = await client.call('sub', 1, 2)
const mul = await client.call('mul', 1, 2)
const div = await client.call('div', 1, 2)
console.log([add, sub, mul, div]) // [3, -1, 2, 0.5]

Service And Metadata

Sidewinder Contracts are expressed as serializable JavaScript objects with embedded JSON schema used to represent method parameter and return types. Contracts can be used for machine readable schematics and published to remote systems, or used to generate human readable documentation.

// ---------------------------------------------------------------------------
// This definition ...
// ---------------------------------------------------------------------------

const Contract = Type.Contract({
    format: 'json',
    server: {
        'add': Type.Function([Type.Number(), Type.Number()], Type.Number()),
        'sub': Type.Function([Type.Number(), Type.Number()], Type.Number()),
        'mul': Type.Function([Type.Number(), Type.Number()], Type.Number()),
        'div': Type.Function([Type.Number(), Type.Number()], Type.Number()),
    }
})

// ---------------------------------------------------------------------------
// is equivalent to ...
// ---------------------------------------------------------------------------

const Contract = {
  type: 'contract',
  format: 'json',
  server: {
    'add': {
      type: 'function',
      returns: { type: 'number' },
      parameters: [
        { type: 'number' },
        { type: 'number' }
      ]
    },
    'sub': {
      type: 'function',
      returns: { type: 'number' },
      parameters: [
        { type: 'number' },
        { type: 'number' }
      ]
    },
    'mul': {
      type: 'function',
      returns: { type: 'number' },
      parameters: [
        { type: 'number' },
        { type: 'number' }
      ]
    },
    'div': {
      type: 'function',
      returns: { type: 'number' },
      parameters: [
        { type: 'number' },
        { type: 'number' }
      ]
    }
  }
}

Build Local

Sidewinder is built as a mono repository with each publishable package located under the libs directory. Sidewinder uses the Hammer build tooling for automated tests, builds and publishing. Sidewinder requires Node 14 LTS. The following shell commands clone the project and outline the commands provide through npm scripts.

# clone
$ git clone [email protected]:sinclairzx81/sidewinder.git
$ cd sidewinder
$ npm install

# tasks
$ npm start         # starts the example project
$ npm test          # runs the full sidewinder test suite
$ npm test channel  # runs the sidewinder channel test suite only
$ npm run format    # runs code formatting across the project
$ npm run build     # builds all packages to target/build
$ npm run clean     # cleans all build artifacts

sidewinder's Issues

Take a nap.

Hi @sinclairzx81 ,

I played with sidewinder a bit and it's awesome (in the truest sense of the word; not in the "dude these nachos are 'awesome'" sense of the word), but I am genuinely worried that you don't sleep.

Really great work!
Keep it up,
But take a nap,
Jesse

Value.Check() fails if optional property missing from object.

Currently Value.Check() will fail if the schema is of type Object and the supplied value is missing any properties from the Object schema, even if those properties are Optional.

Example:

const schema = Type.Object({testProperty: Type.Optional(Type.String())})
const check = Value.Check(schema, {}) // Will return false
const nextCheck = Value.Check(schema, {testProperty: ''}) // Will return true

This does not align with the behaviour of Value.Upcast(), which will only create non-optional properties. So this does not work:

const schema = Type.Object({testProperty: Type.Optional(Type.String())})
const upcast = Value.Upcast(schema, null)
const check = Value.Check(schema, upcast) // Will return false, despite upcast

But this does:

const schema = Type.Object({testProperty: Type.String()})
const upcast = Value.Upcast(schema, null)
const check = Value.Check(schema, upcast) // Will return true

Validation fails when using $push operator in mongo database

I'm currently running into an issue where validation fails when using the $push operator in a mongo database to add an item to an array. It appears the validator is expecting an array, when I'm passing in a single value to be added to the array.

As a minimal example, consider this model:

export type Plant = Static<typeof Plant>
export const Plant = Type.Object({
  plantType: PlantType,
  size: Type.Number()
})

export type GardenBed = Static<typeof GardenBed>
export const GardenBed = Type.Object({
  _id: Type.String(),
  plants: Type.Array(Plant)
})

Then I create a mongo database like this:

export const PlantDatabaseSchema = Type.Database({
  gardens: GardenBed
})

export class PlantDatabase extends MongoDatabase<typeof PlantDatabaseSchema> {
  constructor(db: Db) {
    super(PlantDatabaseSchema, db)
  }

/**
   * Adds a plant to the garden bed.
   * Currently non-functional due to validation error.
   * @param request plot to update
   */
  public async plantInGardenBed(plant: Plant) {
    await this.collection('gardens').updateOne({
      _id: request._id
    }, {
      $push: {
        plants: plant
      }
    })
  }
}

When I attempt to call plantInGardenBed I get a validation error like this:

ValidateError: Data did not to validate
    at Validator.assert ()
    at MongoCollection.validateUpdate ()
    at Object.2 ()
    at matchArguments ()
    at MongoCollection.updateOne ()
    at PlantDatabase.plantInGardenBed ()
    at GardenService.addPlantToGardenBed ()
    at processTicksAndRejections ()
    at async Object.callback ()
    at async ServiceMethods.execute () {
  errors: [
    {
      type: 0,
      schema: [Object],
      path: '/plots',
      value: [Object],
      message: 'Expected array'
    }
  ]

I'm basing my query format on the mongodb docs for $push. It could be that I'm missing something on how to format this, but I can't seem to see an issue with what I'm doing based on the mongodb docs, so suspect it might be an issue with the way it is being validated.

Validation fails when optional type has undefined value

I'm encountering an issue where setting the value of an optional type to undefined fails validation. Adding the following test to tests/validator/validator.ts also fails:

it('Should validate undefined optional', () => {
    const T = Type.Optional(Type.String())
    const V = new Validator(T)
    const R1 = V.check(undefined)
    Assert.equal(R1.success, true)
  })

My understanding is that the above test should pass.

2020-12: Recursive Schema Validation Issue

This is a reference issue for ajv-validator/ajv#1964 where recursive schemas cannot be validated on 2020-12 spec when defined in a nested schema. Top level validation of the schema works ok, nested schemas fail with unusual errors.

This may be a result of incorrect utilization of the $dynamicRef and $dynamicAnchor keywords on 2020-12.

For future reference.

AJV validation fails when using custom SchemaOptions

I'm having an issue at the moment where if I use a custom SchemaOption, for example:

welcomePage: Type.String({ default: '', label: 'Welcome Page' })

I get an AJV validation error:

Error: strict mode: unknown keyword: "label"

It only appears to happen when building the SideWinder Server (the SideWinder client builds without issue). Is there something I need to do on the server side to enable custom SchemaOptions?