Coder Social home page Coder Social logo

graft's Introduction

Warning: This library is incomplete and unmaintained. Check out UpringJS by one of the original founders if you want something similar.

Graft

The Graft project explores what the web could become, if we extended microservice architectures into the client.

![Gitter](https://badges.gitter.im/Join Chat.svg)

Interested in Graft and jsChan? Watch @mcollina presentation at NodeConf.eu 2014, "Full Stack Through Microservices"

Motivation

When you graft something, it involves joining together parts to create a new whole. One that is hopefully more adaptable, resilient and ultimately interesting.

"Instead of pretending everything is a local function even over the network ..., what if we did it the other way around? Pretend your components are communicating over a network even when they aren't?" -- Docker's Solomon Hykes on LibChan - [link]


Our Projects

  • graft: the library that ties everything together.
  • jschan: our 'standard carrier'. A port of libchan.
  • aetherboard: our 'hello world' demo. A collaborative whiteboard.

Our Process

  • discover how to connect tools through microservices.
  • explore the tools that are already available.
  • adapt those that could be integrated.
  • innovate to build those that don't exist yet.

Our Principles

  • favor small tools that serve only one purpose, but do so well.
  • eschew state, because it only leads to trouble.
  • focus on javascript, because it is universal.
  • evaluate and document, not prescribe.
  • educate.

Our Goals

  • be the premier javascript implementation of libchan.
  • be completely supported for node.js as soon as possible.
  • use Node.JS streams to replicate the semantics of Go Channels.
  • be functional and usable on the browser as we test the waters.
  • use virtual stream objects to provide an api similar to Gulp.
  • attempt control flow abstractions similar to HighlandJS.
  • experiment, document and learn.

API


graft()

The main object of this library. A Graft instance is a Transform stream with objectMode: true. The objects on the output of a Graft instance are Requests. On the input side, you can write just normal JS objects, and everything else you can write to a jsChan channel. These objects will be automatically wrapped up in a Request.

Internally, each Graft instance is backed by jschan.memorySession()`.

In order to process the requests, you can just:

var graft = require('graft')();
var through = require('through2');

graft.pipe(through.obj(function(msg, enc, cb) {
  console.log(msg); // prints { hello: 'world' }
  // process your request
  cb();
}));

graft.write({ hello: 'world' });

graft.ReadChannel()

Returns a nested read channel, this channel will wait for data from the other party.

graft.WriteChannel()

Returns a nested write channel, this channel will buffer data up until is received by the other party.

graft.branch(function(req), stream)

Passes the request to the first argument, and if that returns a truthy value, it calls write(req) on the associated stream. It respect backpressure.

graft.where(obj, stream)

Shortcut for the most common usage of graft.branch(), it allows to rewrite:

graft.branch(function(msg) {
  return msg.hello === 'world'
}, stream)

into:

graft.where({ hello: 'world' }, stream)

Request Interface

Each Graft request is the first message sent on a top-level channel, and it is composed of:

  • all the properties of the message
  • _channel, the associated channel
  • _session, the associated session

Each request will have its own channel, but the session is generic for every client.

The _channel and _session properties will not be enumerable.


spdy.client()

Creates a new spdy client to pipe to:

var graft = require('graft')();
var spdy = require('graft/spdy');

graft.pipe(spdy.client({ port: 12345 }));

graft.write({ hello: 'world' });

spdy.server()

Creates a new spdy server that you can pipe to a graft instance:

var graft = require('graft')();
var spdy = require('graft/spdy');
var through = require('through2');

spdy
  .server({ port: 12345 })
  .pipe(graft)
  .pipe(through.obj(function(msg, enc, cb) {
    console.log(msg); // prints { hello: 'world' }
    // process your request
    cb();
  }));

ws.client()

Creates a new ws client to pipe to:

var graft = require('graft')();
var ws = require('graft/ws');

graft.pipe(ws.client({ port: 12345 }));

graft.write({ hello: 'world' });

It works even from a Browser, using WebPack or Browserify.


ws.server()

Creates a new ws server that you can pipe to a graft instance:

var graft = require('graft')();
var ws = require('graft/ws');
var through = require('through2');

ws
  .server({ port: 12345 })
  .pipe(graft)
  .pipe(through.obj(function(msg, enc, cb) {
    console.log(msg); // prints { hello: 'world' }
    // process your request
    cb();
  }));

You can even pass an existing http server that will be hooked up, like so:

var graft   = require('graft');
var ws      = require('graft/ws');
var http    = require('http');
var server  = http.createServer();

ws
  .server({ server: server })
  .pipe(graft())

About LibChan

Libchan is the connective tissue to all our endeavours. It is a microservices library announced by the Docker project, and it is going to form the basis of all of the tools they build in the future.

It's most unique characteristic is that it replicates the semantics of go channels across network connections, while allowing for nested channels to be transferred in messages. This would let you to do things like attach a reference to a remote file on an HTTP response, that could be opened on the remote end for reading or writing.

The protocol uses SPDY as it's default transport with MSGPACK as it's default serialization format. Both are able to be switched out, with http1+websockets and protobuf fallbacks planned.

While the RequestResponse pattern is the primary focus, Asynchronous Message Passing is still possible, due to the low level nature of the protocol.

Contributors

License

MIT

graft's People

Contributors

adrianrossouw avatar mcollina avatar tamagokun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

graft's Issues

Service Discovery

Still a big picture question for us.

* Just to clarify, we are referring al already deployed modules.*

find our 'underscore.js for streams'

Right now graft is pretty great at taking multiple jschan sessions and orchestrating them to build new applications. But to make this be actually user-friendly and something that can be easily taught/demonstrated, it needs to have a good heap of sugar on top of it.

we need a library that will allow us to have the same kind of relationship to it, as backbone has to underscore. This issue serves as a way to collect all the various options.

Some of our requirements :

  1. The ability to cleanly tie the library into the existing Graft object.
  2. A series of additional higher order functions, such as .filter, .map, .where etc.
  3. The ability to extend the functions through the use of mixins.
  4. Relatedly, what we choose to implement needs to be non-kitchen sink.
  5. It needs to handle flow control to handle our nested streams.
  6. It will need to be based on streams2/streams3 to avoid compatibility mode forever and always.

related issues : #2 #5

Implement AetherBoard example app

It's kind of hard to make any decisions about what an API needs to look like, without at the very least trying to build something in the problem domain. (see: #2 )

So we've decided to build out a collaborative whiteboard app called AetherBoard.

The project itself is something I had been working on to be able to pull off in a node knockout situation, and is a new iteration of a project we did for the 2011 contest

We were going to aim for pseudocode at first, but after plotting out the spec, I think it's more feasible to build the pieces and then figure out how they want to be connected to each other.

We’re hoping to have a working demo by nodeconf, since @mcollina is doing a jschan presentation.

check out the issue queue and lend a hand if you are interested ?

https://github.com/AetherBoard/AetherBoard/issues

Status of this Project

Hi,
I'm currently reviewing approaches for building microservices with Node.js.
I would like to know whether this project's status.
No activity in several months seems to be dead for a Node.js project?!
BG

Is Graft still being pursued?

Not much activity recently, is this repo still active? In what way does the death of SPDY (in favor of HTTP/2) effect this project? Has anyone built out a bigger example that can be open-sourced? Thanks.

evaluate http2 implementations

Graft is built atop a protocol called libchan, which used certain properties of SPDY to be able to provide features similar to Go's channels over the network.

Since SPDY has been killed, and all work is going towards HTTP2, it's kind of put the upstream protocol in flux. I mean, the stuff that's there still works, and works today... but there is some settling left to be done in the eco system.

To work past this on our side, we need to find / help build a mature http2 implementation for node.js.

Clarify usage of spdy and graft client

Are we supposed to create a new instance of SPDY and websocket client Session each time we have a graft message to send? My SPDY and Websocket graft client sessions to not seem to stay open for long periods of time, when attached to a graft object used by a long-running server.

API

What API do we want for Graft?
I think something seneca-like is really nice, but built on jsChan (with no concept of request/response).

To some extent, I envision Graft as a router for jsChan, in a similar way to relationship between Express and Connect.

What do you think?

we should standardise on tap for our tests

The TestAnything protocol is a stream oriented way to run and consume tests.

It's more in-line with our stated goals, and I feel it's probably technically superior to what we are using now. (not that i have any issues with mocha).

Porting the tests to tape or similar is probably a very good task for somebody who is trying to figure out how Graft works, so if that sounds like you, please do help =)

http/REST support

so, we need to support HTTP and REST to some extent.

What I think is:

var graftHttp = require('graft/http');
http.createServer(graftHttp);

// or with connect

connect.use(graftHttp);

This will create messages with these properties:

  • req, the actual readable stream
  • res, the actual writable stream
  • headers
  • statusChan, Readable channel that can be used to change the status code.

What do you think?

Is this project dead?

Hi there,

Just stumbled on this project and it looks interesting, but saw that there hasn't been any public activity for a year. Has the project been abandoned?

README with examples and APIs

The current README is missing both the examples and the API. We have not really much to show, but still it's worth adding stuff.

Validation as a uService

I think we might want to provide a validation-as-a-microservice module, based around Joi.

What do you think?
I'm placing it here just because it's a place...

It should be kind-of compatible with GraftJS/graft-http#1.

io.js v2.4.0 error

given a spdy server created with graft/spdy on port 3447, I have this client code, modeled after the example in README.md:

'use strict';

let graft = require('graft')();
let spdy = require('graft/spdy');

let time = graft.WriteChannel();

graft.pipe(spdy.client({ port: 3447 }));

graft.write({
  topic: 'register',
  id: 'time',
  data: time
});

setInterval(function() {
  time.write(new Date());
}, 1000);

(mind you, I have no idea if this code does what I think it does, because...)

events.js:141
      throw er; // Unhandled 'error' event
            ^
Error: Protocol "http:" not supported. Expected "https:".
    at new ClientRequest (_http_client.js:53:11)
    at Object.exports.request (http.js:31:10)
    at ClientSession.newStream [as _createNewStream] (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/jschan/lib/spdy/client.js:104:18)
    at createChannel (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/jschan/lib/spdy/client.js:142:11)
    at ClientSession.WriteChannel (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/jschan/lib/spdy/client.js:156:10)
    at SPDYClient.Client._write (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/lib/client.js:36:30)
    at doWrite (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_writable.js:279:12)
    at writeOrBuffer (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_writable.js:266:5)
    at SPDYClient.Writable.write (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_writable.js:211:11)
    at Graft.ondata (/Volumes/alien/projects/digs/digs-graft/node_modules/graft/node_modules/readable-stream/lib/_stream_readable.js:572:20)

UPDATE: I realize this may be the fault of the server code. Here's the server, in all its glory:

'use strict';

process.env.DEBUG = 'digs-graft*';

let graft = require('graft');
let spdy = require('graft/spdy');
let rc = require('rc-yaml');
let through = require('through2');
let pkg = require('./package.json');
let domain = require('domain');
let debug = require('debug')('digs-graft');

let cfg = rc(pkg.name, {
  port: 3447
});

let digs = graft();
let router = graft();

function data() {
  return through.obj(function(msg, enc, done) {
    let debug = require('debug')('digs-graft:data');
    debug(msg);
  });
}

function register() {
  return through.obj(function(msg, enc, done) {
    let debug = require('debug')('digs-graft:register');
    debug(`client ${msg.id} registered`);

    let d = domain.create();
    d.on('error', function(err) {
      debug(err);
    });
    d.run(function() {
      router.branch(function() {
        return msg.data; // branch if data property is truthy
      }, data());
    });
  });
}

digs.where({
  topic: 'register'
}, register());

spdy.server({
  port: cfg.port
})
    .pipe(digs);

question(s): overlap with messaging protocols (IoT)?

I just discovered this project, and my interest is sparked.

Say I wanted to get real-time data from a sensor and pipe it "somewhere". Would Graft help me do that?

I do understand libchan is more of a request/response thing, but the README.md also mentions it's possible to use it for asynchronous messaging (though, unfortunately, this is not elaborated upon).

I also understand a goal of Graft is ease integration with the browser. This is something I may want to take advantage of, but for now let's just say my data is going to go "somewhere".

It seems like devices want to work over things like RabbitMQ, MQTT, etc. I get the impression these protocols are popular because they are simple, and can be deployed on devices with memory/storage constraints.

But it also seems (and I may be wrong here) that publishing the same message every n milliseconds (in the interest of providing "real-time" data) would be inefficient and potentially slow--especially since the published message would have to go through a broker. Naturally, a stream or socket--with a direct connection--would make more sense, right? And it'd make even more sense if the "endpoint" wasn't necessarily a browser, but another device?


Imagine some sort of setup wherein you were monitoring electricity usage. If the electricity usage gets too high, you wish to reduce it. Say you want to dim the lights by some function of the usage, and then brighten the lights again if the usage goes back down.

So perhaps you would have your meter open a socket to some server, which would then transform the data through your function, and pipe the data to a dimmer, which would reduce or increase the brightness as necessary.

But perhaps then you're using too much juice and the lights are all but off. At this point, you still need to reduce the power consumption. So then somebody (not sure which microservice) would then tell the climate control system to take it easy. And so on, and so forth.

Oh, and you want to see a dashboard in a browser of all this craziness.

Is libchan, and thus Graft, a solution?

graft.branch should be asynchronous

currently the first argument to .branch returns a .truthy to decide whether or not to direct messages to the second argument.

it occured to me that we might need it to be asynchronous (only authenticated users past this point), and we might need to be in situations where it returns a different stream instead of the one specified (ie: pipe things onto it)

Module Discovery and verification

A big part of our mission statement is to research and document how to use the various stream-oriented libraries that are being added to npm every day.

It's become very apparent to me over the last few months that this is probably one of the biggest challenges of building systems this way. There are hundreds, if not thousands, of stream modules, and just evaluating them is a full-time job.

I have some ideas on how we can make this easier for our users, but we really need to pose the question to a wider audience.

authentication of channels / streams with passport.js

I was just thinking about how it would be possible to have a channel that requires you to provide certain credentials before messages to pass. It also occured to me that it might be possible to adapt passport.js to do this.

Passport seems to be designed with stuff like this in mind : jaredhanson/passport#383

For instance, this bit of pseudocode would be to allow file uploads, but only for messages that authenticate against the passport local auth, or twitter, or json web token. whatever.

var graft = require('graft');
var passport = require('passport');
var LocalStrategy = require('passport-local').Strategy;
var passportStream = require('passport-stream')(passport);
var fs = require('fs');

passport.use(new LocalStrategy(/* snip */));

var graftServer = graft()
  .pipe(passportStream.authenticate('local'))
  .pipe(uploadService)

// client
var graftClient = graft();
graftClient.pipe(graftServer);

graft.write({ 
  username: 'adam',
  password: '***',
  file: fs.createReadStream('/file/to/upload'),
  location: '/path/on/server'
});

project governance

As we are just starting this project, we need to discuss some topics up front. This is not as glamorous as the jschan code, but it is just as important.

I've been building open source communities for a long time, and I was absolutely astounded at how completely the book "Producing OSS" mapped onto how I see the problem space.

@pelger @mcollina - all of us should read the getting started chapter before the next meeting, so we can discuss how we want to manage/grow our project.

v0.1.0

Are we ready to release this one?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.