lelylan / lelylan

Open Source Lightweight Microservices Architecture for the Internet of Things. For developers.

Lelylan. Open Source Internet of Things

Lelylan is an IoT cloud platform based on a lightweight microservices architecture.

The Lelylan platform is both hardware-agnostic and platform-agnostic. This means that you can connect any hardware, from the ESP8266 to the most professional embedded hardware solution and everything in between - and it can run on any public cloud, your own private datacenter, or even in a hybrid environment, whether virtualized or bare metal.

Notes

This repository defines the architecture documentation. Check out the microservices list for the code implementation.

Why Lelylan

Research in the Internet of Things is global and growing fast, but lacks the standard tools. Many companies are building their own solutions. By sharing what we have learned during the years, we want to create a shared code base with a clear focus on developers. To see Lelylan in action checkout the tutorials in the dev center.

Resources

• Architecture (request life cycle)
• Internet of Things API (how it works)
• Blog (latest from the community)

Getting Started

Requirements

Lelylan is tested against

• Ruby MRI ~1.9.3
• Node ~0.8.8
• MongoDB ~2.6
• Redis ~2.6

(remember to run the mongod and redis-server)

Setup

• Install MongoDB
• Install Redis

Installation

Development

Lelylan is composed by different microservices. Follow the installation guidelines for each of them.

Microservice	Description
API Proxy	Proxy API
Devices API	Device monitoring and control
Types API	Device type structure
Subscriptions API	Realtime subscription
Profiles API	Profile information
OAuth 2.0	User authentication and authorization
Physical Proxy	Forward requests to the physical world
MQTT Node	Forward and receive MQTT requests
MQTT Server	MQTT server/broker
Webhooks	Realtime HTTP notification
Websockets	Full-duplex communication over TCP
Dev Center	Lelylan Dev Center
Devices Dashboard	Lelylan Devices Dashboard
Types Dashboard	Lelylan Types Dashboard

You can now access the APIs from http://0.0.0.0:8200 (API proxy URL).

Development Environment Variable

During deployment, every microservice needs to be set to the following environment variables (remember to change them with your own microservices, mongodb, redis and cache URLs).

Environment Variable	Description
RACK_ENV=development	Development rack environment
RAILS_ENV=development	Development rails environment
NODE_ENV=development	Development node environment
LELYLAN_API_URL	Api Proxy microservice URL
LELYLAN_API_HOST	Api Proxy microservice URL
LELYLAN_DEV_URL=dev.lelylan.com	Dev Center microservice URL
LELYLAN_PEOPLE_URL=people.lelylan.com	OAuth 2.0 microservice URL
LELYLAN_DEVICES_URL=devices.lelylan.com	Devices API microservice URL
LELYLAN_TYPES_URL=types.lelylan.com	Types API microservice URL
LELYLAN_SUBSCRIPTIONS_URL=subscriptions.lelylan.com	Subs. API microservice URL
LELYLAN_PROFILES_URL=profiles.lelylan.com	Profiles API microservice URL
MONGOLAB_PEOPLE_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	OAuth 2.0 MongoDB URL
MONGOLAB_DEVICES_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Devices API MongoDB URL
MONGOLAB_TYPES_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Types API MongoDB URL
MONGOLAB_JOBS_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Event Bus MongoDB URL
MONGOLAB_SUBSCRIPTIONS_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Subs. MongoDB URL
MEMCACHIER_SERVERS=<host>:<port>	Cache server
MEMCACHIER_USERNAME=<username>	Cache server username
MEMCACHIER_PASSWORD=<password>	Cache server password
REDIS_URL=redis://<user>:<pass>@<host>:<port>/	Background Job Redis URL
REDIS_RATE_LIMIT_URL=redis://<user>:<pass>@<host>:<port>/	Late Limit Redis URL

We are studying solutions such as Docker, Mesos, and Ansible to simplify the installation process. If you are experimenting in the same area get in touch with lelylan team.

Docker Development Installation

• Configure environment variables in docker-compose-dev.yml
• Run docker compose

docker-compose -f docker-compose-dev.yml up -d

Use subdomains for all microservices

Sed to replace domains

sed -i '/VIRTUAL_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/DEFAULT_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/PUBLIC_URL/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/LELYLAN_DEV_URL/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/PUBLIC_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/LELYLAN_PROXY_URL/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/LELYLAN_API_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml

Microservice	Default Domian	Domian after sed commands
api-proxy	api-proxy	api-proxy.lelylan.com
devices	devices	devices.lelylan.com
types	types	types.lelylan.com
subscriptions	subscriptions	subscriptions.lelylan.com
profiles	profiles	profiles.lelylan.com
people	people	people.lelylan.com
physicals	physicals	physicals.lelylan.com
nodes	nodes	nodes.lelylan.com
mqtt	mqtt	mqtt.lelylan.com
webhooks	webhooks	webhooks.lelylan.com
websockets	websockets	websockets.lelylan.com
dev	dev-center	dev-center.lelylan.com
devices-dashboard-ng	devices-dashboard-ng	devices-dashboard-ng.lelylan.com
types-dashboard-ng	types-dashboard-ng	types-dashboard-ng.lelylan.com

Production

Lelylan is composed by different microservices. Follow the installation guidelines for each of them.

Microservice	Description
API Proxy	Proxy API
Devices API	Device monitoring and control
Types API	Device type structure
Subscriptions API	Realtime subscription
Profiles API	Profile information
OAuth 2.0	User authentication and authorization
Physical Proxy	Forward requests to the physical world
MQTT Node	Forward and receive MQTT requests
MQTT Server	MQTT server/broker
Webhooks	Realtime HTTP notification
Websockets	Full-duplex communication over TCP

You can now access the APIs from http://0.0.0.0:8200 (API proxy URL).

Production Environment Variable

During deployment, every microservice needs to be set to the following environment variables (remember to change them with your own microservices, mongodb, redis and cache URLs).

Environment Variable	Description
RACK_ENV=production	Production rack environment
RAILS_ENV=production	Production rails environment
NODE_ENV=production	Production node environment
LELYLAN_API_URL	Api Proxy microservice URL
LELYLAN_API_HOST	Api Proxy microservice URL
LELYLAN_PEOPLE_URL=people.lelylan.com	OAuth 2.0 microservice URL
LELYLAN_DEVICES_URL=devices.lelylan.com	Devices API microservice URL
LELYLAN_TYPES_URL=types.lelylan.com	Types API microservice URL
LELYLAN_SUBSCRIPTIONS_URL=subscriptions.lelylan.com	Subs. API microservice URL
LELYLAN_PROFILES_URL=profiles.lelylan.com	Profiles API microservice URL
MONGOLAB_PEOPLE_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	OAuth 2.0 MongoDB URL
MONGOLAB_DEVICES_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Devices API MongoDB URL
MONGOLAB_TYPES_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Types API MongoDB URL
MONGOLAB_JOBS_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Event Bus MongoDB URL
MONGOLAB_SUBSCRIPTIONS_URL=mongodb://<user>:<pass>@<host>:<port>/<name>	Subs. MongoDB URL
MEMCACHIER_SERVERS=<host>:<port>	Cache server
MEMCACHIER_USERNAME=<username>	Cache server username
MEMCACHIER_PASSWORD=<password>	Cache server password
REDIS_URL=redis://<user>:<pass>@<host>:<port>/	Background Job Redis URL
REDIS_RATE_LIMIT_URL=redis://<user>:<pass>@<host>:<port>/	Late Limit Redis URL

We are studying solutions such as Docker, Mesos, and Ansible to simplify the installation process. If you are experimenting in the same area get in touch with lelylan team.

Docker Production Installation

• Configure environment variables in docker-compose.yml
• Run docker compose

docker-compose up -d

Use subdomains for all microservices

Sed to replace domains

sed -i '/VIRTUAL_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/DEFAULT_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/PUBLIC_URL/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/LELYLAN_PROXY_URL/ s/$/.lelylan.com/' docker-compose-dev.yml
sed -i '/LELYLAN_API_HOST/ s/$/.lelylan.com/' docker-compose-dev.yml

Microservice	Default Domian	Domian after sed commands
api-proxy	api-proxy	api-proxy.lelylan.com
devices	devices	devices.lelylan.com
types	types	types.lelylan.com
subscriptions	subscriptions	subscriptions.lelylan.com
profiles	profiles	profiles.lelylan.com
people	people	people.lelylan.com
physicals	physicals	physicals.lelylan.com
nodes	nodes	nodes.lelylan.com
mqtt	mqtt	mqtt.lelylan.com
webhooks	webhooks	webhooks.lelylan.com
websockets	websockets	websockets.lelylan.com

Roadmap

The Roadmap provides the description of the items that the project has decided to concentrate on. It should serve as a reference point for Lelylan contributors to understand where the project is going, and helps to determine whether a contribution could be conflicting considering the future goals.

The fact that a feature isn't listed here doesn't mean that a patch for it will automatically be refused (we also miss important things). We are always happy to receive patches for new cool features that we haven't thought about, or didn't consider as a priority. Nevertheless understand that such patches might take longer for us to review.

Checkout the roadmap to see our near future goals.

Contributing to Lelylan

This Contributing document tries to define a contributor's guide explaining how to contribute to one or more Lelylan Microservice. It contains information about reporting issues as well as some useful tips and guidelines for experienced open source contributors.

Checkout the contributing to help us with Lelylan.

Support

Use the available communication channels to communicate your ideas, problems or suggestions.

License

Lelylan is licensed under the Apache License, Version 2.0.

lelylan's Issues

Add a section to add questions?

Maybe could be helpful or we can just use stacktrace, but here a solution to handle questions from users. Already registered as lelylan/[email protected].

https://goresponsa.com/en/dashboard

Node Client Library

Write a Javascript client library. Take some inspiration from libraries that you can find on node toolbox

Widgets & Themes

Start thinking about a way to visualize your data that can be enough free to be built from the community. Some JS libraries can be used to define some basic widgets (for types and for devices), but at the same time we need to define also a theme system so that it can be easy enough to personalize all styles with specific icons and color sets.

This should be open-source and could be used to build up a community around.

Event property-updated

Right now the data we save when properties change and functions are executed is not what we want. We save only the value, but we need to send also the URI and the ID.

Market Store

Lelylan Market store must be able to collect an high number of services coming from outside. We shouldn't limit the capability of third party to create new solutions. This means that somebody add an application to the market store by its own and we will simple validate the acceptance of the application. This should be pretty flexible and let us give a minimal control. A small inspiration can be given by the add on console of heroic. It will be quite important to have different type of organizations (stars, new, categories). Learn more from existing stores.

Actually an app on the market store is a registered app with few more logics. When clicking on the application icon you go to the OAuth page where you grant the access for the specific application. Together with this, the only difference lives in the capability of the market store. You do not change anything. The app automatically set a set of webhooks for the application through the APIs. This means that the app store is just an interface to the accepted applications. It will be the new authorized applications page, with some more features.

When the user accepts the application you get a confirmation that the process has been successfully added. At this point you have a link to the dashboard and you can highlight the new link to the visual part of the application if existing.

And here comes the visual part. An application can be made in two ways. The first one logical and the second one is visual. The logical one will only use the webhooks. The visual one will give you a new User Experience. This user experience lives in another website and we should not care about this. But here raises a linking problem. We need to uniform the User interface, as little as we can, but giving a fast way to move between all existing APPS.

Actually we can give an header. This header contains all common links like

• Home
• Admin
• Logout
• Applications

Between those you see applications. This page will redirect the user to the authorized apps (filtered through visualization). Through this page I'll be able then to move to other apps. This is a 2 click system and it would be nice to have a list of all accessible UI on the top menu. For the first step keep using common parts.

This system should be also available to build a single (or defined set) of products market. Here the thing starts becoming complicated as we set a number of applications only for a set of devices (each app can have all, some devices, and it should be listed on the app page).

This means that when I'm in the dashboard of Lelylan I see all the apps and for each app all the devices associated. But when I access the market store defined for another product, I should filter only the apps for that market store and for the specific device(s).

Think at this scenario. I've Lelylan where I've plenty of devices and plenty of applications added. For each application I've a list of devices (probably this list is given from a set of tokens, also for the same app? Do not overcomplicate!!) Then I go to light.me. In here I register to Lelylan and add my light. Then I go to the market place and I see my light. In here I see also the possible applications I can use for this light and I decide which ones to add. This means that I do not see any of the other applications in here, but when I go back to Lelylan I see also this device with the added applications.

The market store definition is quite critical and complex. Before starting making it take at least a couple of days and split its definition in several mailstones. We need to be able to provide a basic service, and then we can add more and more functionalities. This implementation requires at least a couple of monist.

Resource owner id

Add resource_owner_id to all models. Remove created_from.

• location
• type
• device

Data limit

Find a way to limit access on data saved. This could be perfect to let people experiment but it looks pretty complex as we use different databases. I guess the main limit is related to the history and to the consumptions.

Settings cleanup

Cleanup the settings file in testing.

Demo

Add a basic demo where you show a fake set of devices that people can use. It would be nice if in there we could use a widget system that everyone can contribute at.

Icons

Write to Melissa to meet and give her a full set of icons she should draw, maybe with the help of Ale. Start writing a Docs where to define all of them so that we have an idea of what we need.

Improve URI validator

Right now the URI validator we have created works both for one URI (String) and a list of URIs (Array). To make the validator complete you must be able to set an optional parameter to set the fact that the field can accept URIs or an URI.

Something like this should work.
validates list: uri: {array: true} # accept list of uris

This is needed because if later on we want to make a check on the URI to extract something we could have some errors and it would be better to find it before. If we do not have this validation option we could send an array of URIs where only one can be accepted and have unexpected errors.

End Beta Invitation

Remove invitation system.

To remove it find out all files with the key OPEN_SIGNUP. They will get you little explanation on what can be done. Mainly the sections are:

• model :inevitable key and fields.
• routing, controller and views to delete.
• remove tests related.
• add signup tests.
• add signup links.

Services name

Set all services name to plural. Devices, Types, Locations and People.

Communication webhooks blocking

This is a complex system. You could have the need related to the fact that when a webhook 'halt' the execution, the physical change shouldn't be made.

Actually this is not easy to make and could delay the execution of lots of commands. A probably better and simple solution could be to check one service at a time in an esier way. I first run the rule engine, and then eventually apply the physical change. Things are similar.

Find the simplest solution.

HTTPS

Add HTTPS support.

New shared examples

Apply shared examples on all controllers on all services.

JSON 5xx

Set the 500 family pages as JSON.

Scopes

Rethink at how to structure the scopes. Right now we are using read+write but its a bit too general, also in vision of adding new resources, we should make it much more modular.

• Devices: devices.read, devices.write
• Devices: locations.read, location.write

Plus ability to select devices and locations to access at. Right now we handle only devices, but it would be nice to let people see only specific locations. For example if I have 10 houses I do not want anyone know I've all of them. Think really well about this.

A first step is to define scopes and use only devices filter.
A second step is to understand how to filter also locations.

JSON Errors

Add issue to set json errors as json and not as message. This should be possible using Mongoid 3.1.

Invitation API

Start thinking at how to handle a more advanced authorisation solution where a device functionalities is accessible by many users, which in turns are able to forward the authorisation to other people.

Explore.

Add OAuth2 to all services

• people
• location
• types
• devices

Error handling

Improve error handling using the ErrorResponse class defined in Doorkeeper. Reorganize also the classes you use and apply the concern concept on people, location, types and devices

401

Make return a presonalized 401 error response in JSON from Doorkeeper.

• people
• location
• types
• devices

Wizards

Remember to keep big attention on how you guide the user. As Cosm shows in its quickstart remember to give some easy tool. As RWW said, we must make this for everyone.

Weird URI validations (sometimes does not validates)

We found out a strange behavior on errors. While creating a type if I send a category with not valid uris it should raise an error but it does not. It just does not connect anything in the category connection. This is really strange as validations are active and on model tests are working just fine. Find out why it act like this.

Invitation API

Start thinking at how to handle a more advanced authorisation solution where a device functionalities is accessible by many users, which in turns are able to forward the authorisation to other people.

Explore.

CORS

Add CORS rack middleware to all API services and let the ability to access to all services from everywhere. We need maximum flexibility in the beginning.

JSON encoding

When there is a validation error the JSON that you get back returns encoded in a strange way, like it it was not understanding it was JSON. This creates problem to show the JSON in the 422 rendering because there is no way to show the JSON, as it gets broken from the methods I've made to clean it up. You can find the code at the file helper_error.rb, function clean_body. The problem is well explained in stackoverflow.

Apply on people, location, type and device services.

Resource Owner Uri

All resources should have the resource owner uri in its representation.
Device should also have the creator uri.

Request limit

Add limit to the possible request. By default a user get a specific limit but through an API it should be possible (for admins) to access and change it. Then, going forward in time, just think at the payment service.

Gem for validators

Make a gem for validation uri and validation ownership. In this way you have them easily tested and working smoothly.
If you use them in into initializers you could die waiting for the tests to run.

Improve Validation

Find the best way to add validation at inputs that are not directly binded to model fields. This should be a clean way to validate URIs and more, as we need to check different stuff.

Webhooks

Add webhooks.

attr_protected

Use attr_protected instead of attr_accessible.

Services name

Set all services name to plural. Devices, Types, Locations and People.

Market store

Think at a structure to build a market store. We need to think at two main scenarios.

• One where applications lives into Lelylan.
• One where applications can live into an independent service that can include some of them. This is really important if we want to build a smart light for Kickstarter and let people add any kind of plugin. If they add it in Lelylan they get also to know the platform.

Services Communication

Think more at the structure they are built and experiment the usage of MongoDB copped collections (see Heroku article).

MongoDB capped collections let MongoDb push data to all listeners. Capped collections are high performance, with the simple tradeoff about their size and about the impossibility of not being shareable. This could be a basic flow to use.

• Ruby save the property changes on a capped collection.
• Listening Node.js webhooks service send a request to all connected services.
• Listening Node.js physical device service send a request to the physical device.
• Listening websocket app send realtime updates to the client.

At this point

• Node.js webhooks service call all service. If None of those services are blocking you can directly call the physical device. If some of those services are blocking you need to organize a simple protocol to block physical changes and to eventually store previous state (you should pass the link at the history resource before the last change). This is really important for interconnecting services such as rule engine. They could halt a request, and for this reason we can't call the physical device service directly.
• Node.js physical device when receiving the physical device response updates the device representation. Actually this service could be a webhook uri that is activated in the moment you have a physical device. Think about this possibility. It looks quite good and modulable.
• Node.js websocket always send a notification for every change is happening to the physical world.

Caching

Add caching. Read documentation in the tech doc.

• HTTP caching
• fragment caching + action caching

README

Add a standard README to all services. It's also needed a docs documenting the structure of the services and their key points.

JSON error parsing

With rails 3 exception handling is becoming more complex. As some functionalities are defined into middleware you can not catch them using rescue_from. Right now there is not a proper solutions. We need to find (or wait for) a good one mainly to be able to catch malformed JSON and give a notification to the final user.

Add index

Add index to mongoid (people, types and devices) and to postgres (location).

Diagnostic services

Think at some diagnostic services that can be used by the creator of the device to monitor the device status. This could be a service you sell to manufacturers, so think about.

Try added services

Test on production

• /activation services
• /devices/:id/accesses.
• Check all signatures (sent from physical and sent from /accesses)

Physical nodes

Build a physical node using MQTT and one using WebSocket.

Physical Devices.

Think at the implementation of a physical node (or whatever you call it) and start building one. Before think about the communication flow with Lelylan and then start making it. First, it could be easier to use Pusher for this. It already exists. Second start to build something new with Erlang (Pilu) od Mosquitto (seems to have pretty good open-source project and its under heavy development).

Streaming

Add and learn how to build a streaming channel.

Make API tests work with rack tests

This article states how we do not need to use capybara to test our API, as what we do is simply using a 'dirty version of rack test' when using capybara for API. Really interesting. Read it.

http://www.elabs.se/blog/34-capybara-and-testing-apis

Add performance check services

Add New Relic, Pingdom and Blits. Learn really well how to use them and fine tune your application to be able to improve the throughput to the maximum allowed level.

Move consumptions to new service

Rule engine

Define a rule engine. Check the Cosm study as we put some lines on how to make it.