Linux/OSX | Windows | Coverity |
---|---|---|
Integrating different products and technologies is often an expensive part of product development, and can slow down innovation. Sometimes it is a bump in the road, and sometimes it is a 4000ft mountain that must be climbed. There are companies that make lots of money from selling products that solve the integration problem.
We believe time and money are better spent on innovation, not integration, and that the only way to reduce the cost of integration is large-scale collaboration. To kickstart this collaboration we need an independent, open source, liberally licensed framework that offers a structured approach to 1) sharing projects and 2) sharing data between projects.
That is why, for the past 5 years, we have been working on Corto.
Corto (corto /‘korto/ adjective, Spanish. 1 short in length or duration) is a library that provides:
- a connector framework for connecting any kind of data (realtime or at rest) and
- a simple application-facing API to interact with this data and
- a package management framework for sharing corto applications and packages
The small footprint of Corto makes it a handy building block in IoT systems where you can run it on small devices. The library is optimized to be very fast (~30 nanoseconds event latency) and to enable working with large datasets in resource-constrained environments. Corto never loads more data in memory than the application needs.
People have used Corto to forward data from one technology to another (for example, MQTT to InfluxDB), as a data integration platform in industrial automation systems, and even as a low-footprint web backend.
This repository contains the core library of corto and the corto tool. The corto tool is like your swiss army knife when using Corto. It helps you create, build, debug, test and install corto projects, and a whole lot more.
Corto has been validated on the following platforms:
- Ubuntu 14.04 (32/64 bit)
- OS X 10.10.2 (64 bit)
- Yocto 2.1 (32 bit)
Make sure to also explore the other repositories in the cortoproject organization:
The cortoproject/web
project in particular is worth checking out. It automatically adds web connectivity to your applications, like a REST interface and websockets.
The cortoproject/admin
project is a very handy web-application that allows you to browse through the data that is connected to Corto. Make sure to also install cortoproject/web
if you want to use admin.
Corto uses rake (https://en.wikipedia.org/wiki/Rake_(software) for building, which is a platform-independent, ruby based buildsystem written by Jim Weirich. Before building corto, make sure to have rake installed. To build the latest version, use the following commands (on Ubuntu):
sudo apt-get install rake libffi-dev
git clone https://github.com/cortoproject/corto
cd corto
source configure
rake
This will build the core API and corto tool. The core API is typically used for writing connectors and generic tools. All other APIs are built on top of the core API.
The following packages are useful when you want to do corto application development (all packages are in the cortoproject organization):
- c-binding
- xml
- corto-language
- test
These packages are only required during development, and you do not need them when you are distributing your application. The c-binding
API contains code generators that take as input a datamodel, which can be specified in either xml
or the native corto definition language (corto-language
). The test
package contains a testframework for testing corto projects.
The following libraries are required for these packages:
- libxml2-dev
- flex
- bison
To build the latest versions of these packages, do (on Ubuntu):
sudo apt-get install libxml2-dev flex bison
git clone https://github.com/cortoproject/c-binding
git clone https://github.com/cortoproject/xml
git clone https://github.com/cortoproject/corto-language
git clone https://github.com/cortoproject/test
corto build c-binding xml corto-language test
The corto build
tool is a front-end for the rake-based buildsystem. The buildsystem automates many tasks like code generation, dependencies between packages and installation of binaries.
To create a new project, use the following command:
corto create MyApp
Subsequently, you can run this project by typing (it will build automatically):
corto run MyApp
Creating a package is similar to creating an application. To create a new package, use the following command:
corto create package mypackage
Notice that the package has been created with a mypackage.cx
file. This file will contain the datamodel for the package in the native corto language. Here is an example datamodel:
#package /mypackage
class WeatherStation::
temperature: float64
pressure: float64
precipitation: float64
When you change the mypackage.cx
file, simply use corto build mypackage
to generate code from the updated definition and rebuild the project. Corto generates a simple CRUD API to interact with data in a corto system. Here is a small C application that uses the generated code from the example datamodel:
// Create a new weatherstation instance
mypackage_WeatherStation ws = mypackage_WeatherStationCreateChild(root_o, "ws", 65.0, 500.0, 0.5);
// Update the instance with new sensor readings
mypackage_WeatherStationUpdate(ws, 67, 600, 0.5);
Corto has a plugin-architecture with add-ons stored in a hierarchically organized package repository. Corto stores packages in well-defined locations, so you can load packages by their logical name, instead of some path to a file. This makes it easier to share packages and their dependencies across operating systems.
Packages are stored in ~/.corto
, where ~
is your home directory. Inside you will find the following subdirectories:
bin etc include lib
bin
contains applications, lib
contains packages, include
contains include files and etc
contains all kinds of supporting files. It is not a coïncidence that this structure looks similar to the /usr/local
directory on Linux. When you install a corto project, files are copied to /usr/local
.
Files in each of these directories are stored in corto/0.2
, where 0.2
is your current major and minor corto version. From there the file structure represents the hierarchical organization of the packages. For example, the corto/fmt/xml
package is stored here:
~/.corto/lib/corto/0.2/corto/fmt/xml/libxml.so
Include files for the xml
package are stored here:
~/.corto/include/corto/0.2/corto/fmt/xml
Corto projects use the fully qualified include path, so that name clashes between packages are prevented. To include the XML project, do:
#include <corto/fmt/xml/xml.h>
Because packages are not stored in a single directory (to prevent name clashes) corto bakes hard-coded paths to packages path into your applications. This ensures that application always uses the right library.
Sometimes these hardcoded paths can be inconvenient, for example when you want to put corto libraries in another project. For this purpose, the corto buildsystem also builds a binary that does not hardcoded paths, and you can easily embed in other projects. The buildsystem will tell you where it stores this binary.
Corto can translate a language-independent description of a datamodel into a target programming language. The corto library has an comprehensive framework for describing types, and from these types code can be generated. At this point the only well-supported language is C
, with C++
and python
bindings underway.
Corto generates code by running the corto preprocessor, which is part of the corto tool. The buildsystem calls corto pp automatically when you have a definition file in your project that has the same name as your project. For example, the foo
project can have a foo.cx
definition file that contains the datamodel.
When you want to use corto code generation, just put a definition file in your project, and build it with corto build
.