murraju / onyx

Distributed, fault tolerant data processing for Clojure

Onyx

What is it?

• a cloud scale, fault tolerant, distributed computation system
• written in Clojure, for Clojure
• batch and stream processing hybrid
• exposes an information model for the description and construction of distributed workflows
• enabled by hardware advances in the last 10 years
• Competes against Storm, Cascading, Map/Reduce, Dryad, Apache Sqoop, Twitter Crane, etc

What would I use this for?

• Realtime event stream processing
• Continuous computation
• Extract, transform, load
• Data transformation à la map-reduce
• Data cleaning
• Data ingestion and storage medium transfer

Installation

Available on Clojars:

[com.mdrogalis/onyx "0.3.2"]

Quick Start Guide

Feeling impatient? Hit the ground running ASAP with the onyx-starter repo and walkthrough.

User Guide 0.3.2

• What does Onyx offer?
• Concepts
• Environment
• Hardware
• APIs
• Constraints
• Architecture
• Coordinator and Peer Configuration
• Information Model
• Plugins
• HornetQ Internal Plugin
• Job and Peer Execution Scheduling
• Reliability Guarantees
• Coordinator High Availability
• Logging
• Performance Tuning
• Examples
• Frequently Asked Questions

Official plugin listing

Official plugins are vetted by Michael Drogalis. Ensure in your project that plugin versions directly correspond to the same Onyx version (e.g. onyx-core-async version 0.3.2 goes with onyx version 0.3.2). Fixes to plugins can be applied using a 4th versioning identifier (e.g. 0.3.2.1).

• onyx-hornetq
• onyx-datomic
• onyx-sql
• onyx-kafka
• onyx-core-async

Generate plugin templates through Leiningen with onyx-plugin.

Need help?

Check out the Onyx Google Group.

Want the logo?

Feel free to use it anywhere. You can find a few different versions here.

Running the tests

A simple lein midje will run the full suite, which takes about 15 minutes on my quad-core MacBook Pro. Expect a long, 5-10 minute pause during the coordinator tests as it's being completely slammed with requests and checked for correctness. The pause is waiting for the Coordinator to catch up and close out cleanly.

Contributing

Contributions are welcome. Please fork the repository and send a pull request to the master branch.

Branching

Onyx uses a similiar branching strategy to Clojure itself. Onyx uses semantic versioning, and each minor version gets its own branch. All work is done on master or feature branches and dropped into a major.minor.x branch when it's time to cut a new release. Pull requests into the master branch are welcome.

Commit rights

Anyone who has a patch accepted may request commit rights. Please do so inside the pull request post-merge.

Contributor list

• Michael Drogalis
• Owen Jones
• Bruce Durling

Project Maturity

Onyx is not a battle-tested framework compared to Storm or Cascading. Please keep this in mind when considering taking it to production. That being said, the test suite uses simulation testing to create exceptionally bad data center network partitions, and checks against a number of correctness criteria afterwards. Community effort will help a great deal on this front.

It's worth noting that nearly all inter-cluster communication happens via HornetQ. HornetQ is a battle tested queueing platform from the JBoss stack that has been in production for many years. This does give Onyx a significant leg up on reliability.

Performance Benchmarks

At the time of writing this, I do not have any performance benchmarks to publish. Creating a correct, useful benchmark is extremely difficult. I'm working on it - hang tight.

Author

This project is authored by Michael Drogalis, an independent software consultant. Get in touch (mjd3089.at.rit.edu) to work together.

License

Copyright © 2014 Michael Drogalis

Distributed under the Eclipse Public License, the same as Clojure.