jeffreywugz / zookeeper-mq Goto Github PK
View Code? Open in Web Editor NEWThis project forked from andreisavu/zookeeper-mq
Reliable and Highly Available Message Queue based on ZooKeeper (Prototype)
License: Apache License 2.0
This project forked from andreisavu/zookeeper-mq
Reliable and Highly Available Message Queue based on ZooKeeper (Prototype)
License: Apache License 2.0
Message Queue based on ZooKeeper ================================ NOTICE: This is work in progress and it's not production ready (more like a proof of concept) NOTICE: It should be easy to rewrite the code in Java by following the same concepts. Objectives ---------- Build a highly available & reliable message queue. The main focus is not on perfomance! This queue should be able to survive hardware and software upgrades and failures with no downtime for years :) The queue should be accesible using both the default zookeeper binary protocol and the zookeeper rest interface. ZooKeeper Data Tree Structure ----------------------------- <chroot-prefix OR />/ /queue /items/ /item-000-1 /item-050-2 ... /partial - items reserved but not marked as done (consumer failure) /item-1 /item-2 ... /consumers /consumer-1 /active - ephemeral znode /item /consumer-2 /active - ephemeral /item ZooKeeper Availability ---------------------- A ZooKeeper cluster with N servers will still be available if CEIL(N/2)-1 nodes fail! E.g. if you run a cluster with five nodes you can take one offline for maintenance anytime and you will still be able to handle an unexpected failure. Requirements ------------ Tested with Python 2.6 and zkpython 3.4.0--1, built on 02/19/2011 14:52 GMT You also need simplejson for the rest interface How to install zkpython? ------------------------ Fetch the latest sources from the trunk and run: $ apt-get install sun-java6-jdk libtool libcppunit-dev automake build-essential ant $ cd zookeeper $ ant compile compile_jute $ cd src/c/ $ autoreconf -if $ ./configure $ make && sudo make install $ cd ../contrib/zkpython $ ant compile $ sudo ant install After all this you should be able to do: $ python -c "import zookeeper;print zookeeper.__version__" 3.4.0 Fault Injection Testing Scenarios --------------------------------- I'm not going to test the producer and the consumer code. I'm trying to to test the availability and reliability of the ZooKeeper quorum when used for storing the queue data. Start a ZooKeeper quorum and the fault injector: $ cd test_5_trunk $ ./clean.sh $ ./start.sh $ python fail.py leader Push 500 elements: $ ./put_all.py or $ PYTHONPATH=lib.linux-x86_64-2.6/ LD_LIBRARY_PATH=lib.linux-x86_64-2.6/ ./put_all.py Check 500 elements (also checks that the sequence is strictly increasing): $ ./check_all.py or $ PYTHONPATH=lib.linux-x86_64-2.6/ LD_LIBRARY_PATH=lib.linux-x86_64-2.6/ ./check_all.py It should work like a charm! :) Keep in mind that you are not going to experience this amount of critical failures in production. License ------- Apache License Version 2.0 Resources --------- Inspired by this article wrote by Henry Robinson on the Cloudera blog: http://www.cloudera.com/blog/2009/05/building-a-distributed-concurrent-queue-with-apache-zookeeper/
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.