Coder Social home page Coder Social logo

fbzmq's Introduction

fbzmq

Build Status

fbzmq provides a framework for writing services in C++ while leveraging the awesomeness of libzmq (message passing semantics). At a high level it provides

  • Lightweight C++ wrapper over libzmq which leverages newer C++ constructs and stricter type checking. Most notably it provides the ability to send/receive thrift objects as messages over wire without worrying about wire encoding/decoding protocols.
  • Powerful Async Framework with EventLoop, Timeouts, SignalHandler and more to enable developers to write asynchronous applications efficiently.
  • Suite of monitoring tools that make it easy to add logging and counters to your service.

Examples

Here is a simple example demonstrating some powerful abstractions of fbzmq which makes writing asynchronous applications easier on top of libzmq

// Create context
fbzmq::Context context{1};

// Create eventloop
fbzmq::ZmqEventLoop evl;

// Create thrift serializer
apache::thrift::CompactSerializer serializer;

// Create REP and PUB server sockets
fbzmq::Socket<ZMQ_REP, ZMQ_SERVER> reqSocket;
fbzmq::Socket<ZMQ_PUB, ZMQ_SERVER> pubSocket;
reqSocket.bind("tcp://[::1]:12345");
pubSocket.bind("tcp://[::1]:12346");

// Attach callback on reqSocket
evl.addSocket(RawZmqSocketPtr{*reqSocket}, ZMQ_POLLIN, [&](int evts) noexcept {
  // It must be POLLIN event
  CHECK(evts | ZMQ_POLLIN);

  // Read request
  auto request = reqSocket.recvThriftObj<thrift::Request>(serializer);

  // Send response
  thrift::Response response;
  response.request = request;
  response.success = true;
  reqSocket.sendThriftObj(response, serializer);
});

// Create periodic timeout to send publications and schedule it every 10s
auto timeout = fbzmq::ZmqTimeout::make(&evl, [&]() noexcept {
  thrift::User user;
  user.name = "TestPerson";
  user.email = "[email protected]";
  pubSocket.sendThriftObj(nodeData, serializer);
});
timeout->scheduleTimeout(std::chrono::seconds(10), true /* periodic */);

// Let the magic begin
evl.run()

Requirements

We have tried fbzmq on Ubuntu-14.04, Ubuntu-16.04 and CentOS-7. This should work on all Linux based platforms without any issues.

  • Compiler supporting C++14 or higher
  • libzmq-4.0.6 or greater

Building fbzmq

Repo Directory Structure

At the top level of this repo are the build and fbzmq directories. Under the former is a tool, fbcode_builder, that contains scripts for generating a docker context to build the project. The fbzmq directory contains the source for the project.

Dependencies

  • cmake
  • gflags
  • gtest
  • libsodium
  • libzmq
  • zstd
  • folly
  • fbthrift

One Step Build - Ubuntu-16.04

We've provided a script, build/build_fbzmq.sh, well tested on Ubuntu-16.04, to install all necessary dependencies, compile fbzmq and install C++ binaries as well as python tools. Please modify the script as needed for your platform. Also, note that some library dependencies require a newer version than provided by the default package manager on the system and hence we are compiling them from source instead of installing via the package manager. Please see the script for those instances and the required versions.

Build using Docker

Learn more here.

Build Steps

// Step into `build` directory
cd build

//  Install dependencies and fbzmq
sudo bash ./build_fbzmq.sh

make test

Installing fbzmq

fbzmq builds a static library and install steps installs library as well all header files to /usr/local/lib/ and /usr/local/include/ (under fbzmq sub directory)

sudo make install

Build and Install python libraries

cd fbzmq/fbzmq/py
python setup.py build
sudo python setup.py install

How fbzmq works

zmq/* are a straight forward C++ wrappers over raw libzmq objects (like zmq_msg_t, zmq_ctx_t, etc) in C. ZmqEventLoop is an event loop which allows you to attach different event callbacks on sockets as well as schedule timeouts. Internally it maintains a dynamic poll-list which is updated only when new socket/fd is added/removed and uses zmq_poll APIs for polling the list. It also maintains internal heap of timeouts ordered on their expiry times.

Full documentation

To understand how to use library, take a look at examples/ directory. All of our code is very well documented and refer to appropriate header files for up to date and detailed documentation of various APIs/functionalities.

License

fbzmq is MIT-licensed.

fbzmq's People

Contributors

ahornby avatar amyreese avatar avalonalex avatar bkoray avatar chadaustin avatar cooperlees avatar dgrnbrg-meta avatar fanzeyi avatar genevievehelsel avatar jstrizich avatar lnicco avatar lukaspiatkowski avatar mcallahan avatar michel-slm avatar mizuchi avatar orvid avatar pedroerp avatar phshaikh avatar pkaush avatar saifhhasan avatar shri-khare avatar simpkins avatar snarkmaster avatar vitaut avatar wez avatar xavierd avatar yfeldblum avatar yi-xian avatar yns88 avatar zertosh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fbzmq's Issues

Compilation error on PowerPC

[ 50%] Building CXX object CMakeFiles/fbzmq.dir/service/logging/LogSample.cpp.o
/builder/powerpc_8540/build/sdk/build_dir/target-powerpc_8540_musl/fbzmq-2019.06.10.00/fbzmq/async/ZmqTimeout.cpp: In static member function 'static std::unique_ptr<fbzmq::ZmqTimeout> fbzmq::ZmqTimeout::make(fbzmq::ZmqEventLoop*, fbzmq::TimeoutCallback)':
/builder/powerpc_8540/build/sdk/build_dir/target-powerpc_8540_musl/fbzmq-2019.06.10.00/fbzmq/async/ZmqTimeout.cpp:40:59: error: 'new' of type 'fbzmq::{anonymous}::ZmqTimeoutWrapper' with extended alignment 16 [-Werror=aligned-new=]
       new ZmqTimeoutWrapper(eventLoop, std::move(callback)));
                                                           ^
/builder/powerpc_8540/build/sdk/build_dir/target-powerpc_8540_musl/fbzmq-2019.06.10.00/fbzmq/async/ZmqTimeout.cpp:40:59: note: uses 'void* operator new(std::size_t)', which does not have an alignment parameter
/builder/powerpc_8540/build/sdk/build_dir/target-powerpc_8540_musl/fbzmq-2019.06.10.00/fbzmq/async/ZmqTimeout.cpp:40:59: note: use '-faligned-new' to enable C++17 over-aligned new support

I'm guessing this is a C++17 feature being used.

https://downloads.openwrt.org/snapshots/faillogs/powerpc_8540/packages/fbzmq/compile.txt

Add conan support to our cmake

Please use this template for reporting suspected bugs or requests for help.

Issue Description

Have support for Conan to load the library in CMake

Environment

Linux

  • tag or commit hash on which this occured
  • OS version: <e.g. ubuntu-16.04>
    ...

Minimal test code / Steps to reproduce the issue

What's the actual result?

What's the expected result?

fbzmq publisher unable to publish on topic

Please use this template for reporting suspected bugs or requests for help.

Issue Description

From the current documentation and examples, it is unclear to me whether the API supports publishing a message on a specific topic. Is this possible using the fbzmq API?

Environment

  • tag or commit hash on which this occured
  • OS version: <e.g. ubuntu-16.04>
    Mint 18.3 (Ubuntu 16.04)

Minimal test code / Steps to reproduce the issue

What's the actual result?

Current send*() methods on the API do not seem to support publishing on a topic for a ZMQ_PUB socket.

What's the expected result?

send*() methods on the API have a parameter to publish on a topic.

Update deps_common.sh, please

Issue Description

The current dependency install script seems to be out of date.
I experienced the following issues:

  • THRIFT_COMPILER_INCLUDE was set to NOTFOUND and I had to manually download the ThriftLibrary.cmake since the current script points to a tag that does not contain this script (and had to set to that path manually - how is it supposed to be set?)
  • Afterwards PROTOCOL and TRANSPORT are still set to NOTFOUND and I have no clue as to which libraries these are.

Environment

  • master branch, latest commit b542717
  • OS version: ubuntu-16.04
    ...

Minimal test code / Steps to reproduce the issue

  • setup clean ubuntu 16.04 vm, clone git, run deps_ubuntu_16.04.sh, run "cmake .."

What's the actual result?

  • "CMake Error at CMakeLists.txt:32 (include):
    include could not find load file:

    THRIFT_COMPILER_INCLUDE-NOTFOUND/thrift/ThriftLibrary.cmake

    CMake Error at CMakeLists.txt:40 (thrift_object):
    Unknown CMake command "thrift_object".

    -- Configuring incomplete, errors occurred!
    See also "/home/c1/fbzmq/fbzmq/build/CMakeFiles/CMakeOutput.log"."

and:

  • "CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    PROTOCOL
    linked by target "fbzmq" in directory /home/c1/fbzmq/fbzmq
    TRANSPORT
    linked by target "fbzmq" in directory /home/c1/fbzmq/fbzmq"

What's the expected result?

  • cmake should run without errors

Pass

Please use this template for reporting suspected bugs or requests for help.

Issue Description

Environment

  • tag or commit hash on which this occured
  • OS version: <e.g. ubuntu-16.04>
    ...

Minimal test code / Steps to reproduce the issue

What's the actual result?

What's the expected result?

Server/client

Hi,
I'm trying to use server/client example to create more realistic POC with fbzmq. What I'm trying to achieve is the multiple clients accessing single server asynchronously. So, I guess I need DEALER socket on client and ROUTER socket on server, which allow non-blocking request/response. Right? Should I explicitly specify fbzmq::NonblockingFlag{true} on socket? On both sides? Should I do my own enveloping on messages when I work asynchronously or fbzmq makes it transparent for me?
Anything else I have to take into consideration when designing such a pattern?

Sincerely,
Ernest

fbzmq threading model

Hi,
I have a question about processing messages the right way. I see that the socked callback attached to the socket is executed in the same thread where the server's (referring to example server) method run() is called. Meaning it would kill any asynchronous calls to the server since the message processing (I guess message receiving made on zmq polling threads) may take a long time, and thus blocking parallel message processing. What is the right approach to employ in fbzmq? is there any build-in mechanism to offload message processing to a worker thread pool? Should I use my own? If later, how socket callback should treat the sendOne part of the server?
Sincerely,
E

fbzmq memory model

Hi,
Quick question. Somewhere in the build script I saw jemalloc mentioned. How it would play with my application that uses tcmalloc tailored to work with shared memory allocated on hugepages? can I disable jemalloc? Can I (easily) replace it with my tcmalloc implementation?
Sincerely,
Ernest

error: lambda capture 'kCount' is not required to be captured for this use

/usr/ports/net/fbzmq/work/fbzmq-2020.09.07.00/fbzmq/async/tests/ZmqEventLoopTest.cpp:292:36: error: lambda capture 'kCount' is not required to be captured for this use [-Werror,-Wunused-lambda-capture]
    evl.scheduleTimeoutAt(now, [i, kCount, &count, &evl]() noexcept {
                                 ~~^~~~~~
/usr/ports/net/fbzmq/work/fbzmq-2020.09.07.00/fbzmq/async/tests/ZmqEventLoopTest.cpp:320:36: error: lambda capture 'kCount' is not required to be captured for this use [-Werror,-Wunused-lambda-capture]
    evl.scheduleTimeoutAt(now, [i, kCount, &count, &evl]() noexcept {
                                 ~~^~~~~~
2 errors generated.

Version: 2020.09.07.00
clang-10
FreeBSD 12.2

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.