facebookarchive / logdevice

Distributed storage for sequential data

LogDevice

Note: This is an archived project and is no longer supported or updated by Facebook.

LogDevice is a scalable and fault tolerant distributed log system. While a file-system stores and serves data organized as files, a log system stores and delivers data organized as logs. The log can be viewed as a record-oriented, append-only, and trimmable file.

LogDevice is designed from the ground up to serve many types of logs with high reliability and efficiency at scale. It's also highly tunable, allowing each use case to be optimized for the right set of trade-offs in the durability-efficiency and consistency-availability space. Here are some examples of workloads supported by LogDevice:

• Write-ahead logging for durability
• Transaction logging in a distributed database
• Event logging
• Stream processing
• ML training pipelines
• Replicated state machines
• Journals of deferred work items

For full documentation, please visit logdevice.io.

Requirements

LogDevice is only supported on Ubuntu 18.04 (Bionic Beaver). However, it should be possible to build it on any other Linux distribution without significant challenges. LogDevice relies on some c++17 features, so building LogDevice requires a compiler that supports c++17.

Quick Start

• Run a local cluster in Docker

Documentation

• Overview
• Concepts and Architecture

Build LogDevice

• Build

Join the LogDevice community

• Facebook: LogDevice Users Group
• GitHub Issues

Contributors

See the CONTRIBUTING file for how to help out.

License

LogDevice is BSD licensed, as found in the LICENSE file.

logdevice's Issues

Find boost-python reliably cross linux distributions

This issue breaks out the second point from #31 into a separate issue.

The below text is copied from the original issue
The _boost_py_component is not correctly set when building on Fedora, causing boost-python to not be found. I was able to make it work by setting _boost_py_component to python3, but that's obviously not an acceptable solution for all distros. Not sure how to solve this one, I remember hearing something about the struggles with this versioning last summer. There seems to be a solution for boost python libs merged into CMake 3.11, but I didn't look into it too deeply.

Error message when not finding the correct boost python version.

CMake Error at /usr/share/cmake/Modules/FindBoost.cmake:1879 (message):
  Unable to find the requested Boost libraries.

  Boost version: 1.60.0

  Boost include path: /usr/include

  Could not find the following Boost libraries:

          boost_python-py35

  Some (but not all) of the required Boost libraries were found.  You may
  need to install these additional Boost libraries.  Alternatively, set
  BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
  to the location of Boost.
Call Stack (most recent call first):
  CMake/logdevice-deps.cmake:16 (find_package)
  CMakeLists.txt:34 (include)

Add support for dynamically opening and closing shards (disk hot swap)

Right now, LogDevice needs to be restarted whenever a shard gets disabled, in order for the filesystem to unmount. This restart is quite a heavy operation (sequencers move around, recoveries, ...) that can be simplified by stopping and starting the RocksDB instances while running.

This could perhaps be implemented using an Admin command.

Suboptimal usage of shared_ptr in a multi_index_container

You have the following in AllServerReadStreams.h:

https://github.com/facebookincubator/LogDevice/blob/d706fcac4c77715b2acb56d0e292a9a4b9614971/logdevice/server/read_path/AllServerReadStreams.h#L402-L405

Actually, multi_index_container does support in-place construction of elements via emplace, so the use of shared_ptrs here is unnecessary, and suboptimal.

Add a way to configure which plugins to use

Currently if multiple plugins are loaded of the same type (aside from built-in plugins and plugins that allow multiple active instances of the same types - e.g. ConfigSources), PluginRegistry will fail, as it doesn't have a way to decide which one of those to use.

Instead it should be configurable (via settings or via the configuration file) which plugin to use for each plugin type. Then PluginRegistry::setCurrentSinglePlugins() should use that information to pick the active plugin:
https://github.com/facebookincubator/LogDevice/blob/e6424a70735b589d0be1ed8a25f7a273a2198df3/logdevice/common/plugin/PluginRegistry.cpp#L85-L88

It should also subscribe to config/setting changes so that you can reconfigure which plugin to use on the fly.

New docs test prevents unit tests running when it fails

Unit tests need to run regardless of the outcome of the "docs" test

Stop and start logdevice cluster(4 server), client get SIGABRT

W0817 11:08:48.520524  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 202767 msec: BUFFERED_WRITER_APPEND (id: 25769808310), p :HI_PRI
W0817 11:08:48.520537  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 202748 msec: BUFFERED_WRITER_APPEND (id: 171798693947), p :HI_PRI
E0817 11:08:48.526716  108418 [logdevice:WG18] BufferedWriterImpl.cpp:308] execute() skipped at least 1000 log entries
E0817 11:08:48.526749  108418 [logdevice:WG18] BufferedWriterImpl.cpp:308] execute() BufferedWriterShard with id 86 not found.  Unexpected!
W0817 11:08:48.526799  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() skipped at least 1000 log entries
W0817 11:08:48.526815  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182733 msec: BUFFERED_WRITER_APPEND (id: 25769808644), p :HI_PRI
W0817 11:08:48.526862  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182714 msec: BUFFERED_WRITER_APPEND (id: 171798694281), p :HI_PRI
W0817 11:08:48.526896  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182694 msec: BUFFERED_WRITER_APPEND (id: 4294972200), p :HI_PRI
W0817 11:08:48.526921  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182674 msec: BUFFERED_WRITER_APPEND (id: 25769808645), p :HI_PRI
W0817 11:08:48.526939  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182654 msec: BUFFERED_WRITER_APPEND (id: 171798694282), p :HI_PRI
W0817 11:08:48.527008  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182633 msec: BUFFERED_WRITER_APPEND (id: 4294972201), p :HI_PRI
W0817 11:08:48.527032  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182614 msec: BUFFERED_WRITER_APPEND (id: 25769808646), p :HI_PRI
W0817 11:08:48.527070  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182594 msec: BUFFERED_WRITER_APPEND (id: 171798694283), p :HI_PRI
W0817 11:08:48.527093  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182574 msec: BUFFERED_WRITER_APPEND (id: 4294972202), p :HI_PRI
W0817 11:08:48.527118  108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182554 msec: BUFFERED_WRITER_APPEND (id: 25769808647), p :HI_PRI
C0817 11:08:48.528394  108418 [logdevice:WG18] BufferedWriterSingleLog.cpp:288] readyToSend() Check failed: batch.state == Batch::State::CONSTRUCTING_BLOB

Thread 30 "logdevice:WG18" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff857fa700 (LWP 108418)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff27d9801 in __GI_abort () at abort.c:79
#2  0x00007ffff5b264db in facebook::logdevice::dbg::ld_check_fail_impl(facebook::logdevice::dbg::CheckType, char const*, char const*, char const*, int) () from /usr/local/lib/liblogdevice.so
#3  0x00007ffff5b65e48 in facebook::logdevice::BufferedWriterSingleLog::readyToSend(facebook::logdevice::BufferedWriterSingleLog::Batch&) () from /usr/local/lib/liblogdevice.so
#4  0x00007ffff5b6a491 in facebook::logdevice::BufferedWriterSingleLog::ContinueBlobSendRequest::execute() () from /usr/local/lib/liblogdevice.so
#5  0x00007ffff5ae27e1 in facebook::logdevice::Worker::processRequest(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >) () from /usr/local/lib/liblogdevice.so
#6  0x00007ffff5ae3207 in facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&)::{lambda()#1}::operator()() () from /usr/local/lib/liblogdevice.so
#7  0x00007ffff5ae4e3a in void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&)::{lambda()#1}>(folly::detail::function::Data&) () from /usr/local/lib/liblogdevice.so
#8  0x00007ffff59c40cf in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fff14072150) at /LogDevice/logdevice/external/folly/folly/Function.h:376
#9  0x00007ffff5ae2f0d in facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::{lambda()#1}::operator()() () at /usr/include/c++/7/typeinfo:100
#10 0x00007ffff5ae4dac in void folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::{lambda()#1}>(folly::detail::function::Data&) ()
    at /usr/include/c++/7/typeinfo:100
#11 0x00007ffff59c40cf in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fff857f90a0) at /LogDevice/logdevice/external/folly/folly/Function.h:376
#12 0x00007ffff618f6c0 in facebook::logdevice::EventLoopTaskQueue::executeTasks(unsigned int) () at /usr/include/c++/7/typeinfo:100
#13 0x00007ffff618e5e9 in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::{lambda(unsigned int)#1}::operator()(unsigned int) const () at /usr/include/c++/7/typeinfo:100
#14 0x00007ffff618f972 in void facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::{lambda(unsigned int)#1}&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::{lambda(unsigned int)#1}&, unsigned int) () at /usr/include/c++/7/typeinfo:100
#15 0x00007ffff618e7dd in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short) () at /usr/include/c++/7/typeinfo:100
#16 0x00007ffff618f787 in void facebook::logdevice::EventHandler<&facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler, &facebook::logdevice::(anonymous namespace)::preflight_noop, &facebook::logdevice::(anonymous namespace)::postflight_noop>(int, short, void*) () at /usr/include/c++/7/typeinfo:100
#17 0x00007ffff02998f8 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#18 0x00007ffff029a33f in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#19 0x00007ffff618a0eb in facebook::logdevice::EventLoop::run() () at /usr/include/c++/7/typeinfo:100
#20 0x00007ffff618956a in facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}::operator()() const () at /usr/include/c++/7/typeinfo:100
#21 0x00007ffff618a787 in void std::__invoke_impl<void, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_other, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&) () at /usr/include/c++/7/typeinfo:100
#22 0x00007ffff618a555 in std::__invoke_result<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>::type std::__invoke<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_result&&, (facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)...) () at /usr/include/c++/7/typeinfo:100
#23 0x00007ffff618a9b2 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) () at /usr/include/c++/7/typeinfo:100
#24 0x00007ffff618a983 in std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::operator()() () at /usr/include/c++/7/typeinfo:100
#25 0x00007ffff618a962 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> > >::_M_run() () at /usr/include/c++/7/typeinfo:100
#26 0x00007ffff307e66f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#27 0x00007ffff2b916db in start_thread (arg=0x7fff857fa700) at pthread_create.c:463
#28 0x00007ffff28ba88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) 

Skip tests when building LogDevice

Is there a way I can skip testing when doing a LogDevice build. This is just to tryout examples so I really don't care much about test.. Is this possible?

Integrate with open source monitoring/alerting

LogDevice exports a lot of stats which are currently only available through the Admin commands (stats).

We should consider integrating with Prometheus, Influx or other tooling.
The plugin system is already in place, but has no implementation yet:

https://github.com/facebookincubator/LogDevice/blob/debb8abc5794b909989a153843f2982db4ab7be9/logdevice/common/PluginPack.h#L81

Unified approach for printing a log id

Log IDs are printed to the log in a huge number of different styles, making it hard to search. All of these occur:

• log %lu
• logid %lu
• log_id %lu
• Log %lu
• Logid %lu
• LogID %lu
• log:%lu (no space)
• log_:%lu
• Log:%lu
• log: %lu (with space)
• Log: %lu
• LogID: %lu
• %lue%un%u (log ID with lsn, e.g. "42e420n1337")

Would be nice to codemod all of these to a single style, e.g. "log:%lu". We don't care which particular style wins, only that it be easily searchable.

• Maybe add logid_to_string() similar to lsn_to_string()? That would be slower because of the allocation in std::string
• Maybe have a macro similar to PRId64?
• Add a lint rule for things like "log %lu", "log: %lu" etc?

Probably also unify the style of log+lsn, e.g. to "log:42-e420n1337".

Until then, here's a regex to search for log 42: [Ll]og_?([Ii][Dd])?:? ?42\b|\b42e

Deprecate logdevice/common/configuration/Log.h

The data structures used from logdevice/common/configuration/Log.h are deprecated in favor of the new structures in logdevice/common/configuration/logs/LogsConfigTree.h.

All of the server/common code has been cleaned up already, but quite a few tests still use this.

[ld:admin] NodeID.h:53] index() Check failed: isNodeID() Caught coredump signal 6

After run a cmd select * from info_config on ldshell , all my four nodes crash down

I0911 14:05:33.984902      39 [ld:s0:db-bg-fl] PartitionedRocksDBStore.cpp:6386] flushBackgroundThreadRun() Flushing: []. Total stats: active: 0.001MB in 1 memtables, flushing: 0.000MB, pinned: 0.000MB. Eval time: 0ms, flush init: 0ms, throttle eval: 0ms, npartitions: 8. Existing memtables: metadata A:0.001MB, []
C0911 14:05:38.533110      12 [ld:admin] NodeID.h:53] index() Check failed: isNodeID()
handle_fatal_signal(): Caught coredump signal 6
I0911 14:05:39.467410      81 [ld:srv:WG30] Socket.cpp:1052] close() Closing socket C116(10.35.98.12:14178). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467462      55 [ld:srv:WG4] Socket.cpp:1052] close() Closing socket C90(10.35.98.12:13640). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467620      83 [ld:srv:WF0] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16112). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467662      81 [ld:srv:WG30] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467686      51 [ld:srv:WG0] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467705      77 [ld:srv:WG26] Socket.cpp:1052] close() Closing socket C3(10.35.98.12:64316). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467729      59 [ld:srv:WG8] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467746      83 [ld:srv:WF0] Socket.cpp:1052] close() Closing socket C1(10.35.98.12:22200). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467764      73 [ld:srv:WG22] Socket.cpp:1052] close() Closing socket C6(10.35.98.12:64340). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467803      55 [ld:srv:WG4] Socket.cpp:1052] close() Closing socket C8(10.35.98.12:64350). Reason: PEER_CLOSED: connection closed by peer
    @ 000055fe6aa25d9c facebook::logdevice::handle_fatal_signal(int)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a0fe613 coredump_signal_handler(int)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 00007f8b07b2988f (unknown)
    @ 00007f8b03d15e97 gsignal
    @ 00007f8b03d17800 abort
    @ 000055fe6b419c64 facebook::logdevice::dbg::ld_check_fail_impl(facebook::logdevice::dbg::CheckType, char const*, char const*, char const*, int)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a1683ce facebook::logdevice::NodeID::index() const
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a207b44 facebook::logdevice::commands::InfoConfig::metadata(facebook::logdevice::Configuration const&)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a2078db facebook::logdevice::commands::InfoConfig::run()
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a1f7eff facebook::logdevice::CommandProcessor::processCommand(char const*, folly::SocketAddress const&)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a1f16d4 facebook::logdevice::CommandListener::processCommand(bufferevent*, char const*, bool, facebook::logdevice::CommandListener::ConnectionState*)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6a1f0195 facebook::logdevice::CommandListener::readCallback(bufferevent*, void*)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 00007f8b06d3ae31 (unknown)
    @ 00007f8b06d408f7 (unknown)
    @ 00007f8b06d4133e event_base_loop
    @ 000055fe6b660d4f facebook::logdevice::EvBase::loop()
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b1408a6 facebook::logdevice::EventLoop::run()
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b13fd25 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}::operator()() const
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b140da0 void std::__invoke_impl<void, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_other, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b140a66 std::__invoke_result<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>::type std::__invoke<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_result&&, (facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)...)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b1412b3 decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b141284 std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::operator()()
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 000055fe6b141263 std::thread::_State_impl<std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> > >::_M_run()
                       -> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
    @ 00007f8b0473b66e (unknown)
    @ 00007f8b07b1e6da start_thread
    @ 00007f8b03df888e clone
handle_fatal_signal(): processed segments: 209
handle_fatal_signal(): unmapped pages - unmap calls: 283 - 12

Sample standalone client app based on examples

I think the learning curve with LogDevice is too steep.. I have tried a few time but haven't managed to get off the fly very well so I am thinking if you could break the examples folder into some simple standalone projects that shows how to do a basic read and write but shows all the dependencies, we could use that as project templates to get going quicker...

Also, we need to list somewhere the external dependencies needed for a client. Is everything in the external folder required? or Just fizz and folly?

Question: Why hand write protocol ?

I find there are many hand write protocol files in logdevice/common.
Doesn't fbthrift or any other RPC meet the demand here?

Help with writing basic client writer

my_write.cpp.zip
Hi guys,

I wrote the attached simple client trying to write a dummy message into a log using ldwrite as an example. I get the SUCCESSFUL write logged but the append fails.

Write to server SUCCESSFULL
error: append failed: SHUTDOWN: shutting down

Does any one know where the shutting down message comes from. The ldwrite client works fine even after my test... so the error is wrong.. and my message doesn't get appended to the log

IsLogEmptyTest.SimpleGracePeriod flaking on LogDevice-OSS

Circle CI has reported 2 incidents of IsLogEmptyTest.SimpleGracePeriod failing recently. Need to debug.

Incidents:
https://circleci.com/gh/facebookincubator/LogDevice/852
https://circleci.com/gh/facebookincubator/LogDevice/809

Running 100 cycles of the test:

rm -f passed failed
for i in {1..100}; do \
    test/integration_test --gtest_filter=IsLogEmptyTest.SimpleGracePeriod \
        && echo "${i}" >> passed || echo "${i}" >> failed; \
done

Now run 1000 cycles without reproducing; will probably need to look at the specific failures.

Implement adjustable ratio or upper limit for shadow traffic

Right now shadow traffic is configurable from LogsConfig as a percentage.

Given that shadowed clusters organically grow traffic, this increases the throughput as well in the shadowing clusters.

We ideally want to express an absolute threshold such that the sample rate is calculated dynamically.

Install all Python modules with setup.py; rather than copying directly with CMake

First attempt at installation of Python modules of LogDevice was implemented by determining the dist directory, setting variable PYTHON_MODULE_INSTALL_PREFIX and then installing files into the calculated location. This method is convoluted and goes against the preferred method for packaging Python modules using distutils and setuputils.
For ldshell; we created a setup.py and then install it with a straight forward command line call:

https://github.com/facebookincubator/LogDevice/blob/2c0d9d5d9a38dfc5e60886acabc0a036eb43031e/logdevice/ops/CMakeLists.txt#L10

Switch to this method for install of all Python modules, including:

• LogDevice client binding
• Admin client
• LD Query

To find all examples search code for PYTHON_MODULE_INSTALL_PREFIX

CMake rule to automate update of LogDevice Client API doxygen documentation

Currently the doxygen documentation has been manually created, to keep it up-to-date with in-code changes; need to automated the process of running.

Docker image fails because of the missing libboost_regex.so.1.65.1 shared lib

Building the docker image in docker/Dockerfile.ubuntu completes successfully but whenever you try to run it, it fails with:

$ docker run -it --rm logdevice /usr/local/bin/ld-dev-cluster                                                                                             
/usr/local/bin/ld-dev-cluster: error while loading shared libraries: libboost_regex.so.1.65.1: cannot open shared object file: No such file or directory

And ldshell fails with:

root@950359196c00:/usr/local/lib# /usr/local/bin/ldshell
Traceback (most recent call last):
  File "/usr/local/bin/ldshell", line 11, in <module>
    load_entry_point('ldshell==0.1', 'console_scripts', 'ldshell')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python3.6/dist-packages/ldshell-0.1-py3.6.egg/ldshell/main.py", line 11, in <module>
  File "/usr/local/lib/python3.6/dist-packages/ldshell-0.1-py3.6.egg/ldshell/logdevice_plugin.py", line 11, in <module>
  File "/usr/local/lib/python3.6/dist-packages/ldshell-0.1-py3.6.egg/ldshell/autoload/commands/logsconfig.py", line 12, in <module>
ImportError: libboost_python3-py36.so.1.65.1: cannot open shared object file: No such file or directory

Running apt-get update && apt-get install libboost-regex1.65.1 libboost-python1.65.1 in the image solves the issue.

I checked the image build logs and found:

Removing pinentry-curses (1.1.0-1) ...
Removing libassuan0:amd64 (2.5.1-2) ...
Removing libboost-python1.65.1 (1.65.1+dfsg-0ubuntu5) ...
Removing libboost-regex1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Removing openssh-client (1:7.6p1-4) ...

It seems that the cleanup that happens in the Dockerfile here removes some of the runtime dependencies as well.

ldshell `logs show` is broken in OSS build

Steps to reproduce:

• Start a local cluster using ld-dev-cluster using the ubuntu docker image
• ldhsell into this cluster
• Run logs show
• It crashes with:

root@/dev/shm/tmp/logdevice/IntegrationTestUtils.510e-92fc-9dcd-57c2/logdevice.conf> logs show
Error running command: 'ascii' codec can't encode character '\u25bc' in position 5: ordinal not in range(128)
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/nubia/internal/cmdbase.py", line 363, in run_interactive
    ret = fn(**args_dict)
  File "/usr/local/lib/python3.6/dist-packages/ldshell/autoload/commands/logsconfig.py", line 514, in show
    node, max_depth=max_depth, log_groups=True, show_version=True
  File "/usr/local/lib/python3.6/dist-packages/ldshell/autoload/commands/logsconfig.py", line 334, in _print_directory
    return _print_directory_helper(0, directory, max_depth, log_groups, show_version)
  File "/usr/local/lib/python3.6/dist-packages/ldshell/autoload/commands/logsconfig.py", line 309, in _print_directory_helper
    cprint(space_shift + "\u25BC " + directory.fully_qualified_name, "green")
  File "/usr/local/lib/python3.6/dist-packages/termcolor.py", line 124, in cprint
    print((colored(text, color, on_color, attrs)), **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\u25bc' in position 5: ordinal not in range(128)
------------------------------------------------------------

Chain LogDevice documentation pipeline together in CircleCI Jobs

With PR #44, documentation is reproducible from source, with the following process:

1. Build LogDevice on Ubuntu Bionic machine (additional doxygen package required)
2. Run 'make docs' which will update generated markdown files settings.md, ldquery.md and doxygen generated client API documentation under website/static/api.
3. Check in updated files to source control, create a PR and integrate which triggers...
4. Docusaurus run to update website

To keep things up-to-date, with no human intervention the manual steps need to be automated:

Run make docs as part of the build job under Ubuntu Bionic CircleCI build, archiving the output of the build CircleCI job and making available to the deploy-website job; which using this as input updates the website
At this point generated files in website/static/api to be removed as they are output files rather than source files.

There is a question over settings.md and ldquery.md; as these are legible when browsed in GitHub; it might make sense to leave the generated files under source control?
If we do follow this option, add a test to ensure that any changes that update documentation include the corresponding update to those generated files; else we risk having different documents on the website to those viewed on master.

Add a dynamic loader for plugins

Currently the only way to load plugins is to build them as a static library, export the logdevice_plugin() symbol and link them to the binary that uses the logdevice library. StaticPluginLoader then picks them up:
https://github.com/facebookincubator/LogDevice/blob/e6424a70735b589d0be1ed8a25f7a273a2198df3/logdevice/common/plugin/StaticPluginLoader.cpp#L13-L31

This is inconvenient, as this requires a rebuild of the binary every time something in the plugin changes. In addition to that, a dynamic plugin loader should be developed. It should pick up .so files from a certain folder, and then load all the plugins exported by those.

Presto support querying from LogDevice data?

if it can support!!!
it is really perfect!
i hate using hadooooooooooooooooooooooooooooooop!

Publish a docker image on dockerhub

Building the logdevice's docker image (and logdevice itself) takes too long, it would be great if you integrate the repo with docker hub to automatically build and publish images out of master (or any stable branch).

Footer links are broken

Example: https://logdevice.io/en/help.html
They are broken because of an additional en in the url.

Package LogDevice for Debian/Ubuntu

To simplify installation and deployment of LogDevice to a cluster; create a Debian package (.deb) that can be installed to nodes in a cluster; rather than requiring that either that each node builds LogDevice or that executables and config are manually copied between nodes.

Issues with building LogDevice on Fedora Docker containers

I've found two issues with the Fedora docker containers, where both Fedora containers suffer from the same issue.

1. A crash occurs because gtest_discover_tests is not found. This is because this function was added to the GoogleTest cmake module in 3.10, and the Fedora docker containers use cmake 3.9.1.
   I suggest simply not using gtest_discover_tests when on a lesser version, as the tests seem to be added without this extra functionality anyways when GoogleTest can not be successfully loaded. I'll happily submit a PR for this.

2. The issue with _boost_py_component moved to #35

Typo in api web pages

This is really nothing but there is a typo in the api section of the website.
In the bottom of the site it say :
LogDevic API v2.35

I think that the last "e" is missing!
I tried to build the doc using Doxygen but I was not able to build the same pages so i did not find where the problem is coming from.

Anyway this is below low priority so feel free not fixing it ;)

Collect core-files for post-mortem on CircleCI test failure

We have a selection of tests with very-low flake counts; to allow analysis of these; collect core dumps from the test run on failure.

Build breakage on LogDevice-OSS master

https://circleci.com/gh/facebookincubator/LogDevice/883

option --gtest_list_tests unknown; most likely thing is that we've linked something in to the test target that redefined function main().

ERROR: unknown command line flag 'gtest_list_tests'
CMake Error at /usr/share/cmake-3.10/Modules/GoogleTestAddTests.cmake:39 (message):
  Error running test executable.

    Path: '/root/project/_build/test/common_test'
    Result: 1
    Output:
      



common/CMakeFiles/common_test.dir/build.make:2846: recipe for target 'test/common_test' failed
make[2]: *** [test/common_test] Error 1
make[2]: *** Deleting file 'test/common_test'
CMakeFiles/Makefile2:601: recipe for target 'common/CMakeFiles/common_test.dir/all' failed
make[1]: *** [common/CMakeFiles/common_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Add endian awareness to the protocol

Right now, most of the message serialization uses memcpy directly to construct the over-the-wire payload.

Whenever these structures include integers, they are implicitly encoded in the machine's endianness. This will cause a binary incompatibility on machines that do not provide flexible endianness per process (e.g. MIPS?)

We should abstract this away using helper functions from the htons, htonl, ntohs, ntohl family.
Alternatively, we could use flatbuffers and gate this under protocol negotiations. This would greatly speed up adding message types in the future.

LogDevice client API read slow than expected

I use bufferwriter to write logs to logdevice, one line one record per second, write for 14 hours, and then read it through the ldcat tool, total 5MB data takes 12 seconds.

Is this normal? What I'm missed?

std::unique_ptr<facebook::logdevice::Reader> reader = client->createReader(1);
...
reader->startReading(log, facebook::logdevice::LSN_OLDEST, until_lsn);
...
ssize_t nread = reader->read(100, &data, &gap);
...

CircleCI LogDevice Build Broken (needing yarpl)

At first glance, looks like a recent change to wangle means it now requires yarpl.

Build Failed with Dockerfile.ubuntu

[ 70%] Built target ldtail
/usr/bin/ld: ../fbthrift-prefix/src/fbthrift-build/lib/libthriftprotocol.a(BinaryProtocol.cpp.o): undefined reference to symbol '_ZN6google14FlagRegistererC1IiEEPKcS3_S3_PT_S5_'
//usr/lib/x86_64-linux-gnu/libgflags.so.2.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
lib/CMakeFiles/ldclient_test.dir/build.make:345: recipe for target 'test/logdevice_client_test' failed
make[2]: *** [test/logdevice_client_test] Error 1
CMakeFiles/Makefile2:1301: recipe for target 'lib/CMakeFiles/ldclient_test.dir/all' failed
make[1]: *** [lib/CMakeFiles/ldclient_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 70%] Linking CXX static library ../lib/liblogdevice_admin.a
[ 70%] Linking CXX static library ../../lib/liblocallogstore_test_util.a
[ 70%] Built target logdevice_admin
[ 70%] Built target locallogstore_test_util
[ 70%] Linking CXX shared library ../../lib/client.so
[ 70%] Linking CXX executable ../test/common_test
[ 70%] Built target logdevice_python
[ 70%] Linking CXX static library ../lib/libtest_util.a
[ 70%] Built target test_util
/usr/bin/ld: ../folly-prefix/src/folly-build/libfolly.a(IOThreadPoolExecutor.cpp.o): undefined reference to symbol '_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_'
//usr/lib/x86_64-linux-gnu/libgflags.so.2.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [test/common_test] Error 1
common/CMakeFiles/common_test.dir/build.make:3569: recipe for target 'test/common_test' failed
CMakeFiles/Makefile2:928: recipe for target 'common/CMakeFiles/common_test.dir/all' failed
make[1]: *** [common/CMakeFiles/common_test.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
The command '/bin/sh -c mkdir /build && cd /build &&     cmake /LogDevice/logdevice/ &&     make -j $(nproc) &&     make install &&     cp /build/bin/ld* /usr/local/bin/' returned a non-zero code: 2

Stream Processing - CEP - Siddhi SQL - Feature Request

Hi Team,
Since LogDevice is doing Stream processing can we have a Complex event processing like Siddhi SQL
https://wso2.github.io/siddhi/

Ubuntu Docker Build failed

Below issue is found while building ubuntu docker image:
docker build -t logdevice-ubuntu -f docker/Dockerfile.ubuntu --target=builder .

Compilation terminated while compiling admin/CMakeFiles/logdevice_admin.dir/AdminAPIHandler.cpp.o due to included file common/fb303/cpp/FacebookBase2.h not found.

I instead found as <workdir>/LogDevice/common/fb303/cpp/FacebookBase2.h

Figure out how to build / load / combine plugins

Right now, there is a PluginPack that registers a single binary (.so) for different purposes.
We need to figure out:

• How to separate each part into a separate binary. E.g. I'd like to use Prometheus for logging and ElasticSearch for tracing. It should allow combinations flexibly.
• How to build plugins in open source. Should we have a sub-directory? Separate cmake files for each? Separate repositories?

Build failed if use logdevice as submodole

I use Logdevice in my project as submodule
[submodule "Src/3rd/logdevice"]
path = Src/3rd/logdevice
url = git://github.com/facebookincubator/LogDevice.git

An error occurred while building:
/Documents/ProjectRoot/Src/3rd/logdevice/logdevice/common/configuration/nodes/NodesConfiguration.thrift:9] Could not find include file logdevice/common/membership/Membership.thrift

Add FBThrift to LogDevice-oss builds

A number of additional service integrations are based on FBThrift, to allow these to be added to the open source version; we need FBThift in the LogDevice open source build.

Notes on the investigation so far:

FBThrift requires Wangle; Wangle wants the header files and config of folly to be in the "installed" structure; where the generated headers and source headers are in one place
To achieve this; we need to create the concept of a "ROOT_DIR", as we have in cross compilation environments like buildroot or Yocto. CMake supports this through the configuration of CMAKE_FIND_ROOT_PATH; we need to install folly into this and then pass the location to the CMake config for wangle ... and it builds successfully.

Build failed on target rocksdb-prefix

Below error occurred on building log device (running 'make -j $(nproc)' on Ubuntu 18.04.2 LTS (Bionic Beaver) Virtual Box Image.

[ 66%] Built target logdevice_server
make[2]: *** No rule to make target 'rocksdb-prefix/src/rocksdb-build/librocksdb.a', needed by 'bin/logdeviced'. Stop.
CMakeFiles/Makefile2:158: recipe for target 'CMakeFiles/logdeviced.dir/all' failed
make[1]: *** [CMakeFiles/logdeviced.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2

Question: is it possible to read inconsistent data?

I read the docs/API introduction, overview, concepts and Terminology, and have some questions about the design.

The append request will be sent to multiple storage server.

LogDevice achieves durability by storing several identical copies of each record (typically two or three) on different machines.

What happened if the client panic and the first replica of record flushed but the second lost?(1) There will be an unacknowledged record with only one replica in the cluster. And if another client read the log with mode SCD from LSN_OLDEST now, is it possible that it got the unacknowledged record?(2) or client should read at least 2 same replicas of a record before return the result?(3) then How to distinguish between two records that the first is an unacknowledged dirty record with 1 replica and the second is an acknowledged record with 2 replicas but 1 crashed?(4)

Is LogDevice Production ready?

Is LogDevice Production ready? Why no release version?

How to create/write/read a normal file ( like nginx.log ) to logdevice ?

As a normal user, I want to call the API to create/write/read a log file to logdevice. The input parameter is a normal log file name instead of logID. What should I do?
Do I need to record the mapping between log file names and log IDs?

Or

Make a full path of a log as the name of the logdevice loggroup name like :

/app1_var_log_message
/app1_export_logs_nginx_error.log
/app1_export_logs_nginx_access.log
/app2_var_log_message
/app2_export_logs_nginx_error.log
/app2_export_logs_nginx_access.log

?

Lose records ?

I write a test program, create 100 bufferWriters for 100 logIDs on a Client, and each bufferWriter append 1 records per second. Each record is about 1000 log lines. Write 10,000 seconds

when writing is done (I got no Error on bufferWriter append or callback), I try to read it from logdevice,only tens of thousands or millions line can be read on each logID

some times ldcat and ldtail get :

terminate called without an active exception
Aborted

ERROR logs on logdeviced :

E0829 04:23:30.641971      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:38.642558      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:23:38.642574      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:40.642708      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:48.643318      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:23:48.643335      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:50.643435      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:58.643947      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:23:58.643964      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:00.644095      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:08.644877      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:08.644897      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:10.645066      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:18.645593      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:18.645629      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:20.645748      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:28.646311      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:28.646327      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:30.646427      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:38.647038      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:38.647050      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:40.647157      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:48.647568      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:48.647588      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:50.647682      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:58.648241      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:58.648260      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:00.648512      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:08.649041      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:08.649060      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:10.649161      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:18.649557      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:18.649589      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:20.649701      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:28.650109      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:28.650128      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:30.650232      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:38.650681      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:38.650697      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:40.650872      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:48.651624      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:48.651642      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:50.651819      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:58.652476      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:58.652507      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:26:00.652731      82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.

my disks are only 6% used on 4 server:

/dev/sda5 634G 34G 569G 6% /home/ld
/dev/sda5 634G 38G 565G 7% /home/ld
/dev/sda5 634G 28G 575G 5% /home/ld
/dev/sda5 634G 70G 532G 12% /home/ld

my cluster conf:

{
        "cluster": "test",
        "server_settings":{
                "rocksdb-auto-create-shards": true,
                "zk-create-root-znodes": false
        },
        "nodes": [{
                        "node_id": 0,
                        "host": "10.35.98.12:16111",
                        "gossip_port": 16112,
                        "generation": 1,
                        "roles": ["sequencer", "storage"],
                        "sequencer": true,
                        "storage": "read-write",
                        "num_shards": 1
                },
                {
                        "node_id": 1,
                        "host": "10.35.98.15:16111",
                        "gossip_port": 16112,
                        "generation": 1,
                        "roles": ["sequencer", "storage"],
                        "sequencer": true,
                        "storage": "read-write",
                        "num_shards": 1
                },
                {
                        "node_id": 2,
                        "host": "10.35.98.16:16111",
                        "gossip_port": 16112,
                        "generation": 1,
                        "roles": ["sequencer", "storage"],
                        "sequencer": true,
                        "storage": "read-write",
                        "num_shards": 1
                },
                {
                        "node_id": 3,
                        "host": "10.35.98.17:16111",
                        "gossip_port": 16112,
                        "generation": 1,
                        "roles": ["sequencer", "storage"],
                        "sequencer": true,
                        "storage": "read-write",
                        "num_shards": 1
                }
        ],
        "internal_logs": {
                "config_log_deltas": {
                        "replicate_across": {
                                "node": 2
                        }
                },
                "config_log_snapshots": {
                        "replicate_across": {
                                "node": 2
                        }
                },
                "event_log_deltas": {
                        "replicate_across": {
                                "node": 2
                        }
                },
                "event_log_snapshots": {
                        "replicate_across": {
                                "node": 2
                        }
                }
        },
        "metadata_logs": {
                "nodeset": [0, 1, 2, 3],
                "replicate_across": {
                        "node": 2
                }
        },
        "zookeeper": {
                "quorum": [
                        "10.35.98.12:2181",
                        "10.35.98.15:2181",
                        "10.35.98.16:2181"
                ],
                "timeout": "30s"
        }
}

Stop "building" gh-pages

Looking at the build history in CircleCi; I think we try to "build" gh-pages when and update is triggered through an update to the master branch. It could well be that this is also the cause of that rather annoying "Projects currently running on CircleCI 1.0 are no longer supported. Please migrate to CircleCI 2.0."; which I know to be emitted when a branch does not have a valid .circleci/config.yml file.

There are a few CircleCI forum posts that seem related, for example:
https://discuss.circleci.com/t/excluding-a-branch-from-ci/17811

LogDevice-OSS build failing...

We get the following error while building:

/usr/bin/ld: ../folly-prefix/src/folly-build/folly/libfollybenchmark.a(Benchmark.cpp.o): undefined reference to symbol '_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_'
//usr/lib/x86_64-linux-gnu/libgflags.so.2.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
examples/CMakeFiles/ld-dev-cluster.dir/build.make:344: recipe for target 'bin/ld-dev-cluster' failed
make[2]: *** [bin/ld-dev-cluster] Error 1
CMakeFiles/Makefile2:1916: recipe for target 'examples/CMakeFiles/ld-dev-cluster.dir/all' failed
make[1]: *** [examples/CMakeFiles/ld-dev-cluster.dir/all] Error 2

Looks like this failure might have started occurring around the following commit: 63333fc

Protocol Documentation

The (TCP) data communication could use some documentation to help the development of client libraries for other languages than C++.

build with issue:Unknown CMake command "thrift_generate".

I encounter error when I try to build logdevice:
OpenTracing includes: /home/hadoop/LogDevice/_build/external/opentracing/src/OpenTracing-build/include
CMake Error at admin/if/common/fb303/if/CMakeLists.txt:13 (thrift_generate):
Unknown CMake command "thrift_generate".

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
DOUBLE_CONVERSION_INCLUDE_DIR (ADVANCED)

Any suggestion?

Missing libzstd-dev dependency in ubuntu_deps

Hi,
I'm going through the installation instructions on a fresh Ubuntu Bionic EC2 instance, and I found this error.

After installing these packages:

$ sudo apt-get install -y $(cat LogDevice/logdevice/build_tools/ubuntu.deps)

I got this error:

ubuntu@ip-10-204-122-179:~/LogDevice/_build$ cmake ../logdevice/
...
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ZSTD_INCLUDE_DIR
...

I resolved this by installing the libzstd-dev package too. I believe that should be added to LogDevice/logdevice/build_tools/ubuntu.deps.

Thanks!
Rob

MessageSerializationTest unit test failures

Getting regular fails on Circle CI for:

• MessageSerializationTest.STORE_WithByteOffsetMapInfo
• MessageSerializationTest.SEALED

See Logs from CircleCI

https://circleci.com/gh/facebookincubator/LogDevice/902#tests/containers/0