facebookarchive / logdevice Goto Github PK
View Code? Open in Web Editor NEWDistributed storage for sequential data
Home Page: https://logdevice.io
License: Other
Distributed storage for sequential data
Home Page: https://logdevice.io
License: Other
As a normal user, I want to call the API to create/write/read a log file to logdevice. The input parameter is a normal log file name instead of logID. What should I do?
Do I need to record the mapping between log file names and log IDs?
Or
Make a full path of a log as the name of the logdevice loggroup name like :
/app1_var_log_message
/app1_export_logs_nginx_error.log
/app1_export_logs_nginx_access.log
/app2_var_log_message
/app2_export_logs_nginx_error.log
/app2_export_logs_nginx_access.log
?
Building the logdevice's docker image (and logdevice itself) takes too long, it would be great if you integrate the repo with docker hub to automatically build and publish images out of master (or any stable branch).
Steps to reproduce:
ld-dev-cluster
using the ubuntu docker imagelogs show
root@/dev/shm/tmp/logdevice/IntegrationTestUtils.510e-92fc-9dcd-57c2/logdevice.conf> logs show
Error running command: 'ascii' codec can't encode character '\u25bc' in position 5: ordinal not in range(128)
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/nubia/internal/cmdbase.py", line 363, in run_interactive
ret = fn(**args_dict)
File "/usr/local/lib/python3.6/dist-packages/ldshell/autoload/commands/logsconfig.py", line 514, in show
node, max_depth=max_depth, log_groups=True, show_version=True
File "/usr/local/lib/python3.6/dist-packages/ldshell/autoload/commands/logsconfig.py", line 334, in _print_directory
return _print_directory_helper(0, directory, max_depth, log_groups, show_version)
File "/usr/local/lib/python3.6/dist-packages/ldshell/autoload/commands/logsconfig.py", line 309, in _print_directory_helper
cprint(space_shift + "\u25BC " + directory.fully_qualified_name, "green")
File "/usr/local/lib/python3.6/dist-packages/termcolor.py", line 124, in cprint
print((colored(text, color, on_color, attrs)), **kwargs)
UnicodeEncodeError: 'ascii' codec can't encode character '\u25bc' in position 5: ordinal not in range(128)
------------------------------------------------------------
You have the following in AllServerReadStreams.h
:
Actually, multi_index_container
does support in-place construction of elements via emplace
, so the use of shared_ptr
s here is unnecessary, and suboptimal.
I encounter error when I try to build logdevice:
OpenTracing includes: /home/hadoop/LogDevice/_build/external/opentracing/src/OpenTracing-build/include
CMake Error at admin/if/common/fb303/if/CMakeLists.txt:13 (thrift_generate):
Unknown CMake command "thrift_generate".
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
DOUBLE_CONVERSION_INCLUDE_DIR (ADVANCED)
Any suggestion?
if it can support!!!
it is really perfect!
i hate using hadooooooooooooooooooooooooooooooop!
Below issue is found while building ubuntu docker image:
docker build -t logdevice-ubuntu -f docker/Dockerfile.ubuntu --target=builder .
Compilation terminated while compiling admin/CMakeFiles/logdevice_admin.dir/AdminAPIHandler.cpp.o
due to included file common/fb303/cpp/FacebookBase2.h
not found.
I instead found as <workdir>/LogDevice/common/fb303/cpp/FacebookBase2.h
Log IDs are printed to the log in a huge number of different styles, making it hard to search. All of these occur:
Would be nice to codemod all of these to a single style, e.g. "log:%lu". We don't care which particular style wins, only that it be easily searchable.
Probably also unify the style of log+lsn, e.g. to "log:42-e420n1337".
Until then, here's a regex to search for log 42: [Ll]og_?([Ii][Dd])?:? ?42\b|\b42e
W0817 11:08:48.520524 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 202767 msec: BUFFERED_WRITER_APPEND (id: 25769808310), p :HI_PRI
W0817 11:08:48.520537 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 202748 msec: BUFFERED_WRITER_APPEND (id: 171798693947), p :HI_PRI
E0817 11:08:48.526716 108418 [logdevice:WG18] BufferedWriterImpl.cpp:308] execute() skipped at least 1000 log entries
E0817 11:08:48.526749 108418 [logdevice:WG18] BufferedWriterImpl.cpp:308] execute() BufferedWriterShard with id 86 not found. Unexpected!
W0817 11:08:48.526799 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() skipped at least 1000 log entries
W0817 11:08:48.526815 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182733 msec: BUFFERED_WRITER_APPEND (id: 25769808644), p :HI_PRI
W0817 11:08:48.526862 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182714 msec: BUFFERED_WRITER_APPEND (id: 171798694281), p :HI_PRI
W0817 11:08:48.526896 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182694 msec: BUFFERED_WRITER_APPEND (id: 4294972200), p :HI_PRI
W0817 11:08:48.526921 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182674 msec: BUFFERED_WRITER_APPEND (id: 25769808645), p :HI_PRI
W0817 11:08:48.526939 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182654 msec: BUFFERED_WRITER_APPEND (id: 171798694282), p :HI_PRI
W0817 11:08:48.527008 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182633 msec: BUFFERED_WRITER_APPEND (id: 4294972201), p :HI_PRI
W0817 11:08:48.527032 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182614 msec: BUFFERED_WRITER_APPEND (id: 25769808646), p :HI_PRI
W0817 11:08:48.527070 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182594 msec: BUFFERED_WRITER_APPEND (id: 171798694283), p :HI_PRI
W0817 11:08:48.527093 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182574 msec: BUFFERED_WRITER_APPEND (id: 4294972202), p :HI_PRI
W0817 11:08:48.527118 108418 [logdevice:WG18] Worker.cpp:1270] processRequest() Request queued for 182554 msec: BUFFERED_WRITER_APPEND (id: 25769808647), p :HI_PRI
C0817 11:08:48.528394 108418 [logdevice:WG18] BufferedWriterSingleLog.cpp:288] readyToSend() Check failed: batch.state == Batch::State::CONSTRUCTING_BLOB
Thread 30 "logdevice:WG18" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff857fa700 (LWP 108418)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007ffff27d9801 in __GI_abort () at abort.c:79
#2 0x00007ffff5b264db in facebook::logdevice::dbg::ld_check_fail_impl(facebook::logdevice::dbg::CheckType, char const*, char const*, char const*, int) () from /usr/local/lib/liblogdevice.so
#3 0x00007ffff5b65e48 in facebook::logdevice::BufferedWriterSingleLog::readyToSend(facebook::logdevice::BufferedWriterSingleLog::Batch&) () from /usr/local/lib/liblogdevice.so
#4 0x00007ffff5b6a491 in facebook::logdevice::BufferedWriterSingleLog::ContinueBlobSendRequest::execute() () from /usr/local/lib/liblogdevice.so
#5 0x00007ffff5ae27e1 in facebook::logdevice::Worker::processRequest(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >) () from /usr/local/lib/liblogdevice.so
#6 0x00007ffff5ae3207 in facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&)::{lambda()#1}::operator()() () from /usr/local/lib/liblogdevice.so
#7 0x00007ffff5ae4e3a in void folly::detail::function::FunctionTraits<void ()>::callSmall<facebook::logdevice::Worker::forcePost(std::unique_ptr<facebook::logdevice::Request, std::default_delete<facebook::logdevice::Request> >&)::{lambda()#1}>(folly::detail::function::Data&) () from /usr/local/lib/liblogdevice.so
#8 0x00007ffff59c40cf in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fff14072150) at /LogDevice/logdevice/external/folly/folly/Function.h:376
#9 0x00007ffff5ae2f0d in facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::{lambda()#1}::operator()() () at /usr/include/c++/7/typeinfo:100
#10 0x00007ffff5ae4dac in void folly::detail::function::FunctionTraits<void ()>::callBig<facebook::logdevice::Worker::addWithPriority(folly::Function<void ()>, signed char)::{lambda()#1}>(folly::detail::function::Data&) ()
at /usr/include/c++/7/typeinfo:100
#11 0x00007ffff59c40cf in folly::detail::function::FunctionTraits<void ()>::operator()() (this=0x7fff857f90a0) at /LogDevice/logdevice/external/folly/folly/Function.h:376
#12 0x00007ffff618f6c0 in facebook::logdevice::EventLoopTaskQueue::executeTasks(unsigned int) () at /usr/include/c++/7/typeinfo:100
#13 0x00007ffff618e5e9 in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::{lambda(unsigned int)#1}::operator()(unsigned int) const () at /usr/include/c++/7/typeinfo:100
#14 0x00007ffff618f972 in void facebook::logdevice::LifoEventSemImpl<std::atomic>::AsyncWaiter::processBatch<facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::{lambda(unsigned int)#1}&>(facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short)::{lambda(unsigned int)#1}&, unsigned int) () at /usr/include/c++/7/typeinfo:100
#15 0x00007ffff618e7dd in facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler(void*, short) () at /usr/include/c++/7/typeinfo:100
#16 0x00007ffff618f787 in void facebook::logdevice::EventHandler<&facebook::logdevice::EventLoopTaskQueue::haveTasksEventHandler, &facebook::logdevice::(anonymous namespace)::preflight_noop, &facebook::logdevice::(anonymous namespace)::postflight_noop>(int, short, void*) () at /usr/include/c++/7/typeinfo:100
#17 0x00007ffff02998f8 in ?? () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#18 0x00007ffff029a33f in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.1.so.6
#19 0x00007ffff618a0eb in facebook::logdevice::EventLoop::run() () at /usr/include/c++/7/typeinfo:100
#20 0x00007ffff618956a in facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}::operator()() const () at /usr/include/c++/7/typeinfo:100
#21 0x00007ffff618a787 in void std::__invoke_impl<void, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_other, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&) () at /usr/include/c++/7/typeinfo:100
#22 0x00007ffff618a555 in std::__invoke_result<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>::type std::__invoke<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_result&&, (facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)...) () at /usr/include/c++/7/typeinfo:100
#23 0x00007ffff618a9b2 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) () at /usr/include/c++/7/typeinfo:100
#24 0x00007ffff618a983 in std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::operator()() () at /usr/include/c++/7/typeinfo:100
#25 0x00007ffff618a962 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> > >::_M_run() () at /usr/include/c++/7/typeinfo:100
#26 0x00007ffff307e66f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#27 0x00007ffff2b916db in start_thread (arg=0x7fff857fa700) at pthread_create.c:463
#28 0x00007ffff28ba88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
We have a selection of tests with very-low flake counts; to allow analysis of these; collect core dumps from the test run on failure.
Hi,
I'm going through the installation instructions on a fresh Ubuntu Bionic EC2 instance, and I found this error.
After installing these packages:
$ sudo apt-get install -y $(cat LogDevice/logdevice/build_tools/ubuntu.deps)
I got this error:
ubuntu@ip-10-204-122-179:~/LogDevice/_build$ cmake ../logdevice/
...
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
ZSTD_INCLUDE_DIR
...
I resolved this by installing the libzstd-dev
package too. I believe that should be added to LogDevice/logdevice/build_tools/ubuntu.deps
.
Thanks!
Rob
To simplify installation and deployment of LogDevice to a cluster; create a Debian package (.deb) that can be installed to nodes in a cluster; rather than requiring that either that each node builds LogDevice or that executables and config are manually copied between nodes.
I read the docs/API introduction, overview, concepts and Terminology, and have some questions about the design.
The append request will be sent to multiple storage server.
LogDevice achieves durability by storing several identical copies of each record (typically two or three) on different machines.
What happened if the client panic and the first replica of record flushed but the second lost?(1) There will be an unacknowledged record with only one replica in the cluster. And if another client read the log with mode SCD from LSN_OLDEST now, is it possible that it got the unacknowledged record?(2) or client should read at least 2 same replicas of a record before return the result?(3) then How to distinguish between two records that the first is an unacknowledged dirty record with 1 replica and the second is an acknowledged record with 2 replicas but 1 crashed?(4)
Example: https://logdevice.io/en/help.html
They are broken because of an additional en
in the url.
[ 70%] Built target ldtail
/usr/bin/ld: ../fbthrift-prefix/src/fbthrift-build/lib/libthriftprotocol.a(BinaryProtocol.cpp.o): undefined reference to symbol '_ZN6google14FlagRegistererC1IiEEPKcS3_S3_PT_S5_'
//usr/lib/x86_64-linux-gnu/libgflags.so.2.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
lib/CMakeFiles/ldclient_test.dir/build.make:345: recipe for target 'test/logdevice_client_test' failed
make[2]: *** [test/logdevice_client_test] Error 1
CMakeFiles/Makefile2:1301: recipe for target 'lib/CMakeFiles/ldclient_test.dir/all' failed
make[1]: *** [lib/CMakeFiles/ldclient_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 70%] Linking CXX static library ../lib/liblogdevice_admin.a
[ 70%] Linking CXX static library ../../lib/liblocallogstore_test_util.a
[ 70%] Built target logdevice_admin
[ 70%] Built target locallogstore_test_util
[ 70%] Linking CXX shared library ../../lib/client.so
[ 70%] Linking CXX executable ../test/common_test
[ 70%] Built target logdevice_python
[ 70%] Linking CXX static library ../lib/libtest_util.a
[ 70%] Built target test_util
/usr/bin/ld: ../folly-prefix/src/folly-build/libfolly.a(IOThreadPoolExecutor.cpp.o): undefined reference to symbol '_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_'
//usr/lib/x86_64-linux-gnu/libgflags.so.2.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [test/common_test] Error 1
common/CMakeFiles/common_test.dir/build.make:3569: recipe for target 'test/common_test' failed
CMakeFiles/Makefile2:928: recipe for target 'common/CMakeFiles/common_test.dir/all' failed
make[1]: *** [common/CMakeFiles/common_test.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
The command '/bin/sh -c mkdir /build && cd /build && cmake /LogDevice/logdevice/ && make -j $(nproc) && make install && cp /build/bin/ld* /usr/local/bin/' returned a non-zero code: 2
I've found two issues with the Fedora docker containers, where both Fedora containers suffer from the same issue.
A crash occurs because gtest_discover_tests
is not found. This is because this function was added to the GoogleTest
cmake module in 3.10
, and the Fedora docker containers use cmake 3.9.1
.
I suggest simply not using gtest_discover_tests
when on a lesser version, as the tests seem to be added without this extra functionality anyways when GoogleTest
can not be successfully loaded. I'll happily submit a PR for this.
The issue with _boost_py_component
moved to #35
I use bufferwriter to write logs to logdevice, one line one record per second, write for 14 hours, and then read it through the ldcat tool, total 5MB data takes 12 seconds.
Is this normal? What I'm missed?
std::unique_ptr<facebook::logdevice::Reader> reader = client->createReader(1);
...
reader->startReading(log, facebook::logdevice::LSN_OLDEST, until_lsn);
...
ssize_t nread = reader->read(100, &data, &gap);
...
Is there a way I can skip testing when doing a LogDevice build. This is just to tryout examples so I really don't care much about test.. Is this possible?
Right now, most of the message serialization uses memcpy
directly to construct the over-the-wire payload.
Whenever these structures include integers, they are implicitly encoded in the machine's endianness. This will cause a binary incompatibility on machines that do not provide flexible endianness per process (e.g. MIPS?)
We should abstract this away using helper functions from the htons, htonl, ntohs, ntohl
family.
Alternatively, we could use flatbuffers and gate this under protocol negotiations. This would greatly speed up adding message types in the future.
https://circleci.com/gh/facebookincubator/LogDevice/883
option --gtest_list_tests unknown; most likely thing is that we've linked something in to the test target that redefined function main()
.
ERROR: unknown command line flag 'gtest_list_tests'
CMake Error at /usr/share/cmake-3.10/Modules/GoogleTestAddTests.cmake:39 (message):
Error running test executable.
Path: '/root/project/_build/test/common_test'
Result: 1
Output:
common/CMakeFiles/common_test.dir/build.make:2846: recipe for target 'test/common_test' failed
make[2]: *** [test/common_test] Error 1
make[2]: *** Deleting file 'test/common_test'
CMakeFiles/Makefile2:601: recipe for target 'common/CMakeFiles/common_test.dir/all' failed
make[1]: *** [common/CMakeFiles/common_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
LogDevice exports a lot of stats which are currently only available through the Admin commands (stats
).
We should consider integrating with Prometheus, Influx or other tooling.
The plugin system is already in place, but has no implementation yet:
This issue breaks out the second point from #31 into a separate issue.
The below text is copied from the original issue
The _boost_py_component
is not correctly set when building on Fedora, causing boost-python to not be found. I was able to make it work by setting _boost_py_component
to python3
, but that's obviously not an acceptable solution for all distros. Not sure how to solve this one, I remember hearing something about the struggles with this versioning last summer. There seems to be a solution for boost python libs merged into CMake 3.11, but I didn't look into it too deeply.
Error message when not finding the correct boost python version.
CMake Error at /usr/share/cmake/Modules/FindBoost.cmake:1879 (message):
Unable to find the requested Boost libraries.
Boost version: 1.60.0
Boost include path: /usr/include
Could not find the following Boost libraries:
boost_python-py35
Some (but not all) of the required Boost libraries were found. You may
need to install these additional Boost libraries. Alternatively, set
BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
to the location of Boost.
Call Stack (most recent call first):
CMake/logdevice-deps.cmake:16 (find_package)
CMakeLists.txt:34 (include)
Looking at the build history in CircleCi; I think we try to "build" gh-pages when and update is triggered through an update to the master branch. It could well be that this is also the cause of that rather annoying "Projects currently running on CircleCI 1.0 are no longer supported. Please migrate to CircleCI 2.0."; which I know to be emitted when a branch does not have a valid .circleci/config.yml file.
There are a few CircleCI forum posts that seem related, for example:
https://discuss.circleci.com/t/excluding-a-branch-from-ci/17811
A number of additional service integrations are based on FBThrift, to allow these to be added to the open source version; we need FBThift in the LogDevice open source build.
Notes on the investigation so far:
FBThrift requires Wangle; Wangle wants the header files and config of folly to be in the "installed" structure; where the generated headers and source headers are in one place
To achieve this; we need to create the concept of a "ROOT_DIR", as we have in cross compilation environments like buildroot or Yocto. CMake supports this through the configuration of CMAKE_FIND_ROOT_PATH; we need to install folly into this and then pass the location to the CMake config for wangle ... and it builds successfully.
Circle CI has reported 2 incidents of IsLogEmptyTest.SimpleGracePeriod failing recently. Need to debug.
Incidents:
https://circleci.com/gh/facebookincubator/LogDevice/852
https://circleci.com/gh/facebookincubator/LogDevice/809
Running 100 cycles of the test:
rm -f passed failed
for i in {1..100}; do \
test/integration_test --gtest_filter=IsLogEmptyTest.SimpleGracePeriod \
&& echo "${i}" >> passed || echo "${i}" >> failed; \
done
Now run 1000 cycles without reproducing; will probably need to look at the specific failures.
Currently if multiple plugins are loaded of the same type (aside from built-in plugins and plugins that allow multiple active instances of the same types - e.g. ConfigSource
s), PluginRegistry
will fail, as it doesn't have a way to decide which one of those to use.
Instead it should be configurable (via settings or via the configuration file) which plugin to use for each plugin type. Then PluginRegistry::setCurrentSinglePlugins()
should use that information to pick the active plugin:
https://github.com/facebookincubator/LogDevice/blob/e6424a70735b589d0be1ed8a25f7a273a2198df3/logdevice/common/plugin/PluginRegistry.cpp#L85-L88
It should also subscribe to config/setting changes so that you can reconfigure which plugin to use on the fly.
Currently the only way to load plugins is to build them as a static library, export the logdevice_plugin()
symbol and link them to the binary that uses the logdevice library. StaticPluginLoader
then picks them up:
https://github.com/facebookincubator/LogDevice/blob/e6424a70735b589d0be1ed8a25f7a273a2198df3/logdevice/common/plugin/StaticPluginLoader.cpp#L13-L31
This is inconvenient, as this requires a rebuild of the binary every time something in the plugin changes. In addition to that, a dynamic plugin loader should be developed. It should pick up .so
files from a certain folder, and then load all the plugins exported by those.
Right now, there is a PluginPack
that registers a single binary (.so
) for different purposes.
We need to figure out:
my_write.cpp.zip
Hi guys,
I wrote the attached simple client trying to write a dummy message into a log using ldwrite as an example. I get the SUCCESSFUL write logged but the append fails.
Write to server SUCCESSFULL
error: append failed: SHUTDOWN: shutting down
Does any one know where the shutting down message comes from. The ldwrite client works fine even after my test... so the error is wrong.. and my message doesn't get appended to the log
Below error occurred on building log device (running 'make -j $(nproc)' on Ubuntu 18.04.2 LTS (Bionic Beaver) Virtual Box Image.
[ 66%] Built target logdevice_server
make[2]: *** No rule to make target 'rocksdb-prefix/src/rocksdb-build/librocksdb.a', needed by 'bin/logdeviced'. Stop.
CMakeFiles/Makefile2:158: recipe for target 'CMakeFiles/logdeviced.dir/all' failed
make[1]: *** [CMakeFiles/logdeviced.dir/all] Error 2
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
The data structures used from logdevice/common/configuration/Log.h
are deprecated in favor of the new structures in logdevice/common/configuration/logs/LogsConfigTree.h
.
All of the server/common code has been cleaned up already, but quite a few tests still use this.
We get the following error while building:
/usr/bin/ld: ../folly-prefix/src/folly-build/folly/libfollybenchmark.a(Benchmark.cpp.o): undefined reference to symbol '_ZN6google14FlagRegistererC1IbEEPKcS3_S3_PT_S5_'
//usr/lib/x86_64-linux-gnu/libgflags.so.2.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
examples/CMakeFiles/ld-dev-cluster.dir/build.make:344: recipe for target 'bin/ld-dev-cluster' failed
make[2]: *** [bin/ld-dev-cluster] Error 1
CMakeFiles/Makefile2:1916: recipe for target 'examples/CMakeFiles/ld-dev-cluster.dir/all' failed
make[1]: *** [examples/CMakeFiles/ld-dev-cluster.dir/all] Error 2
Looks like this failure might have started occurring around the following commit: 63333fc
Hi Team,
Since LogDevice is doing Stream processing can we have a Complex event processing like Siddhi SQL
https://wso2.github.io/siddhi/
At first glance, looks like a recent change to wangle means it now requires yarpl.
Currently the doxygen documentation has been manually created, to keep it up-to-date with in-code changes; need to automated the process of running.
Getting regular fails on Circle CI for:
See Logs from CircleCI
https://circleci.com/gh/facebookincubator/LogDevice/902#tests/containers/0
I use Logdevice in my project as submodule
[submodule "Src/3rd/logdevice"]
path = Src/3rd/logdevice
url = git://github.com/facebookincubator/LogDevice.git
An error occurred while building:
/Documents/ProjectRoot/Src/3rd/logdevice/logdevice/common/configuration/nodes/NodesConfiguration.thrift:9] Could not find include file logdevice/common/membership/Membership.thrift
Right now shadow traffic is configurable from LogsConfig as a percentage.
Given that shadowed clusters organically grow traffic, this increases the throughput as well in the shadowing clusters.
We ideally want to express an absolute threshold such that the sample rate is calculated dynamically.
Unit tests need to run regardless of the outcome of the "docs" test
Building the docker image in docker/Dockerfile.ubuntu
completes successfully but whenever you try to run it, it fails with:
$ docker run -it --rm logdevice /usr/local/bin/ld-dev-cluster
/usr/local/bin/ld-dev-cluster: error while loading shared libraries: libboost_regex.so.1.65.1: cannot open shared object file: No such file or directory
And ldshell
fails with:
root@950359196c00:/usr/local/lib# /usr/local/bin/ldshell
Traceback (most recent call last):
File "/usr/local/bin/ldshell", line 11, in <module>
load_entry_point('ldshell==0.1', 'console_scripts', 'ldshell')()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 480, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2693, in load_entry_point
return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2324, in load
return self.resolve()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2330, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python3.6/dist-packages/ldshell-0.1-py3.6.egg/ldshell/main.py", line 11, in <module>
File "/usr/local/lib/python3.6/dist-packages/ldshell-0.1-py3.6.egg/ldshell/logdevice_plugin.py", line 11, in <module>
File "/usr/local/lib/python3.6/dist-packages/ldshell-0.1-py3.6.egg/ldshell/autoload/commands/logsconfig.py", line 12, in <module>
ImportError: libboost_python3-py36.so.1.65.1: cannot open shared object file: No such file or directory
Running apt-get update && apt-get install libboost-regex1.65.1 libboost-python1.65.1
in the image solves the issue.
I checked the image build logs and found:
Removing pinentry-curses (1.1.0-1) ...
Removing libassuan0:amd64 (2.5.1-2) ...
Removing libboost-python1.65.1 (1.65.1+dfsg-0ubuntu5) ...
Removing libboost-regex1.65.1:amd64 (1.65.1+dfsg-0ubuntu5) ...
Removing openssh-client (1:7.6p1-4) ...
It seems that the cleanup that happens in the Dockerfile here removes some of the runtime dependencies as well.
I think the learning curve with LogDevice is too steep.. I have tried a few time but haven't managed to get off the fly very well so I am thinking if you could break the examples folder into some simple standalone projects that shows how to do a basic read and write but shows all the dependencies, we could use that as project templates to get going quicker...
Also, we need to list somewhere the external dependencies needed for a client. Is everything in the external folder required? or Just fizz and folly?
I find there are many hand write protocol files in logdevice/common.
Doesn't fbthrift or any other RPC meet the demand here?
Is LogDevice Production ready? Why no release version?
First attempt at installation of Python modules of LogDevice was implemented by determining the dist directory, setting variable PYTHON_MODULE_INSTALL_PREFIX and then installing files into the calculated location. This method is convoluted and goes against the preferred method for packaging Python modules using distutils and setuputils.
For ldshell; we created a setup.py and then install it with a straight forward command line call:
Switch to this method for install of all Python modules, including:
To find all examples search code for PYTHON_MODULE_INSTALL_PREFIX
After run a cmd select * from info_config
on ldshell , all my four nodes crash down
I0911 14:05:33.984902 39 [ld:s0:db-bg-fl] PartitionedRocksDBStore.cpp:6386] flushBackgroundThreadRun() Flushing: []. Total stats: active: 0.001MB in 1 memtables, flushing: 0.000MB, pinned: 0.000MB. Eval time: 0ms, flush init: 0ms, throttle eval: 0ms, npartitions: 8. Existing memtables: metadata A:0.001MB, []
C0911 14:05:38.533110 12 [ld:admin] NodeID.h:53] index() Check failed: isNodeID()
handle_fatal_signal(): Caught coredump signal 6
I0911 14:05:39.467410 81 [ld:srv:WG30] Socket.cpp:1052] close() Closing socket C116(10.35.98.12:14178). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467462 55 [ld:srv:WG4] Socket.cpp:1052] close() Closing socket C90(10.35.98.12:13640). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467620 83 [ld:srv:WF0] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16112). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467662 81 [ld:srv:WG30] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467686 51 [ld:srv:WG0] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467705 77 [ld:srv:WG26] Socket.cpp:1052] close() Closing socket C3(10.35.98.12:64316). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467729 59 [ld:srv:WG8] Socket.cpp:1052] close() Closing socket N0:1(10.35.98.12:16111). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467746 83 [ld:srv:WF0] Socket.cpp:1052] close() Closing socket C1(10.35.98.12:22200). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467764 73 [ld:srv:WG22] Socket.cpp:1052] close() Closing socket C6(10.35.98.12:64340). Reason: PEER_CLOSED: connection closed by peer
I0911 14:05:39.467803 55 [ld:srv:WG4] Socket.cpp:1052] close() Closing socket C8(10.35.98.12:64350). Reason: PEER_CLOSED: connection closed by peer
@ 000055fe6aa25d9c facebook::logdevice::handle_fatal_signal(int)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a0fe613 coredump_signal_handler(int)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 00007f8b07b2988f (unknown)
@ 00007f8b03d15e97 gsignal
@ 00007f8b03d17800 abort
@ 000055fe6b419c64 facebook::logdevice::dbg::ld_check_fail_impl(facebook::logdevice::dbg::CheckType, char const*, char const*, char const*, int)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1683ce facebook::logdevice::NodeID::index() const
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a207b44 facebook::logdevice::commands::InfoConfig::metadata(facebook::logdevice::Configuration const&)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a2078db facebook::logdevice::commands::InfoConfig::run()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1f7eff facebook::logdevice::CommandProcessor::processCommand(char const*, folly::SocketAddress const&)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1f16d4 facebook::logdevice::CommandListener::processCommand(bufferevent*, char const*, bool, facebook::logdevice::CommandListener::ConnectionState*)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6a1f0195 facebook::logdevice::CommandListener::readCallback(bufferevent*, void*)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 00007f8b06d3ae31 (unknown)
@ 00007f8b06d408f7 (unknown)
@ 00007f8b06d4133e event_base_loop
@ 000055fe6b660d4f facebook::logdevice::EvBase::loop()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b1408a6 facebook::logdevice::EventLoop::run()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b13fd25 facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}::operator()() const
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b140da0 void std::__invoke_impl<void, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_other, facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b140a66 std::__invoke_result<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>::type std::__invoke<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}>(std::__invoke_result&&, (facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}&&)...)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b1412b3 decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b141284 std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> >::operator()()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 000055fe6b141263 std::thread::_State_impl<std::thread::_Invoker<std::tuple<facebook::logdevice::EventLoop::EventLoop(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, facebook::logdevice::ThreadID::Type, unsigned long, bool, std::array<unsigned int, 3ul> const&)::{lambda()#1}> > >::_M_run()
-> /root/project/logdevice/external/folly/folly/io/async/AsyncPipe.cpp
@ 00007f8b0473b66e (unknown)
@ 00007f8b07b1e6da start_thread
@ 00007f8b03df888e clone
handle_fatal_signal(): processed segments: 209
handle_fatal_signal(): unmapped pages - unmap calls: 283 - 12
This is really nothing but there is a typo in the api section of the website.
In the bottom of the site it say :
LogDevic API v2.35
I think that the last "e" is missing!
I tried to build the doc using Doxygen but I was not able to build the same pages so i did not find where the problem is coming from.
Anyway this is below low priority so feel free not fixing it ;)
Right now, LogDevice needs to be restarted whenever a shard gets disabled, in order for the filesystem to unmount. This restart is quite a heavy operation (sequencers move around, recoveries, ...) that can be simplified by stopping and starting the RocksDB instances while running.
This could perhaps be implemented using an Admin command.
I write a test program, create 100 bufferWriters for 100 logIDs on a Client, and each bufferWriter append 1 records per second. Each record is about 1000 log lines. Write 10,000 seconds
when writing is done (I got no Error on bufferWriter append or callback), I try to read it from logdevice,only tens of thousands or millions line can be read on each logID
some times ldcat and ldtail get :
terminate called without an active exception
Aborted
ERROR logs on logdeviced :
E0829 04:23:30.641971 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:38.642558 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:23:38.642574 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:40.642708 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:48.643318 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:23:48.643335 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:50.643435 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:23:58.643947 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:23:58.643964 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:00.644095 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:08.644877 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:08.644897 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:10.645066 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:18.645593 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:18.645629 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:20.645748 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:28.646311 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:28.646327 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:30.646427 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:38.647038 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:38.647050 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:40.647157 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:48.647568 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:48.647588 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:50.647682 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:24:58.648241 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:24:58.648260 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:00.648512 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:08.649041 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:08.649060 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:10.649161 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:18.649557 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:18.649589 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:20.649701 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:28.650109 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:28.650128 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:30.650232 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:38.650681 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:38.650697 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:40.650872 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:48.651624 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:48.651642 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:50.651819 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:25:58.652476 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() skipped at least 3 log entries
E0829 04:25:58.652507 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
E0829 04:26:00.652731 82 [ld:s0:db-bg-hi] PartitionedRocksDBStore.cpp:2564] shouldCreatePartition() Number of partitions (2047) is above the soft limit (2000). Adjusting partition creation triggers to limit the growth.
my disks are only 6% used on 4 server:
/dev/sda5 634G 34G 569G 6% /home/ld
/dev/sda5 634G 38G 565G 7% /home/ld
/dev/sda5 634G 28G 575G 5% /home/ld
/dev/sda5 634G 70G 532G 12% /home/ld
my cluster conf:
{
"cluster": "test",
"server_settings":{
"rocksdb-auto-create-shards": true,
"zk-create-root-znodes": false
},
"nodes": [{
"node_id": 0,
"host": "10.35.98.12:16111",
"gossip_port": 16112,
"generation": 1,
"roles": ["sequencer", "storage"],
"sequencer": true,
"storage": "read-write",
"num_shards": 1
},
{
"node_id": 1,
"host": "10.35.98.15:16111",
"gossip_port": 16112,
"generation": 1,
"roles": ["sequencer", "storage"],
"sequencer": true,
"storage": "read-write",
"num_shards": 1
},
{
"node_id": 2,
"host": "10.35.98.16:16111",
"gossip_port": 16112,
"generation": 1,
"roles": ["sequencer", "storage"],
"sequencer": true,
"storage": "read-write",
"num_shards": 1
},
{
"node_id": 3,
"host": "10.35.98.17:16111",
"gossip_port": 16112,
"generation": 1,
"roles": ["sequencer", "storage"],
"sequencer": true,
"storage": "read-write",
"num_shards": 1
}
],
"internal_logs": {
"config_log_deltas": {
"replicate_across": {
"node": 2
}
},
"config_log_snapshots": {
"replicate_across": {
"node": 2
}
},
"event_log_deltas": {
"replicate_across": {
"node": 2
}
},
"event_log_snapshots": {
"replicate_across": {
"node": 2
}
}
},
"metadata_logs": {
"nodeset": [0, 1, 2, 3],
"replicate_across": {
"node": 2
}
},
"zookeeper": {
"quorum": [
"10.35.98.12:2181",
"10.35.98.15:2181",
"10.35.98.16:2181"
],
"timeout": "30s"
}
}
With PR #44, documentation is reproducible from source, with the following process:
settings.md
, ldquery.md
and doxygen generated client API documentation under website/static/api
.To keep things up-to-date, with no human intervention the manual steps need to be automated:
Run make docs as part of the build job under Ubuntu Bionic CircleCI build, archiving the output of the build
CircleCI job and making available to the deploy-website job; which using this as input updates the website
At this point generated files in website/static/api
to be removed as they are output files rather than source files.
There is a question over settings.md and ldquery.md; as these are legible when browsed in GitHub; it might make sense to leave the generated files under source control?
If we do follow this option, add a test to ensure that any changes that update documentation include the corresponding update to those generated files; else we risk having different documents on the website to those viewed on master.
The (TCP) data communication could use some documentation to help the development of client libraries for other languages than C++.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.