baidu / ins

iNexus, coordinate large scale services

Nexus - Raft Protocol Based Cluster Coordinator

nexus, used to be known as i name service(ins), is a high available key-value store to hold crucial meta data, which also support mechanism to coordinate large scale services.

nexus is based on Raft protocol to keep consistency between multi-nodes, and provide key-value data access, distributed lock mechanism(lock) and event callback function(watch).

Who Is Using Nexus

nexus was born to provide a Google Chubby style service in Galaxy, a cluster manager system. Now it supports several online services in Baidu, Inc., including Galaxy and Tera.

For those who want to save their important data, who want to keep uniqueness in a cluster, or who want to manage large scale services, please refer to this for some classical usage.

Installation and Usage

Use build.sh to solve dependency problem, or export proper environment variable to tell Makefile to find proper headers and libraries. All necessary dependencies are listed at the beginning of Makefile. Proved on g++ and clang++.

The sandbox directory holds a few scripts for starting a test cluster on localhost and old-fashioned client tools.

• Use start_all.sh to start a 5-nodes cluster on current server, and generate flagfile for other scripts
• Use stop_all.sh to stop all nodes, and clean_all.sh to stop cluster and clean up all data and log
• Use stop_one.sh and start_one.sh to temporarily disable and enable a node to check the cluster status
• Use show_cluster.sh to show the status of cluster
• Use ins_shell.sh for old-fasioned nexus client, which uses script to wrap a binary client
• Use run_sample.sh to start a 100k write times test and a 100k read times test

Cluster and client use GFlags to parse command line to alter inner parameters. For more information, please refer to this document.

In addition to script version of client, we introduced a new C++ version client, providing more convenient functions for testing and online operations. This client also gives an example on how to work with nexus SDK.

When deploying to online servers, please use control.sh for a single node to start/stop/restart or check the status of health.

Integration and API

• For C++ user, use make install_sdk to export sdk to PREFIX dir. Please refer to this document for more information
• For Python user, use make python to produce library. Please refer to this document for more information

Dependencies

To build the whole project, following dependencies must be meet:

• snappy 1.1.1 and above
• boost 1.57.0 and above
• GFlags 2.1.2 and above
• Google Protocal Buffers 2.6.1
• sofa-pbrpc 1.1.3 and above

For unittest, following dependency must be meet:

• Google Test 1.7.0 and above

Raft defined log compaction function needs a slight change in LevelDB, the modified code is located in thirdparty/leveldb.

Contributing

We welcome any kind of contributions. If there are any developing questions or complaints, please feel free to contact us: opensearch.

ins's Issues

Data loss and cluster unavailability -- Issue fsyncs on database files and directories

I am running a three node iNexus cluster. iNexus can lose data since it does fsync the database files before acknowledging the client. Specifically, after acknowledging a client, if a crash happens, then the acknowledged data can be lost as the database files are not fsync'd before the acknowledgment. This results in silent data loss after acknowledging the client.

Also, the cluster can become unavailable if a crash occurs and if the rename of .dbtmp file to CURRENT file is not immediately persisted to disk. If this rename is not immediately persisted, then the node during startup after the crash would simply fail to start. This can render the entire cluster unusable. To fix this, an explicit fsync on the parent directory is required. For more information on this, please see http://research.cs.wisc.edu/wind/Publications/alice-osdi14.pdf and https://www.quora.com/When-should-you-fsync-the-containing-directory-in-addition-to-the-file-itself.

About Statifier's Segmentation Fault

利用Statifier静态编译打包可执行程序需要将机器的randomize_va_space置0，禁止地址随机化，否则可能引发段错误。
但是为什么ins的Statifier打包文件却可以在非0的机器上运行？是否有做什么特殊处理？真心求解

better input ( readline )

https://github.com/fxsjy/ins/blob/master/sandbox/ins_shell.sh#L4

read -p "galaxy ins> " cmd arg1 arg2 不支持backspace等按键

如此这般，赶脚好一点： read -e -p "galaxy ins> " cmd arg1 arg2.
man 1 read

 -e     If  the  standard input is coming from a terminal, readline (see READLINE above) is used
         to obtain the line.

TryLock触发KeepAliveTask?

当前TryLock时不会触发后台启动KeepAliveTask线程，如果客户端直接调用TryLock，那么服务端是没有这个客户的session的，在服务端判断LockIsAvilable时，会尝试从leveldb读取相应的key，若读取不到，则再判断当前session是不是在session列表中，若在，则允许加锁（ins_node_impl.cc 1030行）。
这样的话，若用户直接调用TryLock之前，没有调用过TryLock，则所有的TryLock操作都不会成功。这里是不是需要改变一下？

支持batch write

log

Building error:
/common/logging.h:55:57: error: unable to find string literal operator ‘operator""fmt’ with ‘const char [9]’, ‘long unsigned int’ arguments
#define LOG(level, fmt, args...) ins_common::Log(level, "[%s:%d] "fmt, FILE, LINE, ##args)

ins现在是否支持Windows？能否使用VS编译成.lib进行调用？

protoc java client 如何设置host和port

根据protocAPI接口没有找到connect方法啊

ins

非ubuntu用户是否能提供一个build.sh，不用comake2

member change

是否需要实现member change功能？ 我看Raft论文中对这个描述的不是很详细，或许可以利用leveldb的多版本控制来复制一些log到新的成员上去，这样可以不用停止服务，然后再根据论文中的方案进行成员变更。
昨天暴了一个raft成员变更算法的bug,　https://groups.google.com/forum/#!topic/raft-dev/t4xj6dJTP6E

问个问题哈，为什么get时可能会做一下心跳，而scan不会？

统计并获取当前nexus读写压力，当客户端出问题时，避免给nexus带来过大压力

请问 ins 的性能如何？ 有官方 benchmark 数据吗？

请问 ins 的性能如何？ 有官方 benchmark 数据吗？5个 node ,能支撑 k-v IOPS 什么量级？
请问 ins leader 切换时间多少？ 秒级？
多谢答复。

新new出来的InsNodeImpl指针为空，导致rpc注册出core

在ins_main.cc的51行最好对InsNodeImpl指针进行判断下是否NULL，我刚就出现过由于机器内存过低导致rpc注册出core的问题