kraret / terichdb

TerichDB, an open source data store based on terark engine

1.TerichDB

TerichDB is an open source NoSQL data store based on terark storage engine.

TerichDB is:

• Read optimized, faster than ever.
• Data is highly compressed and decompressed is not required before read.
• Use FSA and succinct technologies, totally different from B+ and LSM(Which is widely used in other data stores).
• Native support for regex expression query.
• Benchmark

2.Features

• High Compression Ratio
  • Higher than snappy(2~5 times) and other database products, higher than gzip, sometimes higher than bzip
• Search Directly on Compressed Data without Decompression
  • Compressed data as index, index as compressed data
• Fast Access
  • Optimized for SSD and in-memory workloads
• Flexible Indexing
  • Multiple indexes on one table
  • Unique/non-unique index
  • Composite index(one index on multiple columns)
  • Ordered index (support range query)
• Native regular expression query
• Embeddable & Standalone Database
• Persistent
• Schema Based, with Rich Data Types
• Column Group
• Supported Platforms: Linux, Windows, Mac
• Transparent foundation for developers to build customized products

3.Installation

3.1.Linux & MacOS

sudo wget https://raw.githubusercontent.com/Terark/terichdb/master/online_install.py && (python online_install.py; rm online_install.py)

3.2.Windows

windows installation guide

4.Compile TerichDB

4.1.Dependencies

• libboost_system.so(1.63.0) (boost_1_63_0)
• libboost_filesystem(1.63.0)(Could be found in boost_1_63_0)
• libwiredtiger.so(v2.8.0)
• libtbb.so(tbb44_20160128)
• libgflags
• libsnappy
• libz
• libzstd

4.2.Compiler Support

• Linux : g++-4.7, g++-4.8, g++-4.9, g++-5.3, g++-5.4, g++-6.1, g++-6.2
• Mac OS : g++-4.8, g++-4.9, g++-5.3, g++-6.0, g++-6.1, clang++-7.3
• Windows : vs2015

4.3.dfadb alternative

1. TerichDB is open source but our core data structures and algorithms(dfadb) are not yet.
2. dfadb is supported by a library named terark-fsa-* in our released packages.
3. Developers who want to build their own TerichDB, should place the terark-fsa-* library into their library search path(e.g. /usr/local/lib).
4. User compiled TerichDB can only uses limited APIs, and need to set "ReadonlySegmentClass" : "MockReadonlySegment" in dbmeta.json(TerichDB's schema file).

5. Notes

• UNIX fork() should not be called in the applications using TerichDB, this would cause undefined behaviors.

6.License

TerichDB follows Free Software Foundation's GNU AGPL v3.0

For commercial purposes, please contact us.

terichdb's Issues

Welcome, please leave your messages here

We'll response ASAP.

404 from ur home page

cannot download from http://www.terark.com/zh/download

some questions about terichdb

您好！Terichdb的技术非常值得期待。我这里有几个问题，不知道您是否能够回答一下。

1. 在Terichdb中，key与value是使用不同的压缩技术进行存储的，key是使用nlt_build，value是使用zbs_build？
2. 我看文中提到了与RocksDB的对比，对比的过程中，Terichdb是在full compact之后测试的QPS，是否有在读写混合的场景下，TerakDB的QPS的变化情况？

我最近也在研究一些关于succinct的技术，持续关注与研究中！！！

Clarify persistence guarantees (fault tolerance)

I've walked through the benchmarks and asked me one question - how is this DB fault tolerant? Does it use just mmap with populate? Concurrent writes to mmaped space do not provide any guarantees. And how about long writes (e.g. when the data to be written are huge - hundreds of MB)?

it's hard to install dependencies

i spend all the day to try install them on debian wheezy & jessie, but failed
i saw your offical website said in 0.13.9 have resolved this problem
but where is it?
thx

Any plans to make `dfadb` open source?

Are there any plans to publish sources of dfadb core under some some permissive license (e.g. MIT)? If yes, what is the ETA?

I'm asking, because I'm working on a mid-size fully open-source project, which would leverage the compelling features of terark db, but only in case it's fully open source.

非法指令 (core dumped)

[root@server37 table1]# terark-db-schema-compile.exe dbmeta.json DbTest RowData
非法指令 (core dumped)

export LD_LIBRARY_PATH=/opt/glibc-2.14/lib:$LD_LIBRARY_PATH
[root@server37 table1]# g++ --version
g++ (GCC) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@server37 table1]# gcc --version
gcc (GCC) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

ZipIntStore::zipValues suggestion

terak::db::ZipIntStore::zipValues takes a const void* argument, and later then static cast it to const Int*. So maybe we can just modify the signature a little bit for better readability.
Before:

template<class Int>
void ZipIntStore::zipValues(const void* , size_t)

After:

template<class Int>
void ZipIntStore::zipValues(const Int* , size_t)

build fail

Tera vs terark-db

suprise about another project: https://github.com/baidu/tera

reserved identifier violation

I would like to point out that identifiers like "__penglei_valvec_hpp__" and "__terark_db_table_store_hpp__" do eventually not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

what happened to terark-db/src/terark/db/update_column_impl.hpp ?

seems like it's just a piece of code snippet

Specify license for the published terark db code

I was unable to find any notion of licensing neither in documentation nor in source code nor any license file in the repository. Could you please clearly state under which license is the source code?

lack of file 'terark-base/src/terark/util/truncate_file.hpp'

which is included by 'src/terark/db/db_segment.cpp'.

Building steps (On Ubuntu 14.04, with GCC 4.8.4):

WORK_DIR=/path/to/work/dir
mkdir -p $WORK_DIR
cd $WORK_DIR

# install boost-1.60.0
wget https://github.com/boostorg/boost/archive/boost-1.60.0.tar.gz
tar xfz boost-1.60.0.tar.gz
cd boost-boost-1.60.0
./bootstrap.sh --prefix=`pwd`/output --with-libraries=system,filesystem,date_time
./b2
./b2 install
cd ..

# install tbb-4.4
wget https://www.threadingbuildingblocks.org/sites/default/files/software_releases/source/tbb44_20160128oss_src_0.tgz
tar xfz tbb44_20160128oss_src_0.tgz
cd tbb44_20160128oss
make -j8
cd ..

# install wiredtiger-2.7.0
wget https://github.com/wiredtiger/wiredtiger/releases/download/2.7.0/wiredtiger-2.7.0.tar.bz2
tar xfj wiredtiger-2.7.0.tar.bz2
cd wiredtiger-2.7.0
sh autogen.sh
./configure --prefix=`pwd`/output
make -j8
make install
cd ..

# build terark-db
# firstly, add the following two lines to the head of 'terark-db/Makefile'
#   INCS := -I$WORK_DIR/tbb44_20160128oss/include -I$WORK_DIR/wiredtiger-2.7.0/output/include
#   LIBS := -L$WORK_DIR/tbb44_20160128oss/build/linux_intel64_gcc_cc4.8_libc2.19_kernel3.13.0_release -L$WORK_DIR/wiredtiger-2.7.0/output/lib
#
export BOOST_INC=$WORK_DIR/boost_1_60_0/output/include
make