sjtu-ipads / wukong

A graph-based distributed in-memory store that leverages efficient graph exploration to provide highly concurrent and low-latency queries over big linked data

Wukong for Linked Data

Wukong is a fast and distributed graph-structured store that leverages efficient graph exploration to proivde highly concurrent and low-latency query processing over big linked data.

Feature Highlights

• High-performance and scalable in-memory graph store
• Fast and concurrent SPARQL query processing by graph exloration
• Fast communication by leveraging RDMA feature of InfiniBand network
• A GPU extension of query engine for heterogenous (CPU/GPU) cluster
• A type-centric SPARQL query plan optimizer

For more details see Wukong Project, including new features, roadmap, instructions, etc.

Getting Started

• Installation
• Tutorials
• Manual
• GPU extension
• Q&A

License

Wukong is released under the Apache 2.0 license.

If you use Wukong in your research, please cite our paper:

@inproceedings{osdi2016wukong,
 author = {Shi, Jiaxin and Yao, Youyang and Chen, Rong and Chen, Haibo and Li, Feifei},
 title = {Fast and Concurrent RDF Queries with RDMA-based Distributed Graph Exploration},
 booktitle = {12th USENIX Symposium on Operating Systems Design and Implementation},
 series = {OSDI '16},
 year = {2016},
 month = Nov,
 isbn = {978-1-931971-33-1},
 address = {GA},
 pages = {317--332},
 url = {https://www.usenix.org/conference/osdi16/technical-sessions/presentation/shi},
 publisher = {USENIX Association},
}

Academic and Reference Papers

[OSDI] Fast and Concurrent RDF Queries with RDMA-based Distributed Graph Exploration. Jiaxin Shi, Youyang Yao, Rong Chen, Haibo Chen, and Feifei Li. Proceedings of 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, GA, US, Nov, 2016.

[SOSP] Sub-millisecond Stateful Stream Querying over Fast-evolving Linked Data. Yunhao Zhang, Rong Chen, and Haibo Chen. Proceedings of the 26th ACM Symposium on Operating Systems Principles, Shanghai, China, October, 2017.

[USENIX ATC] Fast and Concurrent RDF Queries using RDMA-assisted GPU Graph Exploration. Siyuan Wang, Chang Lou, Rong Chen, and Haibo Chen. Proceedings of 2018 USENIX Annual Technical Conference, Boston, MA, US, July 2018.

[USENIX ATC] Pragh: Locality-preserving Graph Traversal with Split Live Migration. Xiating Xie, Xingda Wei, Rong Chen, and Haibo Chen. Proceedings of 2019 USENIX Annual Technical Conference, Renton, WA, US, July 2019.

wukong's Issues

在运行Build的时候，进入infinity loop

你好，作者。我试图在Ubuntu 20.04上安装Wukong，但我一直遇到以下问题：
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python (found version "2.7.18")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Configuring done
You have changed variables that require your cache to be deleted.
Configure will be re-run and you may have to reset some variables.
The following variables have changed:
CMAKE_CXX_COMPILER= /usr/bin/c++
CMAKE_CXX_COMPILER= /usr/bin/c++
CMAKE_CXX_COMPILER= /usr/bin/c++

build.sh脚本似乎进入了一个循环，尽管我尝试了多次，结果仍然相同。我已经阅读了你对其他人的建议，并尝试使用最新的仓库进行安装，但我发现安装指南已经过时，缺少了许多依赖项。请问我该如何解决这个bug，或者你有没有更全面的安装指南？非常感谢。

wukong是使用什么存储rdf数据的？

你好！请问wukong是使用什么来存储rdf数据的？

关于wukong系统查询相关问题

在proxy.hpp中run_single_query函数最后几行，代码段为
send_request(request);
reply = recv_reply();
第一条代码实际执行了将分解后的查询分发到各个对应的节点，第二条代码是各个节点监听是否收到消息，如果收到消息且是查询，就把查询赋值给reply。然后这个函数最后参数中的reply对应的是最后一次字查询的reply。
但是并没有找到实际执行查询的过程，即从vector中查找符合要求的项。
想知道wukong系统中实际执行查询是在哪一部分？
另外wukong底层存储spo或者ops是按照三元组有序存储的，基于主语或者宾语的查询是否是采用类似于的快速搜索这种搜索算法？

Can I export a loaded database?

I am trying to run lubm query on some large lubm datasets. It took large amount of time(several hours) to load large lubm dataset. It would be time-consuming to load it every time I want to try some new queries. Is there a way to export a loaded database into files that I can fastly bulkload them later?

关于query分发

query分发在代码中只体现为根据subject，似乎根据论文还应该有根据predicate和object的分发，是实现时有修改还是我漏看了哪部分代码？烦请指教

deps里缺乏文件

您好，我是一名高校研究生，在安装wukong的时候，发现deps里缺少文件librdma-1.0.0.tar.gz

查询失败

    您好，我执行SPARQL查询的时候报错了。我使用的sparql查询是wukong-master/scripts/sparql_query/lubm里面提供的查询语句。数据集的生成也是按照安装教程中的步骤生成的。希望您有空的时候可以回答一下。谢谢！

编译失败，cannot find -libverbs

您好！我在编译wukong的时候遇到了问题。因为我的服务器不支持rdma，所以我使用./build.sh -DUSE_RDMA=OFF命令编译。但是出现了以下的问题。麻烦您有空解答一下。谢谢！
root@node6:/usr/data/pikachu_test/app_install_by_src/wukong-master/scripts# ./build.sh -DUSE_RDMA=OFF
options: -DUSE_RDMA=OFF
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/data/pikachu_test/app_install_by_src/wukong-master/build
Scanning dependencies of target wukong
[ 50%] Building CXX object CMakeFiles/wukong.dir/core/wukong.cpp.o
[100%] Linking CXX executable wukong
/usr/bin/ld: cannot find -libverbs
collect2: error: ld returned 1 exit status
CMakeFiles/wukong.dir/build.make:97: recipe for target 'wukong' failed
make[2]: *** [wukong] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/wukong.dir/all' failed
make[1]: *** [CMakeFiles/wukong.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

编译不通过

您好，我按照步骤装好依赖后，没有对连通各个节点，只在自己主机上编译。然后出现了一个问题，显示找不到mpic++这个工具。
The CMAKE_CXX_COMPILER:

/root/ff/wukong/deps/openmpi-1.6.5-install/bin/mpic++

is not a full path to an existing compiler tool.
这是报错原话，我在bin目录下也找不到mpic++这个文件，请问这是怎么回事，求解答

Do you have a docker image

use a docker to simplfy deployment

增量更新

您好， 我看到wukong里面有DynamicGStore, 这个提供的功能是不是支持图关系的动态更新啊？

got bad completion with status: 0xc, vendor syndrome: 0x81

Dear all,

I'm trying to run Wukong with RDMA enabled, but I'm getting this error at loading time:

got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:0 t:1

It seems that others had this issue:

• #5
• #15

Answers in issue #5 mention using a new version of libRMDA, but I can't access it. Answers in issue #15 mention changing ports or adding SO_REUSEPORT to setsockopt(), but I tried both and I'm still getting the same error.

Any ideas?

关于qp及通信的相关问题

1. qps的含义是什么，对客户端和服务器来说内容有区别吗？
   2.get_local_qp_attr函数利用获取的qid返回的是发送方还是接收方的qp_attr?
   3.设备间建立连接的过程是在init2rtr还是在connect中change_qp_states之前？

How can I fix this? Assertion `(vid == v) && (dir == d) && (pid == p)' failed

I get the following assertion failure: "wukong: /home/jhkim/wukong/core/gstore.hpp:107: ikey_t::ikey_t(sid_t, sid_t, dir_t): Assertion `(vid == v) && (dir == d) && (pid == p)' failed."
I founded that at some moment, pid != p; pid = 86069 and p=1134645. Is it related to NBITS_IDX in core/gstore.hpp and datagen/generate_data.cpp?
I used single machine configuration, and inferenced lubm80 dataset

您好，代码无法运行

您好，我在使用wukong测试例子：tests/gpu/test-adaptor.cpp的时候。
如果使用GPU，会在RDMA_init函数里边停止运行，报错为Connect Device Memory Region failed at dev 0, err Bad address；
如果不使用GPU，将代码改为纯CPU的话，则会在send函数报错，got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:2 t:0；
特别是第二个问题，网上信息较少，请问应该如何解决？
十分感谢！望回复。

在运行run.sh时报错

您好！我在执行./run.sh 3命令时遇到了下面的一些问题，麻烦您在有空的时候解答一下。我使用的是Infiniband网卡，机器之间通信也没有问题，因为受机器内存的限制，我在分配内存的时候减少了空间。
[server11:504910] Warning: could not find environment variable "CLASSPATH"
INFO: TOPO: 1nodes
INFO: node 0 cores: 0 1 2 3
INFO: #0: has 4 cores.
INFO: #0: allocate 5.0625GB memory
INFO: TOPO: 1nodes
INFO: node 0 cores: 0 1 2 3
INFO: #2: has 4 cores.
INFO: #2: allocate 5.0625GB memory
INFO: TOPO: 1nodes
INFO: node 0 cores: 0 1 2 3
INFO: #1: has 4 cores.
INFO: #1: allocate 5.0625GB memory
INFO: initializing RMDA done (2157 ms)
INFO: initializing RMDA done (2161 ms)
INFO: loading ID-mapping file: /home/robert/rdfdata/str_normal
INFO: loading ID-mapping file: /home/robert/rdfdata/str_normal
INFO: initializing RMDA done (2163 ms)
INFO: loading ID-mapping file: /home/robert/rdfdata/str_normal
INFO: loading ID-mapping file: /home/robert/rdfdata/str_index
INFO: loading ID-mapping file: /home/robert/rdfdata/str_index
INFO: loading ID-mapping (attribute) file: /home/robert/rdfdata/str_attr_index
INFO: loading string server is finished (46 ms)
INFO: loading ID-mapping (attribute) file: /home/robert/rdfdata/str_attr_index
INFO: loading string server is finished (47 ms)
INFO: loading ID-mapping file: /home/robert/rdfdata/str_index
INFO: loading ID-mapping (attribute) file: /home/robert/rdfdata/str_attr_index
INFO: loading string server is finished (75 ms)
INFO: allocate 256MB RDMA cache
INFO: gstore = 4294967296 bytes
INFO: header region: 153008209 slots (main = 12582917, indirect = 6543109)
INFO: entry region: 461708984 entries
INFO: allocate 256MB RDMA cache
INFO: gstore = 4294967296 bytes
INFO: header region: 153008209 slots (main = 12582917, indirect = 6543109)
INFO: entry region: 461708984 entries
INFO: 2 files and 2 attributed files found in directory (/home/robert/rdfdata/) at server 0
INFO: 2 files and 2 attributed files found in directory (/home/robert/rdfdata/) at server 1
INFO: allocate 256MB RDMA cache
INFO: gstore = 4294967296 bytes
INFO: header region: 153008209 slots (main = 12582917, indirect = 6543109)
INFO: entry region: 461708984 entries
INFO: 2 files and 2 attributed files found in directory (/home/robert/rdfdata/) at server 2
got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:1 t:0
wukong: /home/robert/wukong/rdma_lib/rdmaio.hpp:769: rdmaio::Qp::IOStatus rdmaio::Qp::poll_completion(uint64_t*): Assertion false' failed. [server11:504915] *** Process received signal *** [server11:504915] Signal: Aborted (6) [server11:504915] Signal code: (-6) [server11:504915] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f4f64c883c0] [server11:504915] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f4f64ac518b] [server11:504915] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f4f64aa4859] [server11:504915] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x25729) [0x7f4f64aa4729] [server11:504915] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36) [0x7f4f64ab5f36] [server11:504915] [ 5] ../build/wukong(+0x113dfb) [0x55af55f15dfb] [server11:504915] [ 6] ../build/wukong(_ZN10BaseLoader13flush_triplesEii+0x1b3) [0x55af55f2db83] [server11:504915] [ 7] ../build/wukong(_ZN10BaseLoader4loadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt6vectorIS8_I8triple_tSaIS9_EESaISB_EESE_RS8_IS8_I13triple_attr_tSaISF_EESaISH_EE+0x44e) [0x55af55f7152e] [server11:504915] [ 8] ../build/wukong(_ZN6DGraphC2EiP3MemP12StringServerNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f3) [0x55af55f5f633] [server11:504915] [ 9] ../build/wukong(main+0x6af) [0x55af55ef7d1f] [server11:504915] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f4f64aa60b3] [server11:504915] [11] ../build/wukong(_start+0x2e) [0x55af55efaade] [server11:504915] *** End of error message *** got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:0 t:0 wukong: /home/robert/wukong/rdma_lib/rdmaio.hpp:769: rdmaio::Qp::IOStatus rdmaio::Qp::poll_completion(uint64_t*): Assertion false' failed.
[server12:447992] *** Process received signal ***
[server12:447992] Signal: Aborted (6)
[server12:447992] Signal code: (-6)
[server12:447992] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7f340c53c0]
[server12:447992] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f7f33f0218b]
[server12:447992] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f7f33ee1859]
[server12:447992] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x25729) [0x7f7f33ee1729]
[server12:447992] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36) [0x7f7f33ef2f36]
[server12:447992] [ 5] ../build/wukong(+0x113dfb) [0x55a443ccadfb]
[server12:447992] [ 6] ../build/wukong(_ZN10BaseLoader13flush_triplesEii+0x1b3) [0x55a443ce2b83]
[server12:447992] [ 7] ../build/wukong(_ZN10BaseLoader4loadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt6vectorIS8_I8triple_tSaIS9_EESaISB_EESE_RS8_IS8_I13triple_attr_tSaISF_EESaISH_EE+0x44e) [0x55a443d2652e]
[server12:447992] [ 8] ../build/wukong(_ZN6DGraphC2EiP3MemP12StringServerNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f3) [0x55a443d14633]
[server12:447992] [ 9] ../build/wukong(main+0x6af) [0x55a443cacd1f]
[server12:447992] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7f33ee30b3]
[server12:447992] [11] ../build/wukong(_start+0x2e) [0x55a443cafade]
[server12:447992] *** End of error message ***
got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:0 t:0
wukong: /home/robert/wukong/rdma_lib/rdmaio.hpp:769: rdmaio::Qp::IOStatus rdmaio::Qp::poll_completion(uint64_t*): Assertion `false' failed.
[server13:433953] *** Process received signal ***
[server13:433953] Signal: Aborted (6)
[server13:433953] Signal code: (-6)
[server13:433953] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fd3872f63c0]
[server13:433953] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fd38713318b]
[server13:433953] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fd387112859]
[server13:433953] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x25729) [0x7fd387112729]
[server13:433953] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36) [0x7fd387123f36]
[server13:433953] [ 5] ../build/wukong(+0x113dfb) [0x5622cef68dfb]
[server13:433953] [ 6] ../build/wukong(_ZN10BaseLoader13flush_triplesEii+0x1b3) [0x5622cef80b83]
[server13:433953] [ 7] ../build/wukong(_ZN10BaseLoader4loadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt6vectorIS8_I8triple_tSaIS9_EESaISB_EESE_RS8_IS8_I13triple_attr_tSaISF_EESaISH_EE+0x44e) [0x5622cefc452e]
[server13:433953] [ 8] ../build/wukong(_ZN6DGraphC2EiP3MemP12StringServerNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f3) [0x5622cefb2633]
[server13:433953] [ 9] ../build/wukong(main+0x6af) [0x5622cef4ad1f]
[server13:433953] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd3871140b3]
[server13:433953] [11] ../build/wukong(_start+0x2e) [0x5622cef4dade]
[server13:433953] *** End of error message ***

mpiexec noticed that process rank 0 with PID 504915 on node server11 exited on signal 6 (Aborted).

Failed to modify RC to RTR state, No such device or address

你好，我在虚拟机上安装了两台Ubuntu16.04，按照INSTALL.md的步骤安装完成后，运行出现了下面问题，想请教一下。谢谢。
root@server:/code/wukong/scripts# ./run.sh 2
[server:06477] Warning: could not find environment variable "CLASSPATH"
INFO: TOPO: 1nodes
INFO: node 0 cores: 0
INFO: #0: has 1 cores.
INFO: TOPO: 1nodes
INFO: node 0 cores: 0
INFO: #1: has 1 cores.
INFO: #0: allocate 1.04688GB memory
INFO: #1: allocate 1.04688GB memory
[librdma] : listener binding: tcp://:19344
[librdma] : listener binding: tcp://:19344
[librdma] qp: Failed to modify RC to RTR state, No such device or address
[librdma] : recv thread exit!

这个是配置文档
cat config
global_num_proxies 1
global_num_engines 2
global_input_folder /code/wukong/datasets/id_lubm_2
global_data_port_base 5500
global_ctrl_port_base 9576
global_memstore_size_gb 1
global_rdma_buf_size_mb 8
global_rdma_rbf_size_mb 4
global_use_rdma 1
global_rdma_threshold 300
global_mt_threshold 2
global_enable_caching 0
global_enable_workstealing 0
global_silent 0
global_enable_planner 1
global_generate_statistics 1
global_enable_vattr 1

cat mpd.hosts
10.211.55.11
10.211.55.10

cat core.bind
0 1 2

Is Wukong+G available now?

Wukong+G, the paper titled "Fast and Concurrent RDF Queries using RDMA-assisted GPU Graph Exploration".