Coder Social home page Coder Social logo

sjtu-ipads / wukong Goto Github PK

View Code? Open in Web Editor NEW
184.0 17.0 29.0 111.41 MB

A graph-based distributed in-memory store that leverages efficient graph exploration to provide highly concurrent and low-latency queries over big linked data

Home Page: http://ipads.se.sjtu.edu.cn/projects/wukong

License: Apache License 2.0

CMake 0.50% C++ 96.47% Shell 0.31% Cuda 2.72%
graph-query-store in-memory rdma low-latency-queries distributed-systems

wukong's Issues

How can I fix this? Assertion `(vid == v) && (dir == d) && (pid == p)' failed

I get the following assertion failure: "wukong: /home/jhkim/wukong/core/gstore.hpp:107: ikey_t::ikey_t(sid_t, sid_t, dir_t): Assertion `(vid == v) && (dir == d) && (pid == p)' failed."
I founded that at some moment, pid != p; pid = 86069 and p=1134645. Is it related to NBITS_IDX in core/gstore.hpp and datagen/generate_data.cpp?
I used single machine configuration, and inferenced lubm80 dataset

编译不通过

您好,我按照步骤装好依赖后,没有对连通各个节点,只在自己主机上编译。然后出现了一个问题,显示找不到mpic++这个工具。
The CMAKE_CXX_COMPILER:

/root/ff/wukong/deps/openmpi-1.6.5-install/bin/mpic++

is not a full path to an existing compiler tool.
这是报错原话,我在bin目录下也找不到mpic++这个文件,请问这是怎么回事,求解答

关于query分发

d7c0771e-aef1-4327-be6b-b8468dd838cc
query分发在代码中只体现为根据subject,似乎根据论文还应该有根据predicate和object的分发,是实现时有修改还是我漏看了哪部分代码?烦请指教

Failed to modify RC to RTR state, No such device or address

你好,我在虚拟机上安装了两台Ubuntu16.04,按照INSTALL.md的步骤安装完成后,运行出现了下面问题,想请教一下。谢谢。
root@server:/code/wukong/scripts# ./run.sh 2
[server:06477] Warning: could not find environment variable "CLASSPATH"
INFO: TOPO: 1nodes
INFO: node 0 cores: 0
INFO: #0: has 1 cores.
INFO: TOPO: 1nodes
INFO: node 0 cores: 0
INFO: #1: has 1 cores.
INFO: #0: allocate 1.04688GB memory
INFO: #1: allocate 1.04688GB memory
[librdma] : listener binding: tcp://:19344
[librdma] : listener binding: tcp://
:19344
[librdma] qp: Failed to modify RC to RTR state, No such device or address
[librdma] : recv thread exit!

这个是配置文档
cat config
global_num_proxies 1
global_num_engines 2
global_input_folder /code/wukong/datasets/id_lubm_2
global_data_port_base 5500
global_ctrl_port_base 9576
global_memstore_size_gb 1
global_rdma_buf_size_mb 8
global_rdma_rbf_size_mb 4
global_use_rdma 1
global_rdma_threshold 300
global_mt_threshold 2
global_enable_caching 0
global_enable_workstealing 0
global_silent 0
global_enable_planner 1
global_generate_statistics 1
global_enable_vattr 1

cat mpd.hosts
10.211.55.11
10.211.55.10

cat core.bind
0 1 2

got bad completion with status: 0xc, vendor syndrome: 0x81

Dear all,

I'm trying to run Wukong with RDMA enabled, but I'm getting this error at loading time:

got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:0 t:1

It seems that others had this issue:

Answers in issue #5 mention using a new version of libRMDA, but I can't access it. Answers in issue #15 mention changing ports or adding SO_REUSEPORT to setsockopt(), but I tried both and I'm still getting the same error.

Any ideas?

在运行Build的时候,进入infinity loop

你好,作者。我试图在Ubuntu 20.04上安装Wukong,但我一直遇到以下问题:
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: /usr/bin/python (found version "2.7.18")
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Configuring done
You have changed variables that require your cache to be deleted.
Configure will be re-run and you may have to reset some variables.
The following variables have changed:
CMAKE_CXX_COMPILER= /usr/bin/c++
CMAKE_CXX_COMPILER= /usr/bin/c++
CMAKE_CXX_COMPILER= /usr/bin/c++

build.sh脚本似乎进入了一个循环,尽管我尝试了多次,结果仍然相同。我已经阅读了你对其他人的建议,并尝试使用最新的仓库进行安装,但我发现安装指南已经过时,缺少了许多依赖项。请问我该如何解决这个bug,或者你有没有更全面的安装指南?非常感谢。

关于qp及通信的相关问题

  1. qps的含义是什么,对客户端和服务器来说内容有区别吗?
    2.get_local_qp_attr函数利用获取的qid返回的是发送方还是接收方的qp_attr?
    3.设备间建立连接的过程是在init2rtr还是在connect中change_qp_states之前?

查询失败

    您好,我执行SPARQL查询的时候报错了。我使用的sparql查询是wukong-master/scripts/sparql_query/lubm里面提供的查询语句。数据集的生成也是按照安装教程中的步骤生成的。希望您有空的时候可以回答一下。谢谢!

屏幕快照 2019-11-04 09 35 40

Can I export a loaded database?

I am trying to run lubm query on some large lubm datasets. It took large amount of time(several hours) to load large lubm dataset. It would be time-consuming to load it every time I want to try some new queries. Is there a way to export a loaded database into files that I can fastly bulkload them later?

在运行run.sh时报错

您好!我在执行./run.sh 3命令时遇到了下面的一些问题,麻烦您在有空的时候解答一下。我使用的是Infiniband网卡,机器之间通信也没有问题,因为受机器内存的限制,我在分配内存的时候减少了空间。
[server11:504910] Warning: could not find environment variable "CLASSPATH"
INFO: TOPO: 1nodes
INFO: node 0 cores: 0 1 2 3
INFO: #0: has 4 cores.
INFO: #0: allocate 5.0625GB memory
INFO: TOPO: 1nodes
INFO: node 0 cores: 0 1 2 3
INFO: #2: has 4 cores.
INFO: #2: allocate 5.0625GB memory
INFO: TOPO: 1nodes
INFO: node 0 cores: 0 1 2 3
INFO: #1: has 4 cores.
INFO: #1: allocate 5.0625GB memory
INFO: initializing RMDA done (2157 ms)
INFO: initializing RMDA done (2161 ms)
INFO: loading ID-mapping file: /home/robert/rdfdata/str_normal
INFO: loading ID-mapping file: /home/robert/rdfdata/str_normal
INFO: initializing RMDA done (2163 ms)
INFO: loading ID-mapping file: /home/robert/rdfdata/str_normal
INFO: loading ID-mapping file: /home/robert/rdfdata/str_index
INFO: loading ID-mapping file: /home/robert/rdfdata/str_index
INFO: loading ID-mapping (attribute) file: /home/robert/rdfdata/str_attr_index
INFO: loading string server is finished (46 ms)
INFO: loading ID-mapping (attribute) file: /home/robert/rdfdata/str_attr_index
INFO: loading string server is finished (47 ms)
INFO: loading ID-mapping file: /home/robert/rdfdata/str_index
INFO: loading ID-mapping (attribute) file: /home/robert/rdfdata/str_attr_index
INFO: loading string server is finished (75 ms)
INFO: allocate 256MB RDMA cache
INFO: gstore = 4294967296 bytes
INFO: header region: 153008209 slots (main = 12582917, indirect = 6543109)
INFO: entry region: 461708984 entries
INFO: allocate 256MB RDMA cache
INFO: gstore = 4294967296 bytes
INFO: header region: 153008209 slots (main = 12582917, indirect = 6543109)
INFO: entry region: 461708984 entries
INFO: 2 files and 2 attributed files found in directory (/home/robert/rdfdata/) at server 0
INFO: 2 files and 2 attributed files found in directory (/home/robert/rdfdata/) at server 1
INFO: allocate 256MB RDMA cache
INFO: gstore = 4294967296 bytes
INFO: header region: 153008209 slots (main = 12582917, indirect = 6543109)
INFO: entry region: 461708984 entries
INFO: 2 files and 2 attributed files found in directory (/home/robert/rdfdata/) at server 2
got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:1 t:0
wukong: /home/robert/wukong/rdma_lib/rdmaio.hpp:769: rdmaio::Qp::IOStatus rdmaio::Qp::poll_completion(uint64_t*): Assertion false' failed. [server11:504915] *** Process received signal *** [server11:504915] Signal: Aborted (6) [server11:504915] Signal code: (-6) [server11:504915] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f4f64c883c0] [server11:504915] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f4f64ac518b] [server11:504915] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f4f64aa4859] [server11:504915] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x25729) [0x7f4f64aa4729] [server11:504915] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36) [0x7f4f64ab5f36] [server11:504915] [ 5] ../build/wukong(+0x113dfb) [0x55af55f15dfb] [server11:504915] [ 6] ../build/wukong(_ZN10BaseLoader13flush_triplesEii+0x1b3) [0x55af55f2db83] [server11:504915] [ 7] ../build/wukong(_ZN10BaseLoader4loadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt6vectorIS8_I8triple_tSaIS9_EESaISB_EESE_RS8_IS8_I13triple_attr_tSaISF_EESaISH_EE+0x44e) [0x55af55f7152e] [server11:504915] [ 8] ../build/wukong(_ZN6DGraphC2EiP3MemP12StringServerNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f3) [0x55af55f5f633] [server11:504915] [ 9] ../build/wukong(main+0x6af) [0x55af55ef7d1f] [server11:504915] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f4f64aa60b3] [server11:504915] [11] ../build/wukong(_start+0x2e) [0x55af55efaade] [server11:504915] *** End of error message *** got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:0 t:0 wukong: /home/robert/wukong/rdma_lib/rdmaio.hpp:769: rdmaio::Qp::IOStatus rdmaio::Qp::poll_completion(uint64_t*): Assertion false' failed.
[server12:447992] *** Process received signal ***
[server12:447992] Signal: Aborted (6)
[server12:447992] Signal code: (-6)
[server12:447992] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7f340c53c0]
[server12:447992] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f7f33f0218b]
[server12:447992] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f7f33ee1859]
[server12:447992] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x25729) [0x7f7f33ee1729]
[server12:447992] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36) [0x7f7f33ef2f36]
[server12:447992] [ 5] ../build/wukong(+0x113dfb) [0x55a443ccadfb]
[server12:447992] [ 6] ../build/wukong(_ZN10BaseLoader13flush_triplesEii+0x1b3) [0x55a443ce2b83]
[server12:447992] [ 7] ../build/wukong(_ZN10BaseLoader4loadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt6vectorIS8_I8triple_tSaIS9_EESaISB_EESE_RS8_IS8_I13triple_attr_tSaISF_EESaISH_EE+0x44e) [0x55a443d2652e]
[server12:447992] [ 8] ../build/wukong(_ZN6DGraphC2EiP3MemP12StringServerNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f3) [0x55a443d14633]
[server12:447992] [ 9] ../build/wukong(main+0x6af) [0x55a443cacd1f]
[server12:447992] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7f33ee30b3]
[server12:447992] [11] ../build/wukong(_start+0x2e) [0x55a443cafade]
[server12:447992] *** End of error message ***
got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:0 t:0
wukong: /home/robert/wukong/rdma_lib/rdmaio.hpp:769: rdmaio::Qp::IOStatus rdmaio::Qp::poll_completion(uint64_t*): Assertion `false' failed.
[server13:433953] *** Process received signal ***
[server13:433953] Signal: Aborted (6)
[server13:433953] Signal code: (-6)
[server13:433953] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7fd3872f63c0]
[server13:433953] [ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7fd38713318b]
[server13:433953] [ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7fd387112859]
[server13:433953] [ 3] /lib/x86_64-linux-gnu/libc.so.6(+0x25729) [0x7fd387112729]
[server13:433953] [ 4] /lib/x86_64-linux-gnu/libc.so.6(+0x36f36) [0x7fd387123f36]
[server13:433953] [ 5] ../build/wukong(+0x113dfb) [0x5622cef68dfb]
[server13:433953] [ 6] ../build/wukong(_ZN10BaseLoader13flush_triplesEii+0x1b3) [0x5622cef80b83]
[server13:433953] [ 7] ../build/wukong(_ZN10BaseLoader4loadERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt6vectorIS8_I8triple_tSaIS9_EESaISB_EESE_RS8_IS8_I13triple_attr_tSaISF_EESaISH_EE+0x44e) [0x5622cefc452e]
[server13:433953] [ 8] ../build/wukong(_ZN6DGraphC2EiP3MemP12StringServerNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x1f3) [0x5622cefb2633]
[server13:433953] [ 9] ../build/wukong(main+0x6af) [0x5622cef4ad1f]
[server13:433953] [10] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fd3871140b3]
[server13:433953] [11] ../build/wukong(_start+0x2e) [0x5622cef4dade]
[server13:433953] *** End of error message ***

mpiexec noticed that process rank 0 with PID 504915 on node server11 exited on signal 6 (Aborted).

编译失败,cannot find -libverbs

您好!我在编译wukong的时候遇到了问题。因为我的服务器不支持rdma,所以我使用./build.sh -DUSE_RDMA=OFF命令编译。但是出现了以下的问题。麻烦您有空解答一下。谢谢!
root@node6:/usr/data/pikachu_test/app_install_by_src/wukong-master/scripts# ./build.sh -DUSE_RDMA=OFF
options: -DUSE_RDMA=OFF
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/data/pikachu_test/app_install_by_src/wukong-master/build
Scanning dependencies of target wukong
[ 50%] Building CXX object CMakeFiles/wukong.dir/core/wukong.cpp.o
[100%] Linking CXX executable wukong
/usr/bin/ld: cannot find -libverbs
collect2: error: ld returned 1 exit status
CMakeFiles/wukong.dir/build.make:97: recipe for target 'wukong' failed
make[2]: *** [wukong] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/wukong.dir/all' failed
make[1]: *** [CMakeFiles/wukong.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

Is Wukong+G available now?

Wukong+G, the paper titled "Fast and Concurrent RDF Queries using RDMA-assisted GPU Graph Exploration".

您好,代码无法运行

您好,我在使用wukong测试例子:tests/gpu/test-adaptor.cpp的时候。
如果使用GPU,会在RDMA_init函数里边停止运行,报错为Connect Device Memory Region failed at dev 0, err Bad address;
如果不使用GPU,将代码改为纯CPU的话,则会在send函数报错,got bad completion with status: 0xc, vendor syndrome: 0x81, with error transport retry counter exceeded, qp n:2 t:0;
特别是第二个问题,网上信息较少,请问应该如何解决?
十分感谢!望回复。

deps里缺乏文件

您好,我是一名高校研究生,在安装wukong的时候,发现deps里缺少文件librdma-1.0.0.tar.gz

增量更新

您好, 我看到wukong里面有DynamicGStore, 这个提供的功能是不是支持图关系的动态更新啊?

关于wukong系统查询相关问题

在proxy.hpp中run_single_query函数最后几行,代码段为
send_request(request);
reply = recv_reply();
第一条代码实际执行了将分解后的查询分发到各个对应的节点,第二条代码是各个节点监听是否收到消息,如果收到消息且是查询,就把查询赋值给reply。然后这个函数最后参数中的reply对应的是最后一次字查询的reply。
但是并没有找到实际执行查询的过程,即从vector中查找符合要求的项。
想知道wukong系统中实际执行查询是在哪一部分?
另外wukong底层存储spo或者ops是按照三元组有序存储的,基于主语或者宾语的查询是否是采用类似于的快速搜索这种搜索算法?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.