miyurud / jasminegraph

Distributed Graph Database Server

JasmineGraph

1. Introduction

JasmineGraph is a C/C++ based distributed graph database server. It has been developed following the Acacia graph database server architecture. JasmineGraph can be run on single computer as well as on a compute cluster.

2. Building and Running JasmineGraph on Docker

JasmineGraph can be run inside a docker image. After cloning the project, build the image as follows:

docker build -t jasminegraph .

Run the image by providing the appropriate volume mount paths and parameters:

docker run -v "/var/run/docker.sock:/var/run/docker.sock:rw" -v "/root/.ssh:/home/user/.ssh" -v "/tmp:/tmp" -v "/var/tmp/jasminegraph-localstore:/var/tmp/jasminegraph-localstore" -v "/var/tmp/jasminegraph-aggregate:/var/tmp/jasminegraph-aggregate" -v "/home/user/Documents/jasminegraph/metadb:/home/ubuntu/software/jasminegraph/metadb" -v "/home/user/Documents/MSc/jasminegraph/performancedb:/home/ubuntu/software/jasminegraph/performancedb" -p 7777:7777 -p 7778:7778 jasminegraph --MODE 1 --MASTERIP <docker0 interface ip> --WORKERS 4 --WORKERIP <docker0 interface ip> --ENABLE_NMON false

Run JasmineGraph in Kubernetes environment by providing appropriate parameters:

./start-k8s.sh --META_DB_PATH "$(pwd)/metadb" \
    --PERFORMANCE_DB_PATH "$(pwd)/performancedb" \
    --DATA_PATH "$(pwd)/data" \
    --LOG_PATH "$(pwd)/logs" \
    --AGGREGATE_PATH "$(pwd)/aggregate" \
    --NO_OF_WORKERS 2 \
    --ENABLE_NMON false \
    --MAX_COUNT 4

Note: NO_OF_WORKERS, MAX_COUNT, and ENABLE_NMON are optional arguments, and it defaults to 2, 4 and false respectively.

Remove all resources created by JasmineGraph in Kubernetes environment:

./start-k8s.sh clean

3. Contact Information

Please contact [Miyuru Dayarathna](miyurud at yahoo dot co dot uk) for further information. Please let us know about bug reports or any further improvements you wish to have in JasmineGraph.

Open Source License

JasmineGraph is licensed under the Apache License, Version 2.0. Task Scheduler which is integrated in JasmineGraph is using MIT License.

References

More details of JasmineGraph's approach for managment and mining of large graph data is available from the following list of papers.

• Chinthaka Weerakkody, Miyuru Dayarathna, Sanath Jayasena, and Toyotaro Suzumura. 2022. "Guaranteeing Service Level Agreements for Triangle Counting via Observation-based Admission Control Algorithm.," in 2022 IEEE 15th International Conference on Cloud Computing (CLOUD), Barcelona, Spain, 2022 pp. 283-288.
• Damitha Senevirathne, Isuru Wijesiri, Suchitha Dehigaspitiya, Miyuru Dayarathna, Sanath Jayasena, and Toyotaro Suzumura. 2020. "Memory Efficient Graph Convolutional Network based Distributed Link Prediction," 2020 IEEE International Conference on Big Data (Big Data), Atlanta, GA, USA, pp. 2977-2986.
• Anuradha Karunarathna, Dinika Senarath, Shalika Madhushanki, Chinthaka Weerakkody, Miyuru Dayarathna, Sanath Jayasena, and Toyotaro Suzumura. 2020. "Scalable Graph Convolutional Network based Link Prediction on a Distributed Graph Database Server.," IEEE 13th International Conference on Cloud Computing (CLOUD), Beijing, China, 2020, pp. 107-115.
• Miyuru Dayarathna, Sathya Bandara, Nandula Jayamaha, Mahen Herath, Achala Madhushan, Sanath Jayasena, and Toyotaro Suzumura. 2017. "An X10-Based Distributed Streaming Graph Database Engine.," 2017 IEEE 24th International Conference on High Performance Computing (HiPC), Jaipur, 2017, pp. 243-252.
• Miyuru Dayarathna, Isuru Herath, Yasima Dewmini, Gayan Mettananda, Sameera Nandasiri, Sanath Jayasena, and Toyotaro Suzumura. 2016 "Acacia-RDF: An X10-Based Scalable Distributed RDF Graph Database Engine.," 2016 IEEE 9th International Conference on Cloud Computing (CLOUD), San Francisco, CA, 2016, pp. 521-528. doi: 10.1109/CLOUD.2016.0075
• Miyuru Dayarathna and Toyotaro Suzumura. 2014. "Towards scalable distributed graph database engine for hybrid clouds." In Proceedings of the 5th International Workshop on Data-Intensive Computing in the Clouds (DataCloud '14). IEEE Press, Piscataway, NJ, USA, 1-8.

jasminegraph's Issues

Change nlohmann_json version(3.9.1) in the readme file

nlohmann_json version 3.9.1 raises the following error when running the jasminegraph on the local machine.

The current newest version(3.11.2) works without any errors. Therefore, please update the readme file.
Version 3.11.2 Link: https://github.com/nlohmann/json/archive/refs/tags/v3.11.2.zip

Build two binaries for server and worker

Use two targets in CMake and, generate two binaries for JasmineGraph server and worker profiles.

Create an Exception class for JasmineGraph

Better create an Exception class for JasmineGraph. We can use the name JasmineGraphException. We can create a top level package called exception and within that host the JasmineGraphException class.

Fix output[0][0]

There can be situations where output[0][0] is null. Need to check such occurences and fix them all.

JasmineGraph server crashes when `merge` command called on graph uploaded without attribute file

Can reproduce by calling merge command on powergrid graph uploaded in integration tests

Integrate the scheduling algorithm to fl branch

We need to get the complete code including the scheduling algorithm committed to the fl branch.

Average Execution Time of Triangle Counting

We need to have a way of determining the average execution time of triangle counting and store that metric in performance db.

Investigate on Buffer size of 300

Need to have constant for Buffer size of 300.

Refactor file storage

Currently, the partitioned graph data of JasmineGraph gets stored in org.jasminegraph.server.instance.datafolder and we have set its value by default to /var/tmp/jasminegraph-localstore. Instead of that we need to change the folder name to /var/tmp/jasminegraph-datastore. Furthermore, need to change this single folder to hold all the data of JasmineGraph.

Missing Model Status Table

Even though in the Frontend the model status table is referred it has not been created. @DamithaSenevirathne Can you please check this and send a fix? See

Set the default training approach to single organization

Need to change the following line in configuration file and set its value to false.

org.jasminegraph.fl.org.training=false

Remove native mode related code segments

We created this issue to keep track of removal of native mode related code.

Complete the docker version of the fl branch

We need to create the docker version of the fl branch so that we could test the fl functionality without setting up the native version locally.

Fix Segfault for Merge Command

Segfault happens when run merge command for certain types of graphs. Merge command is intended for only use with attribute graphs. Hence there should be appropriate checking and show proper error message.

Relate log lines with frontend service session ID

If we take any of the log lines such as follows,

frontend_logger.info("Calculating Page Rank");

Those should carry their corresponding frontend (FE) service session ID as well. This will enable us to differentiate between different sessions. Another example for such log line is as follows.

frontend_logger.info("PageRank Time Taken : " + to_string(msDuration) + " milliseconds");

If we do not keep track of in which FE service session this log line was made, it will become difficult to track between different FE service sessions.

Unify the merge command with the train command in the frontend

There is no point of having a separate merge command in the frontend. We should merge it with the train command. Furthermore, merging currently conducted as the merging of two pandas data frames. Rather than this, first the local store and corresponding central store needs to be merged using C++ and then should construct the Pandas data frame using the data in the merged data store.