Coder Social home page Coder Social logo

nebula-contrib / nebulagraph-lite Goto Github PK

View Code? Open in Web Editor NEW
11.0 1.0 0.0 1.29 MB

The minimal, ad-hoc way of plug and play NebulaGraph with pip install, even inside Colab Notebook!

License: Apache License 2.0

Shell 5.59% Python 76.23% Jupyter Notebook 17.19% Dockerfile 0.98%
graphdatabase nebulagraph colab

nebulagraph-lite's Issues

Unable to start instance in kaggle notebook

Steps

! pip3 install nebulagraph-lite
from nebulagraph_lite import nebulagraph_let as ng_let
n = ng_let()
n.start()

Collecting nebulagraph-lite
  Downloading nebulagraph_lite-0.2.2-py3-none-any.whl.metadata (7.5 kB)
Collecting udocker==1.3.13 (from nebulagraph-lite)
  Downloading udocker-1.3.13-py2.py3-none-any.whl.metadata (36 kB)
Collecting psutil>=5.9.6 (from nebulagraph-lite)
  Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting nebula3-python>=3.5.0 (from nebulagraph-lite)
  Using cached nebula3_python-3.5.0-py3-none-any.whl.metadata (8.4 kB)
Requirement already satisfied: future>=0.18.0 in /opt/conda/lib/python3.10/site-packages (from nebula3-python>=3.5.0->nebulagraph-lite) (1.0.0)
Requirement already satisfied: httplib2>=0.20.0 in /opt/conda/lib/python3.10/site-packages (from nebula3-python>=3.5.0->nebulagraph-lite) (0.21.0)
Requirement already satisfied: pytz>=2021.1 in /opt/conda/lib/python3.10/site-packages (from nebula3-python>=3.5.0->nebulagraph-lite) (2023.3.post1)
Requirement already satisfied: six>=1.16.0 in /opt/conda/lib/python3.10/site-packages (from nebula3-python>=3.5.0->nebulagraph-lite) (1.16.0)
Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /opt/conda/lib/python3.10/site-packages (from httplib2>=0.20.0->nebula3-python>=3.5.0->nebulagraph-lite) (3.1.1)
Downloading nebulagraph_lite-0.2.2-py3-none-any.whl (16 kB)
Downloading udocker-1.3.13-py2.py3-none-any.whl (118 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.6/118.6 kB 4.8 MB/s eta 0:00:00
Using cached nebula3_python-3.5.0-py3-none-any.whl (326 kB)
Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 288.2/288.2 kB 12.3 MB/s eta 0:00:00
Installing collected packages: udocker, psutil, nebula3-python, nebulagraph-lite
  Attempting uninstall: psutil
    Found existing installation: psutil 5.9.3
    Uninstalling psutil-5.9.3:
      Successfully uninstalled psutil-5.9.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
beatrix-jupyterlab 2023.128.151533 requires jupyterlab~=3.6.0, but you have jupyterlab 4.1.2 which is incompatible.
distributed 2023.7.1 requires dask==2023.7.1, but you have dask 2024.2.0 which is incompatible.
jupyterlab 4.1.2 requires jupyter-lsp>=2.0.0, but you have jupyter-lsp 1.5.1 which is incompatible.
jupyterlab-lsp 5.0.3 requires jupyter-lsp>=2.0.0, but you have jupyter-lsp 1.5.1 which is incompatible.
raft-dask 23.8.0 requires dask==2023.7.1, but you have dask 2024.2.0 which is incompatible.
Successfully installed nebula3-python-3.5.0 nebulagraph-lite-0.2.2 psutil-5.9.7 udocker-1.3.13

feat: stop gracefully

It seems now the stop of service will cause metad corruption for always, cleanup is needed :(

❯ tail  ~/.nebulagraph/lite/logs/meta0/*
==> /home/w/.nebulagraph/lite/logs/meta0/metad-stderr.log <==
/usr/local/nebula/bin/nebula-metad(event_base_loop+0x36e)[0x2648e8e]
/usr/local/nebula/bin/nebula-metad(_ZN5folly9EventBase8loopBodyEib+0x46d)[0x2586efd]
/usr/local/nebula/bin/nebula-metad(_ZN5folly9EventBase4loopEv+0x3d)[0x25877ed]
/usr/local/nebula/bin/nebula-metad(_ZN5folly9EventBase11loopForeverEv+0x17)[0x258a427]
/usr/local/nebula/bin/nebula-metad(_ZN5folly20IOThreadPoolExecutor9threadRunESt10shared_ptrINS_18ThreadPoolExecutor6ThreadEE+0x338)[0x2514578]
/usr/local/nebula/bin/nebula-metad(_ZN5folly6detail8function14FunctionTraitsIFvvEE7callBigISt5_BindIFMNS_18ThreadPoolExecutorEFvSt10shared_ptrINS7_6ThreadEEEPS7_SA_EEEEvRNS1_4DataE+0x46)[0x2522ae6]
/usr/local/nebula/bin/nebula-metad[0x2b4d5ef]
/lib64/libpthread.so.0(+0x7ea4)[0x7c7760807ea4]
/lib64/libc.so.6(clone+0x6c)[0x7c77604fe96c]
(safe mode, symbolizer not available)

==> /home/w/.nebulagraph/lite/logs/meta0/metad-stdout.log <==

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.ERROR <==
Log file created at: 2024/01/14 10:52:40
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.FATAL <==
Log file created at: 2024/01/14 10:52:40
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.INFO <==
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20240114 10:52:40.028664  6557 MetaDaemon.cpp:137] localhost = "127.0.0.1":9559
I20240114 10:52:40.337798  6557 NebulaStore.cpp:48] Start the raft service...
I20240114 10:52:40.343425  6557 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20240114 10:52:40.637010  6557 RaftexService.cpp:46] Start raft service on 9560
I20240114 10:52:40.647907  6557 NebulaStore.cpp:82] Scan the local path, and init the spaces_
I20240114 10:52:40.648157  6557 NebulaStore.cpp:90] Scan path "/data/meta/nebula/0"
I20240114 10:52:40.648191  6557 NebulaStore.cpp:292] Init data from partManager for "127.0.0.1":9559
I20240114 10:52:40.648216  6557 NebulaStore.cpp:417] Create data space 0
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.ERROR.20240114-104504.3898 <==
Log file created at: 2024/01/14 10:45:04
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20240114 10:45:04.757242  3898 FileUtils.cpp:377] Failed to read the directory "/data/meta/nebula" (2): No such file or directory
E20240114 10:45:06.707832  3898 FileUtils.cpp:76] Failed to remove the directory "/data/meta/nebula/0/checkpoints/META_UPGRADE_SNAPSHOT_2024_01_14_10_45_06/data" (39): Directory not empty
E20240114 10:45:06.707902  3898 FileUtils.cpp:49] Failed to remove "data" in "/data/meta/nebula/0/checkpoints/META_UPGRADE_SNAPSHOT_2024_01_14_10_45_06"
E20240114 10:45:06.707931  3898 FileUtils.cpp:70] Failed to remove the content of the directory "/data/meta/nebula/0/checkpoints/META_UPGRADE_SNAPSHOT_2024_01_14_10_45_06"

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.ERROR.20240114-105240.6557 <==
Log file created at: 2024/01/14 10:52:40
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.FATAL.20240114-105240.6557 <==
Log file created at: 2024/01/14 10:52:40
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.INFO.20240114-104504.3898 <==
I20240114 10:51:34.553622  3898 RaftexService.cpp:67] Stopping the raftex service on port 9560
I20240114 10:51:34.553655  3898 RaftexService.cpp:77] All partitions have stopped
I20240114 10:51:34.553689  3898 NebulaStore.cpp:346] Stop kv engine...
I20240114 10:51:34.553740  3898 NebulaStore.cpp:36] Cut off the relationship with meta client
I20240114 10:51:34.570483  3898 Part.h:59] [Port: 9560, Space: 0, Part: 0] ~Part()
I20240114 10:51:34.579130  3898 RocksEngine.h:247] Release rocksdb on /data/meta/nebula/0
I20240114 10:51:34.580722  3898 NebulaStore.cpp:44] ~NebulaStore()
I20240114 10:51:34.580850  3898 MetaDaemon.cpp:215] The meta Daemon stopped
I20240114 10:51:34.593420  3898 JobManager.cpp:138] JobManager::shutDown() begin
I20240114 10:51:34.593461  3898 JobManager.cpp:141] JobManager not running, exit

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.INFO.20240114-105240.6557 <==
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
I20240114 10:52:40.028664  6557 MetaDaemon.cpp:137] localhost = "127.0.0.1":9559
I20240114 10:52:40.337798  6557 NebulaStore.cpp:48] Start the raft service...
I20240114 10:52:40.343425  6557 NebulaSnapshotManager.cpp:25] Send snapshot is rate limited to 10485760 for each part by default
I20240114 10:52:40.637010  6557 RaftexService.cpp:46] Start raft service on 9560
I20240114 10:52:40.647907  6557 NebulaStore.cpp:82] Scan the local path, and init the spaces_
I20240114 10:52:40.648157  6557 NebulaStore.cpp:90] Scan path "/data/meta/nebula/0"
I20240114 10:52:40.648191  6557 NebulaStore.cpp:292] Init data from partManager for "127.0.0.1":9559
I20240114 10:52:40.648216  6557 NebulaStore.cpp:417] Create data space 0
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.WARNING.20240114-104504.3898 <==
Log file created at: 2024/01/14 10:45:04
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20240114 10:45:04.757242  3898 FileUtils.cpp:377] Failed to read the directory "/data/meta/nebula" (2): No such file or directory
E20240114 10:45:06.707832  3898 FileUtils.cpp:76] Failed to remove the directory "/data/meta/nebula/0/checkpoints/META_UPGRADE_SNAPSHOT_2024_01_14_10_45_06/data" (39): Directory not empty
E20240114 10:45:06.707902  3898 FileUtils.cpp:49] Failed to remove "data" in "/data/meta/nebula/0/checkpoints/META_UPGRADE_SNAPSHOT_2024_01_14_10_45_06"
E20240114 10:45:06.707931  3898 FileUtils.cpp:70] Failed to remove the content of the directory "/data/meta/nebula/0/checkpoints/META_UPGRADE_SNAPSHOT_2024_01_14_10_45_06"

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.penguin.root.log.WARNING.20240114-105240.6557 <==
Log file created at: 2024/01/14 10:52:40
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

==> /home/w/.nebulagraph/lite/logs/meta0/nebula-metad.WARNING <==
Log file created at: 2024/01/14 10:52:40
Running on machine: penguin
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
F20240114 10:52:40.701972  6635 RocksEngine.cpp:82] Check failed: status.ok() Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data/meta/nebula/0/data/000009.ldb: No such file or directory in file /data/meta/nebula/0/data/MANIFEST-000005

design of nebulagraph-lite

Target system:

  • Linux
  • Google Colab
  • WSL2 Ubuntu
  • macOS Docker Desktop(i.e. Jupyter extension running in container)/OrbStack/Lima

Dependencies:

Usage:

pip install nebulagraph-lite
from nebulagraph_lite import nebulagraph_let as ng_let

n = ng_let()

n.start()

Design

feat: add CLI

We could add CLI support

  • callable with python3 -m nebulagraph_lite.cli, via #21
  • executable under bin should be placed after installation with pip

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.