Coder Social home page Coder Social logo

dolphindbplugin's Introduction

DolphinDB Plugin

DolphinDB plugins currently have release200, release130, release120 and release110. Each plugin version corresponds to a DolphinDB server version. Please refer to the corresponding branch according to the server version you are using.

DolphinDB database supports dynamic loading of external plugins to extend system functionality. The plug-in can be written in C++, and it needs to be compiled into ".so" shared libraries or ".dll" shared library files.

1. Directory Structures

Please switch to the corresponding branch to view the following listed directories for each plugin. The master branch doesn’t contain these directories.

  • includeThe directory contains the class declarations and some tool class declarations for the core data structures of DolphinDB. These classes are important basic tools for implementing plug-ins.
  • demoThe directory contains a demo plug-in implementation.
  • Other directories contain the implementation of the plugins.

2. Load Plugins

2.1 Load via function loadPlugin

Use the loadPlugin function to load external plugins. This function accepts a file path, which describes the format of the plugin, for example:

loadPlugin("/YOUR_SERVER_PATH/plugins/odbc/PluginODBC.txt"); 

2.2 Load through configuration parameter preloadModules

Starting from server version 1.20.0, plugins can be automatically loaded through the configuration parameter preloadModules. Please note that the plugin that needs to be preloaded must exist. Otherwise, there will be an exception when the server starts. Multiple plugins are separated by ',', for example:

preloadModules=plugins::mysql,plugins::odbc

3. DolphinDB Plugin Format

DolphinDB uses a text file to describe the plugin. The file format is as follows: The first line describes the plugin name and shared library file name. Each of the following lines describes the mapping between a shared library function and the DolphinDB function.

module name, lib file
function name in lib, function name in DolphinDB, function type, minParamCount, maxParamCount, isAggregate
...

Explanation

  • module name: plugin module name
  • lib file: shared library file name
  • function name in lib: the function name in the shared library
  • function name in DolphinDB: corresponding function name in DolphinDB
  • function type: operator or system
  • minParamCount: the minimum number of parameters
  • maxParamCount: the maximum number of parameters
  • isAggregate: whether it is an aggregate function

4. Example

PluginDemo.txt:

demo,libPluginDemo.so 
minmax,minmax,operator,1,1,0
foo,foo,system,1,1,0

The description file above defines a plugin named demo. The shared library file is named libPluginDemo.so.

The plugin exports two functions. The first function is named minmax. The name of the function is also minmax in DolphinDB. The function type is "operator" and accepts one parameter. The second function name is echo, the name in DolphinDB is also echo, the function type is "system" and accepts one argument.

After finishing the description file, you can start writing plugins. For content, please refer to demo folder contents.

The compiler needs to use DolphinDB's core library libDolphinDB.so, which implements the classes declared in include directories.

cd demo
g++ -DLINUX -fPIC -DLOCKFREE_SYMBASE -c src/Demo.cpp -I../include -o Demo.o
g++ -fPIC -shared -o libPluginDemo.so Demo.o -lDolphinDB

Note:

  • To be compatible with older compilers, you can compile libDolphinDB.so with option -D_GLIBCXX_USE_CXX11_ABI=0. If the server is the latest version of ABI=1, it is not necessary to add the option.
  • For the path specified by compiling option -L, you can modify it to the path of DolphinDB dynamic library.

After successful compilation, a shared library file named libPluginDemo.so will be generated under the directory.

Enter the following command in the DolphinDB console to load the plugin and use it.

>loadPlugin(Path to PluginDemo.txt); // Load the plugin
(minmax,echo)
>use demo; // Import the plugin's namespace
>demo::minmax([12,3,4]); // You can also use minmax([12,3,4])
[3,12]
>demo::echo("foo");
foo
>echo(1);
1

5. Tips

  • We recommend that you use the command ld to check if the compiler link is successful and if there are undefined references in the so. If the command ld generates an error, then DolphinDB can not load the plug-in correctly.
  • If the program crashes after loading the plugin, try the following steps:
    1. Make sure that the include headers are consistent with the libDolphinDB.so implementation.
    2. Make sure that the version of gcc used to compile the plugin is consistent with the version of libBoardDB.so in order to avoid the incompatibilities between different versions of the compiler ABI.
    3. The plugin and DolphinDB server run in the same process. If a plugin crashes, then the whole system will crash. Therefore, efforts should be made to improve the error detection mechanism. Only the thread where the plugin is located can throw an exception which is captured by the server. Except for the thread, all threads cannot throw exceptions and they must capture exceptions by themselves.
    4. Please make sure LOCKFREE_SYMBASE has been added.
  • If an undefined reference occurs when compiling the plugin and the error message contains std::__cxx11, please check the gcc version used to compile the plugin (gcc 6.2.0 with --disable-libstdcxx-dual-abi).

dolphindbplugin's People

Contributors

ccccccicada avatar dolphindbit avatar dzhou284 avatar hzwale avatar jasonyuchen avatar jiajiaxu123 avatar windancer79 avatar xiaodanzhang avatar yunangao123 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dolphindbplugin's Issues

Wanted: who's using DolphinDBPlugin

Who is Using DolphinDBPlugin

First of all, thanks sincerely for constantly using and supporting DolphinDB. We will try our best to keep DolphinDB better, and keep growing community and eco-system.

The purpose of this issue

  • We want to attract more people to contribute to DolphinDBPlugin.
  • We’d like to listen to the community to make DolphinDBPlugin better.
  • We're willing to learn more DolphinDBPlugin use scenarios in the field for better planning.

What we expect from you

Please, submit a comment in this issue to include the following infomation:

  • your company, school or organization
  • your contact info: email
  • for what business scenario do you use DolphinDBPlugin

kdb+ Plugin Should Support Splayed Tables with Multiple sym Lists.

Is your feature request related to a problem? Please describe.

The current plug-in assumes a universal sym list for all enumerated columns in a splayed table. This assumption will break on complicated schemas where each sym column in a splayed table may be enumerated by a different sym list.

Describe the solution you'd like

A new, enhanced API is needed to support multiple sym lists on a single splayed table.

Submit a PR

  • Yes I am willing to submit a PR!

[New Feature] Development List TO DO

To do list for new Contributors

Contributions are welcome, here is a list for new contributors:

No Plugin Content Priority Difficulty Contributer Status Issue/PR
1 clickhouse   low medium      
2 hdf5 add docs about the perfect way of loading data and use cases. medium low      
3 httpClient support for windows high medium      
4 kafka support for kerberos low medium      
5 kafka support for windows high medium      
6 kdb support for using loadTextEx medium medium      
7 kdb support for reading  compressed data of type q IPC (1), snappy (3), L4Z (4) high high      
8 mqtt support for arm low medium      
9 mqtt publish stream engine calculation results directly to the mqtt server medium medium      
10 mysql add docs about return type medium low      
11 mysql support for type blob high high      
12 mysql support for type decimal high high      
13 oracle   high high      
14 parquet support for reading column of  type repeated high high      
15 parquet support for windows high medium      
16 signal support for rfft, irfft high high      
17 xgboost upgrade the library of xgboost to version 1.7 medium medium      
18 zip support for arm low medium      
19 zip support for compressing folders high medium      

Use CMake's FindPackage module to import zlib

CMake's FindPackage module has been supporting ZLIB import since version 3.1. It's better to push CMake's version requirement from 3.0 to 3.1 and use FindPackage to import zlib instead.

load plugin error

dolphindb: dolphindb/dolphindb:v2.00.7
plugin: downloaded from release200 branch

hbase plugin: successfully loaded (2nd time load notification: The module [hbase] is already in use.)

parquet plugin: Couldn't load the dynamic library [/dolphindb/etc/libPluginParquet.so]: /dolphindb/etc/libPluginParquet.so: undefined symbol: _ZN6Domain8addTableERKSsS1_RSt6vectorI10ColumnDescSaIS3_EERS2_IiSaIiEE

xgboost plugin: Couldn't load the dynamic library [/dolphindb/etc/libPluginXgboost.so]: libgomp.so.1: cannot open shared object file: No such file or directory

windows下如何引入hdf5插件

按照教程中的代码无法引入,显示

Error Message: Syntax Error: [line #3] Can't find module [hdf5]

代码是

loadPlugin("plugins/hdf5/PluginHDF5.txt")

use hdf5

Server will breakdown if call feather::save function by peach

data to be saved:
A dict which key is a date, value is a table.

code:
saveOneDay = def(data_dict, save_path, d){
file_path = save_path + '/' + format(d,'yyyy-MM-dd.pt')
&tb = data_dict[d]
feather::save(tb,file_path)
// parquet::saveParquet(tb,file_path)
return file_path
}
dates = data_dict.keys()
peach(saveOneDay{data_dict, save_path}, dates)

gdb log:
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f6756510771 in arrow::internal::ChunkResolver::ChunkResolver(std::vector<std::shared_ptrarrow::Array, std::allocator<std::shared_ptrarrow::Array > > const&) () from /opt/dolphindb/server/plugins/feather/libPluginFeather.so

if we use parquet::saveParquet to replace feather::save, the code works will.

插件编译问题

是否能把插件编译后的版本(windows和linux)进行打包发布?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.