droonga / droonga-engine Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 5.0 3.19 MB

Droonga engine is a core component in Droonga system

Home Page: http://droonga.org/

License: Other

Emacs Lisp 0.01% Ruby 98.05% Shell 1.11% JavaScript 0.83%

droonga-engine's People

Contributors

Stargazers

Watchers

Forkers

myokoym long-long-float kshoji kitaitimakoto

droonga-engine's Issues

Validation of payload is required

There are no validation of payload. For example, in this code, source_node needs to be formatted as host:port/tag.

select's offset seems not work

Tested on Sharetary.

Don't list "_key" to the result of column_list

Because Groonga's result seems to have no such column.

メモ：書き込みを貯めておくモードを切り替えるメソッド名（cork/uncork）

Node.jsではそのような機能を実現するメソッドとして cork/uncork という名前を使ったっぽい。

http://nodejs.jp/nodejs.org_ja/api/stream.html#stream_writable_cork

Add "catalog" plugin to the list of default plugins

By c84abd0 new "catalog" plugin is added. It must be loaded by default.

droonga-engine should be started/stopped via standard way (ex. `service`, `sv`)

Currently we have to run droonga-engine command directly, with many options. We don't want to do such a painful thing on every startup.

Sometimes fails to start droonga-engine-server after modifying catalog.json

2014-07-29T05:50:02+00:00[3950][info]: engine: catalog loaded path="/home/vagrant/droonga/catalog.json" mtime=2014-07-29 05:50:01 +0000
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: failed to run services: Errno::EBADF: Bad file descriptor
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:110:in `close'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:110:in `close'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:134:in `rescue in on_readable'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:127:in `on_readable'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:191:in `on_readable'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/lib/droonga/command/droonga_engine_service.rb:119:in `run_services'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/lib/droonga/command/droonga_engine_service.rb:55:in `run'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/lib/droonga/command/droonga_engine_service.rb:31:in `run'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/bin/droonga-engine-service:20:in `<top (required)>'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /usr/local/bin/droonga-engine-service:23:in `load'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /usr/local/bin/droonga-engine-service:23:in `<main>'

Add tests for "catalog" plugin

Currently there is no unit test and drntest for "catalog.fetch" command. They have to be done.

Draw up catalog.json format version 3

Known problems in v2 (should be fixed at version 3):

"replicas" appears in a dataset definition directly. It should be placed under "volume".
We cannot put replicas under a slice. (note: actually this can be done on the latest implementation for v2, but it is invalid as a v2 catalog.)
"volume" should be optional. If a slice doesn't have "volume", the slice itself should be parsed as a volume.
Otherwise, we'll see deeply nested JSONs like: "dataset-volume-replicas-slices-slice-volume-replicas-..."
Replicas according to application by use. For example:
- Replica 1: There is only one slice. This should be used for generic search.
- Replica 2: There are time-sliced slices. This should be used for search with year.
- Replica 3: There are category-sliced slices. This should be used for search with category.
For these replicas, search requests should be delivered only for suitable replica for the use case.

undefined method `[]' for nil:NilClass (NoMethodError) error while joining

 vagrant@node2:~/droonga$ droonga-engine-join --host=192.168.100.52 --replica-source-host=192.168.100.50
Joining new replica to the cluster...
{"Acks"=>["192.168.100.52:10031/droonga"], "Responses"=>{}}
{"Acks"=>["192.168.100.52:10031/droonga"], "Responses"=>{}}
/var/lib/gems/1.9.1/gems/droonga-engine-1.0.5/bin/droonga-engine-join:152:in `<top (required)>': undefined method `[]' for nil:NilClass (NoMethodError)
    from /usr/local/bin/droonga-engine-join:23:in `load'
    from /usr/local/bin/droonga-engine-join:23:in `<main>'

http://droonga.org/ja/tutorial/1.0.6/add-replica/ の「新しいreplicaを追加する」でdroonga-engine-joinした時に起こる。

Host name of the node itself should be resolved automatically, on the startup

When I start the server with droonga-engine command, I have to give the name of the node itself via its --host option. However, it should be resoved automatically by droonga-engine itself for convenience.

Droonga::Path.databasesのbasenameがdatabaseでもやっとする

メソッド名には「s」がついていて、返される値には「s」がついていない。

どっちかに揃えたほうがよくないかしら。

Support log rotation

It's needed for long time running.

It can be implemented with other log rotation software such as logrotate.

TABLE_NO_KEYなテーブルにselect --output_columns *するとエラーになる

TABLE_NO_KEY なテーブルは _key がないのに、 * が _key も展開しちゃっている感じ。

_keyをsourceにしたときのcolumn_listのsourceの値がGroonga互換じゃない

たぶん、この定義のテーブルのcolumn_listで再現できる。（実際に↓で試したわけではない。）

table_create Memos TABLE_PAT_KEY ShortText
table_create Terms TABLE_PAT_KEY ShortText --default_tokenizer TokenBigram
column_create Terms index COLUMN_INDEX|WITH_POSITION Memos _key

GroongaとDroongaで試してみるとわかるはず。

書き込みメッセージを停止しないでdroonga-engine-joinできるようにする

現状で既に、メッセージ転送のバッファ（転送先ノードが死亡していたらバッファに溜めておいて、復活したらバッファのメッセージを送る）は可能となっている。
あとは、「サービスは生きていて、Droongaのメッセージ転送の仕組みに則ったメッセージ送受信も可能」というノードに対してどうやって「書き込みメッセージをバッファに溜めておき、復活したらバッファのメッセージを送る」をどのようにして実現するかが問題である。

プラン1：現状のバッファ機構（BufferedTCPSocket）を使う。

概要：
- DROONGA_BASE_DIR/state/status.json あたりを使って「データ移行中」のようなフラグを保持できるようにし、データ移行中のdroonga-engine（送り出し側、受け側両方とも）は、droonga-engine.yamlで決められているポート番号とは違うポートをlistenする・メッセージの宛先に使うようにする。
メリット：
- 送り出し側にオーバーヘッドが生じない。
デメリット：
- クラスタ構成変更時に、起動ポートの変更、catalog.jsonの更新など、コピー元になるノードとコピー先になるノード（クラスタ）の全ノードが一斉に自身の状態の変更と再起動を繰り返すことになる。

プラン2：DispatcherとForwarderの間に新たにバッファ機構を挟み込む。

概要：
メリット：
- 変更箇所がDispatcherとForwarderの間だけにとどまる。
デメリット：
- 送り出し側にオーバーヘッドが生じる。

Unify install.sh and install-master.sh

They should be unified to a universal install script. Then some command line options will be introduced to switch the version.

Node doesn't join to the serf cluster when it is joined by "droonga-engine-join"

When there are two nodes X and Y, and I've done that droonga-engine-join --host=X --replica-source-host=Y, then they nodes should detect each other as the member of the serf cluster. However, ~/droonga/state/live-nodes.json and serf members --rpc-addr=X:7373 says that there still is only one member -myself- in the cluster. This is possibly a regression.

Cannot load too much records in one time

With the instruction https://github.com/droonga/presentation-droonga-meetup-1-introduction/blob/master/benchmark/README.md I tried to load all pages to a Droonga cluster, memory usage increased and never decreased. After 1000000+ records were loaded, a process completely ate the memory and the computer became frozen.

Result of top:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3060 piro      20   0 2844040 2.013g 1.974g S   9.6 26.3   3:48.77 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 23 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-2.pid --+
 3062 piro      20   0  691364 131096  99352 S   0.0  1.6   0:08.13 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 25 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-3.pid --+
 3056 piro      20   0  674796 103684  72264 S   0.0  1.3   0:05.23 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 19 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-0.pid --+
 3058 piro      20   0  662288  70532  38856 S   0.0  0.9   0:01.98 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 21 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-1.pid --+
 3026 piro      20   0  170320  43632   5016 S  27.2  0.5  14:12.79 /usr/bin/ruby1.9.1 -S droonga-engine-service --listen-fd 15 --heartbeat-fd 16 --control-read-fd 11 --control-write-fd 14 --engine-name 192.168.200.4:10031/droonga
 3021 piro      20   0   98328  18996   1696 S   0.0  0.2   0:01.97 /usr/bin/ruby1.9.1 /usr/local/bin/droonga-engine --host=192.168.200.4 --log-file=/home/piro/droonga/droonga-engine.log --daemon --pid-file=/home/piro/droonga/droonga-engine.pid
 1092 lightdm   20   0  871820   8172   2816 S   0.0  0.1   0:02.70 /usr/sbin/unity-greeter
 3024 piro      20   0   15664   5768   3312 S   0.3  0.1   0:23.42 /home/piro/droonga/serf agent -rpc-addr 192.168.200.4:7373 -node 192.168.200.4:10031/droonga -bind 192.168.200.4:7946 -event-handler droonga-engine-serf-event-handler -log-level warn -retry-join 192.168.200.254 -ret+
 1229 lightdm   20   0  510184   3940   1472 S   0.0  0.0   0:00.34 /usr/lib/x86_64-linux-gnu/indicator-keyboard-service --use-gtk

In this result, the topmost item uses 26.3% memory. It increased 5% per one hour on my environment. After I stopped the loading command line (grndump and droonga-send), it didn't decrease but kept the percentage. To release its memory I had to kill the process with kill -KILL PID.

Make droonga-engine-absorb-data (and droonga-engine-join) cancelable

Currently, Ctrl-C for the droonga-engine-absorb-data (and droonga-engine-join) kills the process on my local machine, but processes to absorb data from the source to the destination node are still working on remote machines. Remote processes should be also killed.

--- local-bundle.txt    2015-03-18 14:38:53.906016114 +0900
+++ travis-ci-bundle.txt    2015-03-18 14:37:50.710013918 +0900
@@ -1,6 +1,6 @@
 rake 10.4.2
 archive-zip 0.7.0
-bundler 1.8.5
+bundler 1.7.6
 CFPropertyList 2.2.8
 cool.io 1.3.0
 drnbench 1.0.5
@@ -9,35 +9,34 @@
 droonga-client 0.2.1
 droonga-engine 1.1.0
 droonga-message-pack-packer 1.0.3
-facter 2.4.1
+facter 2.3.0
 faraday 0.9.1
 faraday_middleware 0.9.1
-fluent-logger 0.5.0
+fluent-logger 0.4.10
 gettext 3.1.6
 gqtp 1.0.6
 grn2drn 1.0.5
 groonga-client 0.1.0
-groonga-command 1.0.8
-groonga-command-parser 1.0.2
+groonga-command 1.1.0
+groonga-command-parser 1.0.4
 io-like 0.3.0
 json 1.8.2
-kramdown 1.6.0
+kramdown 1.5.0
 locale 2.1.0
-msgpack 0.5.11
+msgpack 0.5.10
 multipart-post 2.0.0
 packnga 1.0.1
 pkg-config 1.1.6
-power_assert 0.2.3
+power_assert 0.2.2
 rack 1.6.0
 rr 1.1.2
-rroonga 4.0.4
+rroonga 4.0.9
 sigdump 0.2.2
 slop 3.6.0
-sys-proctable 0.9.6
+sys-proctable 0.9.4
 test-unit 3.0.9
 test-unit-notify 1.0.4
 test-unit-rr 1.0.3
 text 1.3.0
 yajl-ruby 1.2.1
 yard 0.8.7.6
-