droonga / droonga-engine Goto Github PK
View Code? Open in Web Editor NEWDroonga engine is a core component in Droonga system
Home Page: http://droonga.org/
License: Other
Droonga engine is a core component in Droonga system
Home Page: http://droonga.org/
License: Other
メソッド名には「s」がついていて、返される値には「s」がついていない。
どっちかに揃えたほうがよくないかしら。
Currently, Ctrl-C for the droonga-engine-absorb-data (and droonga-engine-join) kills the process on my local machine, but processes to absorb data from the source to the destination node are still working on remote machines. Remote processes should be also killed.
Currently there is no unit test and drntest for "catalog.fetch" command. They have to be done.
With the instruction https://github.com/droonga/presentation-droonga-meetup-1-introduction/blob/master/benchmark/README.md I tried to load all pages to a Droonga cluster, memory usage increased and never decreased. After 1000000+ records were loaded, a process completely ate the memory and the computer became frozen.
Result of top
:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3060 piro 20 0 2844040 2.013g 1.974g S 9.6 26.3 3:48.77 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 23 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-2.pid --+
3062 piro 20 0 691364 131096 99352 S 0.0 1.6 0:08.13 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 25 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-3.pid --+
3056 piro 20 0 674796 103684 72264 S 0.0 1.3 0:05.23 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 19 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-0.pid --+
3058 piro 20 0 662288 70532 38856 S 0.0 0.9 0:01.98 /usr/bin/ruby1.9.1 -S droonga-engine-worker --control-read-fd 13 --control-write-fd 21 --job-queue-socket-path /home/piro/droonga/000/db.3026.14628500.sock --pid-file /home/piro/droonga/000/dbdroonga-worker-1.pid --+
3026 piro 20 0 170320 43632 5016 S 27.2 0.5 14:12.79 /usr/bin/ruby1.9.1 -S droonga-engine-service --listen-fd 15 --heartbeat-fd 16 --control-read-fd 11 --control-write-fd 14 --engine-name 192.168.200.4:10031/droonga
3021 piro 20 0 98328 18996 1696 S 0.0 0.2 0:01.97 /usr/bin/ruby1.9.1 /usr/local/bin/droonga-engine --host=192.168.200.4 --log-file=/home/piro/droonga/droonga-engine.log --daemon --pid-file=/home/piro/droonga/droonga-engine.pid
1092 lightdm 20 0 871820 8172 2816 S 0.0 0.1 0:02.70 /usr/sbin/unity-greeter
3024 piro 20 0 15664 5768 3312 S 0.3 0.1 0:23.42 /home/piro/droonga/serf agent -rpc-addr 192.168.200.4:7373 -node 192.168.200.4:10031/droonga -bind 192.168.200.4:7946 -event-handler droonga-engine-serf-event-handler -log-level warn -retry-join 192.168.200.254 -ret+
1229 lightdm 20 0 510184 3940 1472 S 0.0 0.0 0:00.34 /usr/lib/x86_64-linux-gnu/indicator-keyboard-service --use-gtk
In this result, the topmost item uses 26.3% memory. It increased 5% per one hour on my environment. After I stopped the loading command line (grndump and droonga-send), it didn't decrease but kept the percentage. To release its memory I had to kill the process with kill -KILL PID
.
It's needed for long time running.
It can be implemented with other log rotation software such as logrotate.
Tested on Sharetary.
On my environment bundle exec rake test
successes but it fails on Travis CI...
Diff of dependencies:
--- local-bundle.txt 2015-03-18 14:38:53.906016114 +0900
+++ travis-ci-bundle.txt 2015-03-18 14:37:50.710013918 +0900
@@ -1,6 +1,6 @@
rake 10.4.2
archive-zip 0.7.0
-bundler 1.8.5
+bundler 1.7.6
CFPropertyList 2.2.8
cool.io 1.3.0
drnbench 1.0.5
@@ -9,35 +9,34 @@
droonga-client 0.2.1
droonga-engine 1.1.0
droonga-message-pack-packer 1.0.3
-facter 2.4.1
+facter 2.3.0
faraday 0.9.1
faraday_middleware 0.9.1
-fluent-logger 0.5.0
+fluent-logger 0.4.10
gettext 3.1.6
gqtp 1.0.6
grn2drn 1.0.5
groonga-client 0.1.0
-groonga-command 1.0.8
-groonga-command-parser 1.0.2
+groonga-command 1.1.0
+groonga-command-parser 1.0.4
io-like 0.3.0
json 1.8.2
-kramdown 1.6.0
+kramdown 1.5.0
locale 2.1.0
-msgpack 0.5.11
+msgpack 0.5.10
multipart-post 2.0.0
packnga 1.0.1
pkg-config 1.1.6
-power_assert 0.2.3
+power_assert 0.2.2
rack 1.6.0
rr 1.1.2
-rroonga 4.0.4
+rroonga 4.0.9
sigdump 0.2.2
slop 3.6.0
-sys-proctable 0.9.6
+sys-proctable 0.9.4
test-unit 3.0.9
test-unit-notify 1.0.4
test-unit-rr 1.0.3
text 1.3.0
yajl-ruby 1.2.1
yard 0.8.7.6
-
Known problems in v2 (should be fixed at version 3):
"replicas" appears in a dataset definition directly. It should be placed under "volume".
We cannot put replicas under a slice. (note: actually this can be done on the latest implementation for v2, but it is invalid as a v2 catalog.)
"volume" should be optional. If a slice doesn't have "volume", the slice itself should be parsed as a volume.
Otherwise, we'll see deeply nested JSONs like: "dataset-volume-replicas-slices-slice-volume-replicas-..."
Replicas according to application by use. For example:
For these replicas, search requests should be delivered only for suitable replica for the use case.
Currently, Droonga doesn't ignore "filter:null" and "query:undefined" cases.
They should be unified to a universal install script. Then some command line options will be introduced to switch the version.
たぶん、この定義のテーブルのcolumn_list
で再現できる。(実際に↓で試したわけではない。)
table_create Memos TABLE_PAT_KEY ShortText
table_create Terms TABLE_PAT_KEY ShortText --default_tokenizer TokenBigram
column_create Terms index COLUMN_INDEX|WITH_POSITION Memos _key
GroongaとDroongaで試してみるとわかるはず。
There are no validation of payload. For example, in this code, source_node
needs to be formatted as host:port/tag
.
Currently I have to copy the file via scp
or some way, but it should be shared by droonga-engine automatically. For example. it can be transfered with Serf messages.
vagrant@node2:~/droonga$ droonga-engine-join --host=192.168.100.52 --replica-source-host=192.168.100.50
Joining new replica to the cluster...
{"Acks"=>["192.168.100.52:10031/droonga"], "Responses"=>{}}
{"Acks"=>["192.168.100.52:10031/droonga"], "Responses"=>{}}
/var/lib/gems/1.9.1/gems/droonga-engine-1.0.5/bin/droonga-engine-join:152:in `<top (required)>': undefined method `[]' for nil:NilClass (NoMethodError)
from /usr/local/bin/droonga-engine-join:23:in `load'
from /usr/local/bin/droonga-engine-join:23:in `<main>'
http://droonga.org/ja/tutorial/1.0.6/add-replica/ の「新しいreplicaを追加する」でdroonga-engine-joinした時に起こる。
Because Groonga's result seems to have no such column.
When I start the server with droonga-engine
command, I have to give the name of the node itself via its --host
option. However, it should be resoved automatically by droonga-engine itself for convenience.
When there are two nodes X and Y, and I've done that droonga-engine-join --host=X --replica-source-host=Y
, then they nodes should detect each other as the member of the serf cluster. However, ~/droonga/state/live-nodes.json and serf members --rpc-addr=X:7373
says that there still is only one member -myself- in the cluster. This is possibly a regression.
現状で既に、メッセージ転送のバッファ(転送先ノードが死亡していたらバッファに溜めておいて、復活したらバッファのメッセージを送る)は可能となっている。
あとは、「サービスは生きていて、Droongaのメッセージ転送の仕組みに則ったメッセージ送受信も可能」というノードに対してどうやって「書き込みメッセージをバッファに溜めておき、復活したらバッファのメッセージを送る」をどのようにして実現するかが問題である。
プラン1:現状のバッファ機構(BufferedTCPSocket)を使う。
プラン2:DispatcherとForwarderの間に新たにバッファ機構を挟み込む。
TABLE_NO_KEY
なテーブルは _key
がないのに、 *
が _key
も展開しちゃっている感じ。
By c84abd0 new "catalog" plugin is added. It must be loaded by default.
Node.jsではそのような機能を実現するメソッドとして cork/uncork という名前を使ったっぽい。
http://nodejs.jp/nodejs.org_ja/api/stream.html#stream_writable_cork
Currently we have to run droonga-engine
command directly, with many options. We don't want to do such a painful thing on every startup.
2014-07-29T05:50:02+00:00[3950][info]: engine: catalog loaded path="/home/vagrant/droonga/catalog.json" mtime=2014-07-29 05:50:01 +0000
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: failed to run services: Errno::EBADF: Bad file descriptor
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:110:in `close'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:110:in `close'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:134:in `rescue in on_readable'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:127:in `on_readable'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/io.rb:191:in `on_readable'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run_once'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/cool.io-1.2.4/lib/cool.io/loop.rb:88:in `run'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/lib/droonga/command/droonga_engine_service.rb:119:in `run_services'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/lib/droonga/command/droonga_engine_service.rb:55:in `run'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/lib/droonga/command/droonga_engine_service.rb:31:in `run'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /var/lib/gems/1.9.1/gems/droonga-engine-1.0.4/bin/droonga-engine-service:20:in `<top (required)>'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /usr/local/bin/droonga-engine-service:23:in `load'
2014-07-29T05:51:11+00:00[3931][error]: droonga-engine-service: /usr/local/bin/droonga-engine-service:23:in `<main>'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.