whiplash5057 / elasticsearch-document-transfer

Simple Node.js script based on producer-consumer problem. For transfering documents between two elasticsearch servers or between indices of the same elasticsearch server.

elasticsearch-document-transfer

Simple Node.js script to transfer documents between two elasticsearch servers or between indices of the same elasticsearch server.

Implementation based on consumer-producer problem.
Since Node.js is single-threaded, producer and consumer cannot simultaneously access the buffer. And hence, as long as the asynchronous nature of Node.js is handled properly, there shouldn't be any issues related to concurrency.

Usage:

1. Clone the repo.
2. Run npm install in the directory.
3. Add config.js using config-sample.js.
   Exercise caution while setting source and target elastic hosts (and while editing this file in general).
4. (Optional) Set appropriate values in options.js
5. Run the script

   node index.js
   

Note:

If options.consume.byFile is set to true, the documents will be written to a file in a format as expected by the bulk API of elasticsearch. The path of file must be specified in config.js.

If options.consume.byElastic is set to true, the documents will be directly added to the target elastic server. The connection details should be present in config.js.