Performance of Happybase about happybase

[wbolster]

The Batch.send() method is just the one that actually sends the data to the Thrift server, hence it takes the most time (but it is I/O bound). With your approach all ingestion into HBase happens from a single Thrift server, which in turn has to communicate with all the region servers to actually store the data (which in turn trigger HDFS replication traffic). This is obviously slower than ingesting bulk data from within the cluster itself, as is the case with the MapReduce job, since in that case the work will be actually distributed over the cluster.

Have you tried running multiple Thrift servers and using multiple connections, i.e. multiple HappyBase Connection instances to different remote hosts? Of course the splitting/chunking would move to your Python code in that case. The multiprocessing module (especially .imap_unordered() with an appropriate chunk size) may help you here.

[arapat]

Thank you very much for your help. Your suggestion works. It performs much better and takes ~ 15 minutes to write all the data with multiple Thrift servers.

Another optimization we can try is to pre-create the regions (reference: http://hbase.apache.org/book/perf.writing.html). I didn't find any API in happybase for creating regions, so I used HBase shell instead. Then it only takes less than 4 minutes to write all the data.

Your work makes our life much easier. Thanks again.

[wbolster]

Great, glad to hear. As far as I know there is no way to create regions using the Thrift API, so it seems you'll have to use the HBase shell for that.