compression support (in the wdt library itself and not just wcp) about wdt

This is actually a really interesting problem. Compression often is a bottleneck, but sometimes it's the network in the end.

I think the ideal solution is dynamic compression, most likely with lzo since it's normally not CPU bound. You can vary the rate by adjusting compression until the input read speed is highest (since input read rates effectively slow down as output gets slowed down). That might be no compression or max compression.

I haven't seen any utilities that do this, but it should be... not easy but relatively straight forward. And should work without caveats.

[nikunjy]

Hey, compression is on the things we want to do. We have an ongoing effort (but not high pri yet), but you will see it soon in WDT. In the time being:

1. We use gzip in wcp (https://github.com/facebook/wdt/blob/master/wcp.sh#L87)
2. In WDT we have done some testing using folly compression https://github.com/facebook/folly/blob/master/folly/io/Compression.h#L56
   Look at Snappy and LZ4_VARIANT

Please feel free to share any experience you have had with respect to compression and tell us more about what are you using WDT with.

[junhuihuang]

Thanks for your reply. I am testing WDT for transferring log file between two machines, use disk, but the speed is not so good, about 18Mbytes/sec. If adding compression and write data using pwrite function, i think the throughput could be higher.

[uddipta]

Try with -block_size_mbytes=-1 and -num_ports=1.
In case of disk, just one writer thread generally gives best throughput. We have multiple threads to optimize for flash.

Also you can use -enable_perf_stat_collection to find out which system calls are taking more time.

Let us know if that improves throughput or not.