Comments (5)
Which will be done faster by wdt : one large file or many small files (equal size in total, without network throttling) and why? Thanks!
from wdt.
@wakeupnuts we cut large files into smaller blocks to be sent in parallel so in that sense even a single large file benefits from wdt's multiple flow pipeline
If you have a lot of tiny files (smaller than the blocksize), there is overhead in sending their pathname and opening/reading/writing them, so that's likely to be a bit slower even though it's probably faster than any other way (like using "tar") because we have a really compact and efficient encoding of each header
let me know if that answers your question or if you have any additional ones
from wdt.
@ldemailly Thank you very much for the answer, it's exactly what I need. 😄
I've been searching for a fast data transfer tool for a while, in order to minimize the time of a daily data-transfer task (between a data host and a few receivers, < 10 files, 50G~70G in total) in a production system. Wdt looks very promising. I'd like to give a try, and come back with more questions.
One more thing, is there a way to limit/configure the transfer speed of wdt? Consider wdt's performance, there might be a chance that wdt-task consume total bandwidth and starves other tasks.
from wdt.
Looking forward to more questions and helping you use WDT
Yes WDT includes a Throttler so you don't negatively impact the source or destination machine (or network in between) when they are used for other tasks while transfering. Check the avg_mbytes_per_sec option/flag for instance.
ps: I am traveling / in vacation so will try to not work/reply too much until I am back Aug 10 - but the rest of our team may :)
from wdt.
You can now also ask questions on https://gitter.im/facebook/wdt
from wdt.
Related Issues (20)
- Error in gtest when building HOT 4
- client listen to server for some minutes on wdt HOT 1
- configuration breaks: implicitly converting 'Bool' to 'STRING' type. HOT 1
- build failed on ubuntu20.04 HOT 1
- build wdt_url_test failed HOT 5
- fatal error: 'wdt/Wdt.h' file not found HOT 5
- Any plans to make new release? HOT 6
- Regression – build no longer works (using cmake on Ubuntu 18.04) HOT 2
- Socket timeout issue
- Build instructions on building glog from source are no longer correct
- build failed, help me, please HOT 6
- build failed
- /usr/ports/net/wdt/work/wdt-1.27.1612021-190-g6a122f2/../folly/folly/Conv.h not found
- fatal error: 'wdt/util/FileCreator.h' file not found
- error: incompatible integer to pointer conversion assigning to 'char *' from 'int'
- M1 mac dose not work
- Project status HOT 2
- Build Error : Undefined reference to folly::detail::SimdSplitByCharImplToStrings HOT 3
- Cant transfer data[need help] HOT 4
- CMakeLists.txt Out of Date
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wdt.