Comments (2)
I dig a little bit into the code base and find that request concurrency
and # of capacity units
seem to be treated as the same thing here.
https://github.com/markitx/dynamo-backup-to-s3/blob/master/lib/dynamo-restore.js#L303
However based on Dynamo docs they are different.
One write capacity unit represents one write per second for an item up to 1 KB in size. If you need to write an item that is larger than 1 KB, DynamoDB will need to consume additional write capacity units. The total number of write capacity units required depends on the item size.
What if some of individual items in my Dynamo are larger than 1KB? Then writing each of them would consume more than 1 write capacity unit, although it just counts for 1 request.
It could explain why my case consumed much more write capacity units than 200.
from dynamo-backup-to-s3.
Find something deeper in the code base.
https://github.com/markitx/dynamo-backup-to-s3/blob/master/lib/dynamo-restore.js#L172
It actually bundles 25 items together and then make 1 request using AWS API dynamo.batchWriteItem
https://github.com/markitx/dynamo-backup-to-s3/blob/master/lib/dynamo-restore.js#L299
This will well explain why request body size can easily exceed 1KB. On average one item in our DynamoDB should be easily over 1024/25 bytes
from dynamo-backup-to-s3.
Related Issues (20)
- Missing credentials in config for lots of parallel requests HOT 1
- Find new owner HOT 5
- Read credentials and config from the shared .ini file
- maximum backup copies in s3 HOT 2
- --partitionkey necessary, even when using --overwrite on restore
- Setting LIMIT on Dynamo scan doesn't give expected read throughput. HOT 2
- cannot read property of length undefined
- Script runs forever... HOT 2
- Restore did not work because of outdated/incorrect documentation HOT 1
- LSI and GSI
- Unexpected end of input - restorartion HOT 1
- undefined this.batches in restore
- Option save-datapipeline-format not working
- restore db tables : batch multi thread issue
- Resorting a Global Table does not replicate
- Overwrite flag not being honored HOT 1
- large data restore miss HOT 1
- TypeError: Cannot read property push of undefined on dynamo-restore-from-s3 HOT 1
- Remove throughput limit for on-demand tables HOT 3
- Working with on-demand provisioning tables
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dynamo-backup-to-s3.