Comments (4)
Hi,
Thanks for your interest.
-
The code for that test is not included. I re-coded a version of the Btree that uses mmap just to test performance, but it was really hacky/messy, so I did not release the code.
-
Two things there:
- The "Item is too big" is because you try to add an item that is > 4KB. This is not implemented in the released code
- The "present twice" is just a warning. It may happen if you exit before all operations complete (i.e., before all callbacks have been called). It may also happen if you update the same item, changing its size.
E.g.:
add(k1, 1kb value)
update(k1, 2kb value) <- the DB will persist the 2kb value, and place the old location in a "free list", but the old location is not cleared on disk. This is ok. The old value will be ignored when doing the recovery (and print the warning... which in that case is not very accurate :/ )
From the error, it seems that you are trying to grow items:
for(size_t size = ...; size < ...; size++)
update(k, value of size X);
And in that case, the warning/errors you see are expected.
Btw, in that situation, the store will grow because I did not implement the shrinking of slabs on disks. This would be relatively easy/inexpensive to do but is not in the code.
Something like:
./db_bench --benchmarks=$benchmark --threads=$threads --key_size=8 --value_size=1024 --bloom_bits=10 --cache_numshardbits=4 --open_files=500000 --verify_checksum=0 --db=$dbpath --sync=0 --compression_type=none --compression_ratio=0 --write_buffer_size=54217728 --target_file_size_base=67108864 --max_write_buffer_number=2 --max_background_compactions=10 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=8 --level0_stop_writes_trigger=10 --num_levels=5 --delete_obsolete_files_period_micros=300000000 --stats_per_interval=1 --max_bytes_for_level_base=268435456 --use_existing_db=1 --duration=$benchmark_duration_sec --num=$num_elements_db
The key is to have a large number of "$threads" to make sure to reach peak CPU usage. But the observed BW will be highly dependent on the machine (the more CPUs you have, the higher the BW). See #1 for the YCSB implementation in rocksdb.
from kvell.
Thanks for your detailed answers. I've got it. :)
May I have a copy of your code containing mmap? I'd like to test it.
from kvell.
I don't seem to have a copy of it. If I remember correctly, I changed the way the mmap was called in the tcmalloc library (tcmalloc uses mmap to get memory from the kernel). Just a quick and dirty test :)
from kvell.
Ok, it doesn't matter, thank you very much!
from kvell.
Related Issues (11)
- some questions about paper HOT 9
- There are some things I don't understand in the code interface reading HOT 2
- About the YCSB+T benchmark HOT 5
- some questions about the implementation HOT 4
- question about kvell latency HOT 2
- Some questions about the transaction support HOT 1
- questions about remote KVell HOT 1
- ./autogen.sh
- down scaled Kvell HOT 2
- strange behaviors in kvell HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kvell.