Some questions about evaluation about kvell

[BLepers]

Hi,

Thanks for your interest.

1. The code for that test is not included. I re-coded a version of the Btree that uses mmap just to test performance, but it was really hacky/messy, so I did not release the code.

2. Two things there:

• The "Item is too big" is because you try to add an item that is > 4KB. This is not implemented in the released code
• The "present twice" is just a warning. It may happen if you exit before all operations complete (i.e., before all callbacks have been called). It may also happen if you update the same item, changing its size.
  E.g.:
  add(k1, 1kb value)
  update(k1, 2kb value) <- the DB will persist the 2kb value, and place the old location in a "free list", but the old location is not cleared on disk. This is ok. The old value will be ignored when doing the recovery (and print the warning... which in that case is not very accurate :/ )

From the error, it seems that you are trying to grow items:

for(size_t size = ...; size < ...; size++)
    update(k, value of size X);

And in that case, the warning/errors you see are expected.

Btw, in that situation, the store will grow because I did not implement the shrinking of slabs on disks. This would be relatively easy/inexpensive to do but is not in the code.

Something like:

./db_bench --benchmarks=$benchmark  --threads=$threads --key_size=8 --value_size=1024 --bloom_bits=10 --cache_numshardbits=4 --open_files=500000 --verify_checksum=0  --db=$dbpath --sync=0 --compression_type=none --compression_ratio=0  --write_buffer_size=54217728 --target_file_size_base=67108864 --max_write_buffer_number=2 --max_background_compactions=10 --level0_file_num_compaction_trigger=4 --level0_slowdown_writes_trigger=8 --level0_stop_writes_trigger=10 --num_levels=5 --delete_obsolete_files_period_micros=300000000 --stats_per_interval=1 --max_bytes_for_level_base=268435456 --use_existing_db=1 --duration=$benchmark_duration_sec --num=$num_elements_db

The key is to have a large number of "$threads" to make sure to reach peak CPU usage. But the observed BW will be highly dependent on the machine (the more CPUs you have, the higher the BW). See #1 for the YCSB implementation in rocksdb.

[CheneyDing]

Thanks for your detailed answers. I've got it. :)

May I have a copy of your code containing mmap? I'd like to test it.

[BLepers]

I don't seem to have a copy of it. If I remember correctly, I changed the way the mmap was called in the tcmalloc library (tcmalloc uses mmap to get memory from the kernel). Just a quick and dirty test :)

[CheneyDing]

Ok, it doesn't matter, thank you very much!