Comments (3)
As @evanmcc pointed out in another forum, these files are cleaned out when the bitcask is reopened (e.g. on Riak restart), but their presence can slow down merge calculations if there are lots of them relative to non-empty casks.
from bitcask.
I think the major expense here is actually in bitcask_fileops:create_file/2
, and also perhaps fold and merge opens, since it'll list the entire directory when it tries to increment the counter and/or when it tries to get a fileset for the merge. I've added some logging to try and come to grips with the race, maybe I'll have some ideas soon.
from bitcask.
NB. that you need to do -size 18c
to see empty hintfiles.
What looks to me to be happening here is that this pathological case creates only live keys in a 1 or 2 data files at a time, so your merge set gets queued, and then not a single key in them is put into a data file because all of them have been overwritten.
I am not sure that there is a FD leak here. I am not certain how often this case would happen to customers, but I think that it's worth handling to at least make fixes for issues like #114 easier to test (PR incoming tomorrow or tuesday, BTW). My initial idea would just be to lazily create the write file on the first write, then if we never write we never create the file.
from bitcask.
Related Issues (20)
- A bitcask file can get merged repeatedly [JIRA: RIAK-1844] HOT 8
- bitcask NIF doesn't compile cleanly on OS X 10.10 [JIRA: RIAK-1574] HOT 2
- Startup crash trying to read key from corrupted data file [JIRA: RIAK-1819] HOT 5
- Repeated crashes during merge operations [JIRA: RIAK-1903] HOT 4
- Can't manually force a merge, error - efile_driver_unavailable [JIRA: RIAK-2036] HOT 5
- Include Bitcask data file count in merge info messages. [JIRA: RIAK-2039]
- Potential Process Leak [JIRA: RIAK-2227] HOT 2
- Lock file staleness check could be more robust [JIRA: RIAK-2230]
- question about read concurrency [JIRA: RIAK-2330] HOT 1
- Operation aborts if any file contains a corrupt key [JIRA: RIAK-2371] HOT 1
- rebar abort [JIRA: RIAK-2485] HOT 5
- Errors in `merge_single_tombstone` and `open_file` `append` mode [JIRA: RIAK-2733] HOT 2
- bitcask:merge/1 keeps file descriptors of deleted files open [JIRA: RIAK-2814] HOT 2
- File Open behaves differently between NIF and Erlang modes [JIRA: RIAK-2895] HOT 1
- bitcask failed to merge.merge_locked,locked
- c_src/bitcask_nifs.c: pair of bad % specifiers ?
- Build broken with OTP 21: use of the deprecated get_stacktrace/0 in dependency meck HOT 3
- Call bitcask_merge_worker:merge(DirName) is error
- How to run YCSB workload for bitcask?
- In the bitcask intro paper we see it being mentioned that bitcask is able to support datasets larger than 10 X system memory. I am trying to understand how? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bitcask.