Comments (4)
One of the causes of bursts of activity is the Riak KV backend API status()
call, which in turn calls bitcask:status()
. summary_info()
is filtering out files from the keydir that no longer exist, but it has no mechanism in this function to feed back missing files to the keydir.
Line 830 in 3fac8e9
Note that needs_merge()
, which is also a known latency outlier cause, is also calling summary_info()
.
Line 653 in 3fac8e9
from bitcask.
Hrm, this appears to be long-time bitcask behavior, that there is no way for the merge process to tell the keydir that a file has been deleted, so its fstats
entry is kept forever? And that
Line 830 in 3fac8e9
With the patch below:
diff --git a/src/bitcask.erl b/src/bitcask.erl
index aa16822..216e226 100644
--- a/src/bitcask.erl
+++ b/src/bitcask.erl
@@ -787,6 +787,16 @@ summary_info(Ref) ->
%% and is only an estimate/snapshot.
{KeyCount, _KeyBytes, Fstats, _IterStatus} =
bitcask_nifs:keydir_info(State#bc_state.keydir),
+try
+ %%{QQQa, QQQb, QQQc} = now(),
+ %%QQQpath = "/tmp/" ++ filename:basename(State#bc_state.dirname) ++ "." ++ lists:flatten(io_lib:format("~p.~p.~p", [QQQa, QQQb, QQQc])),
+ QQQpath = "/tmp/" ++ filename:basename(State#bc_state.dirname),
+ {ok, QQQfh} = file:open(QQQpath, [write]),
+ [io:format(QQQfh, "~p\n", [QQQxxx]) || QQQxxx <- [KeyCount, _KeyBytes, Fstats, _IterStatus]],
+ file:close(QQQfh)
+catch __X:__Y ->
+error_logger:warning_msg("DEBUG exception at ~p ~p: ~p ~p\n", [?MODULE, ?LINE, __X, __Y])
+end,
%% We want to ignore the file currently being written when
%% considering status!
... I see this for one vnode (but all show the same pattern) in a Riak workload
that is constantly clobbering a small number of keys:
440
9240
[{84,105,127,159471,192877,1382691404,1382691406},
{83,18,24,27336,36448,1382691341,1382691380},
{82,242,567,367518,861120,1382691393,1382691404},
{81,75,690,113909,1047940,1382691380,1382691393},
{80,0,690,0,1047926,1382691368,1382691380},
{79,0,690,0,1047950,1382691356,1382691368},
{78,0,690,0,1047930,1382691344,1382691356},
{77,0,690,0,1047934,1382691332,1382691344},
... yup, casks 76 down to 4 are also 0 live keys & live bytes ...
{3,0,690,0,1047928,1382690525,1382690533},
{2,0,690,0,1047948,1382690517,1382690525},
{1,0,690,0,1047943,1382690510,1382690517}]
{<<0,0,0,0,0,0,0,0>>,0,false}
- If a file is removed, and the
keydirs->fstats
sayslive_keys
andlive_bytes
is 0, then we ought definitely get rid of it. - If a file is removed, and the
keydirs->fstats
says either oflive_keys
orlive_bytes
is not 0, then we have a bug?
What if we deleted the keydirs->fstats
entry as soon as live_keys
and live_bytes
dropped to 0? My sinus-infection-wracked brain isn't clear on whether that's a good idea or not.
from bitcask.
I am tempted to add a new nif fun called keydir_trim_fstats([integer()])
, then since erlang-side bitcask is (should be?) the authority on file existence, have it trim the those files_ids when it updates the list of open files post-merge, around here:
https://github.com/basho/bitcask/blob/develop/src/bitcask.erl#L667
Since we don't log from the keydir code, this would make it easier to do something like:
case bitcask_nifs:keydir_trim_fstats(DeletedFileList) of
ok -> ok;
{error, FileList} ->
lager:error("bitcask keydir reports deleted files ~p had live keys or data")
end,
Which would be hard or API-deforming if we tried to do it from the delete.
from bitcask.
Another reason to not do automatic removal: in my first commit on 1.6.3-release...pevm-status-workaround (my attempt at a quick beam-only hotpatch), I found that since the summary is the only source for fstats, if 0-key or 0-data stuff is gone before it's merged, it'll never get merged.
from bitcask.
Related Issues (20)
- bitcask NIF doesn't compile cleanly on OS X 10.10 [JIRA: RIAK-1574] HOT 2
- Startup crash trying to read key from corrupted data file [JIRA: RIAK-1819] HOT 5
- Repeated crashes during merge operations [JIRA: RIAK-1903] HOT 4
- Can't manually force a merge, error - efile_driver_unavailable [JIRA: RIAK-2036] HOT 5
- Include Bitcask data file count in merge info messages. [JIRA: RIAK-2039]
- Potential Process Leak [JIRA: RIAK-2227] HOT 2
- Lock file staleness check could be more robust [JIRA: RIAK-2230]
- question about read concurrency [JIRA: RIAK-2330] HOT 1
- Operation aborts if any file contains a corrupt key [JIRA: RIAK-2371] HOT 1
- rebar abort [JIRA: RIAK-2485] HOT 5
- Errors in `merge_single_tombstone` and `open_file` `append` mode [JIRA: RIAK-2733] HOT 2
- bitcask:merge/1 keeps file descriptors of deleted files open [JIRA: RIAK-2814] HOT 2
- File Open behaves differently between NIF and Erlang modes [JIRA: RIAK-2895] HOT 1
- bitcask failed to merge.merge_locked,locked
- c_src/bitcask_nifs.c: pair of bad % specifiers ?
- Build broken with OTP 21: use of the deprecated get_stacktrace/0 in dependency meck HOT 3
- Call bitcask_merge_worker:merge(DirName) is error
- How to run YCSB workload for bitcask?
- In the bitcask intro paper we see it being mentioned that bitcask is able to support datasets larger than 10 X system memory. I am trying to understand how? HOT 4
- Recreate hintfile HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bitcask.