bmatheny / memkeys Goto Github PK
View Code? Open in Web Editor NEWa top like tool for inspecting memcache key values in realtime
License: Apache License 2.0
a top like tool for inspecting memcache key values in realtime
License: Apache License 2.0
I tried mctop then memkeys but unfortunately, both didn't work. I used apt-get to install all dependencies and gcc 4.6.3 to compile on Ubuntu 12.04. When I run memkeys (and mctop), both don't display any data. I guess there's some problem with libpcap so that memkeys can not capture any packet (I run memcached on the same machine). Can you help me to figure out what's wrong and how to fix the problem? Thank you so much.
There are two reports available: the default ncurses report and the CSV report. Currently only the ncurses report can be used. It would be useful for a user to be able to choose which report to run based on a command line arg (-R
or --report
), defaulting to ncurses.
Need some basic logging based on specified verbosity.
Hey there!
Thanks so much for open sourcing memkeys. I'm having trouble getting memkeys to run, and am leaving feedback here since github issues seem to be disabled on the official tumblr repo.
I built an RPM from the SPEC file found here: https://github.com/tumblr/memkeys/pull/1/files
I installed it on one of our production memcache boxes and started memkeys:
sudo memkeys -i eth0
Almost instantly memkeys crashes and prints out the following message:
memkeys: symbol lookup error: memkeys: undefined symbol: _ZN7pcrecpp2RE4InitEPKcPKNS_10RE_OptionsErt order (asc|desc)
It looks like something with pcrecpp, and it looks like memkeys is linked against pcre correctly:
~ ldd `which memkeys`
linux-vdso.so.1 => (0x00007fffc11f9000)
libncurses.so.5 => /lib64/libncurses.so.5 (0x00000033d3400000)
libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x0000003d2fe00000)
libpcrecpp.so.0 => /usr/lib64/libpcrecpp.so.0 (0x00007f309f0ca000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003005000000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f309edc4000)
libm.so.6 => /lib64/libm.so.6 (0x0000003005800000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003006800000)
libc.so.6 => /lib64/libc.so.6 (0x0000003004800000)
libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003006400000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000033d3800000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x000000317c400000)
/lib64/ld-linux-x86-64.so.2 (0x0000003004400000)
PCRE version:
~ rpm -qa pcre
pcre-7.8-4.el6.x86_64
Do you have any insight into what could be going on here?
Thanks, memkeys looks really helpful!
Blake
Implement another report view which allows users to interact with the data via a curses UI.
Implement a report class which is capable of outputting stats into a CSV format.
Hi, I'll happily provide any further info or check what's needed.
Here is the the result of the log file:
INFO [20130722-15:50:21.688] [memkeys.cpp:51][getInstance] main: Starting application memkeys. PID 12067
INFO [20130722-15:50:21.756] [util/stats.cpp:48][start] stats: Starting stats engine
INFO [20130722-15:50:21.756] [net/pcap_live.cpp:38][open] pcap: Opening device eth1 for capture
INFO [20130722-15:50:21.756] [net/capture_engine.cpp:102][processPackets] capture-engine: Worker 0 starting capture processing
INFO [20130722-15:50:21.756] [net/capture_engine.cpp:102][processPackets] capture-engine: Worker 1 starting capture processing
INFO [20130722-15:50:21.757] [util/stats.cpp:181][prune] stats: Starting prune with threshold 0.00
INFO [20130722-15:50:21.757] [util/stats.cpp:149][collect] stats: Starting stats collection
INFO [20130722-15:50:21.758] [net/capture_engine.cpp:102][processPackets] capture-engine: Worker 2 starting capture processing
INFO [20130722-15:50:21.772] [net/pcap.cpp:30][setFilter] pcap: Applying filter (tcp port 5555) to pcap session
INFO [20130722-15:50:31.763] [memkeys.cpp:162][signal_cb] capture-engine: Shutting down due to signal
INFO [20130722-15:50:31.763] memkeys: packets (recv/dropped): 8205 / 0 (0.00%)
INFO [20130722-15:50:31.763] [memkeys.cpp:99][tryShutdown] memkeys: Shutting down engine
INFO [20130722-15:50:31.763] [util/stats.cpp:59][shutdown] stats: Stopping stats engine
INFO [20130722-15:50:31.763] [net/capture_engine.cpp:153][processPackets] capture-engine: Worker 2 stopped processing packets
INFO [20130722-15:50:31.765] [net/capture_engine.cpp:153][processPackets] capture-engine: Worker 1 stopped processing packets
INFO [20130722-15:50:31.765] [net/capture_engine.cpp:153][processPackets] capture-engine: Worker 0 stopped processing packets
INFO [20130722-15:50:32.181] [util/stats.cpp:174][collect] stats: Stats collect thread stopped
INFO [20130722-15:50:32.758] [util/stats.cpp:61][shutdown] stats: Repear thread dead
INFO [20130722-15:50:32.758] [util/stats.cpp:63][shutdown] stats: Poller thread dead
INFO [20130722-15:50:32.758] [report/report.cpp:14][shutdown] cursesReport: Shutting down
INFO [20130722-15:50:32.758] [memkeys.cpp:104][tryShutdown] memkeys: Stopping packet capture
INFO [20130722-15:50:32.762] [memkeys.cpp:83][run] memkeys: Finished packet capture
INFO [20130722-15:50:32.762] [main.cpp:50][main] main: Shutting down
INFO [20130722-15:50:32.762] [net/pcap_live.cpp:19][~PcapLive] pcap: Shutting down
INFO [20130722-15:50:32.762] [net/pcap.cpp:114][close] pcap: Closing pcap session
INFO [20130722-15:50:32.767] [net/capture_engine.cpp:36][~CaptureEngine] capture-engine: Capture engine successfully shut down
INFO [20130722-15:50:33.259] [util/stats.cpp:39][~Stats] stats: Stats successfully shut down
INFO [20130722-15:50:33.259] [net/capture_engine.cpp:46][~CaptureEngine] capture-engine: Worker thread 0 dead
INFO [20130722-15:50:33.259] [net/capture_engine.cpp:46][~CaptureEngine] capture-engine: Worker thread 1 dead
INFO [20130722-15:50:33.259] [net/capture_engine.cpp:46][~CaptureEngine] capture-engine: Worker thread 2 dead
I'm seeing packet being captured (packets (recv/dropped): 8205 / 0 (0.00%)), but the main window remains empty.
Running this on :
~/memkeys$ uname -a
Linux memcached08 3.2.0-24-virtual #37-Ubuntu SMP Wed Apr 25 10:17:19 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Using plain _sudo tcpdump -i eth1 port 11111 -w - | strings_ I do get the output I want.
Things we still need for CLI handling:
Sane defaults were picked for pcap_open_live
but it would be nice if users could specify these values as command line flags.
Should use RAII pattern.
Usage, caveats, build requirements, etc.
We should probably change the logfile config semantics such that logging will always be done to a file unless -v
is specified or something similar. Write to a tmp file unless --logfile
is specified.
Hi! While testing memkeys on Debian Stretch/Buster, I got the following segfault right after starting:
__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
440 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
#1 0x00007ffff7c94998 in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007ffff7c85b34 in std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x0000555555565b36 in std::operator<< <char, std::char_traits<char>, std::allocator<char> > (__str=..., __os=...)
at /usr/include/c++/8/bits/basic_string.h:6323
#4 mckeys::CursesReport::renderHeader (this=0x555555598b80) at report/curses.cpp:99
#5 0x0000555555567d51 in mckeys::CursesReport::render (this=0x555555598b80) at report/curses.cpp:65
#6 0x00007ffff7c33b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007ffff7d03fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8 0x00007ffff79114cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Some info:
vagrant@stretch:~/memkeys$ ldd /usr/local/bin/memkeys
linux-vdso.so.1 (0x00007fff715f4000)
libncurses.so.6 => /lib/x86_64-linux-gnu/libncurses.so.6 (0x00007ffbd03b7000)
libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007ffbd0389000)
libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007ffbd0147000)
libpcrecpp.so.0 => /usr/lib/x86_64-linux-gnu/libpcrecpp.so.0 (0x00007ffbd013b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffbd011a000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffbcff96000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffbcfe11000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffbcfdf7000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffbcfc36000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffbcfc31000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007ffbcfbbd000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffbd041e000)
I only added -g
to the CPPFLAGS to get a proper stacktrace. I understand that new libraries are untested (also gcc versions etc..) but I am wondering if anything can be done to make memkeys work on newer platforms :)
Update the sniffer class to use the new stats class, instead of just outputting everything to stdout.
Need unit tests for Config class.
reset()
method would do...Need to implement a class to handle sniffing based on the port and interface specified via the config.
pcap_open_live
params to be mostly configurable: snaplen/capture size, promisc or no, error timeoutImplement a stats class to handle new memcache requests. This might look something like:
case class Stat(key: Symbol, size: Long, created: Long = System.currentTimeMillis) {
private[this] val _calls = new java.util.concurrent.atomic.AtomicLong();
def elapsed: Long = System.currentTimeMillis - created
def bandwidth(e: Long = elapsed): Double = (size * reqRate(e) * 8) / 1000
def calls(): Long = _calls.get
def reqRate(e: Long = elapsed): Double = {
_calls.get / e
}
def incrementCalls() {
_calls.incrementAndGet
}
}
trait Stats {
import scala.collection.mutable.{Map => MutableMap}
import java.util.concurrent.ConcurrentHashMap
val capStats = MutableMap[Symbol, Int](
'recv -> 0,
'drop -> 0
)
val stats: ConcurrentHashMap[Symbol, Stat] =
new ConcurrentHashMap[Symbol,Stat]()
def increment(key: Symbol, size: Long) {
val stat = stats.putIfAbsent(key, Stat(key, size))
stat.incrementCalls()
}
def prune(threshold: Double) {
val now = System.currentTimeMillis
stats.keys.foreach { k => // Not valid, but you get the point
if (stats.get(k).reqRate(now) < threshold) {
stats.remove(k)
}
}
}
}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.