bmatheny / memkeys Goto Github PK

View Code? Open in Web Editor NEW

98.0 98.0 58.0 265 KB

a top like tool for inspecting memcache key values in realtime

License: Apache License 2.0

Shell 9.04% C++ 83.74% C 0.55% Makefile 2.52% M4 4.16%

memkeys's People

Contributors

Stargazers

Watchers

memkeys's Issues

memkeys does not run on Ubuntu 12.04 (memcached on the same machine)

I tried mctop then memkeys but unfortunately, both didn't work. I used apt-get to install all dependencies and gcc 4.6.3 to compile on Ubuntu 12.04. When I run memkeys (and mctop), both don't display any data. I guess there's some problem with libpcap so that memkeys can not capture any packet (I run memcached on the same machine). Can you help me to figure out what's wrong and how to fix the problem? Thank you so much.

Allow report type to be user specified

There are two reports available: the default ncurses report and the CSV report. Currently only the ncurses report can be used. It would be useful for a user to be able to choose which report to run based on a command line arg (-R or --report), defaulting to ncurses.

Logging

Need some basic logging based on specified verbosity.

PCRE symbol table runtime error

Hey there!

Thanks so much for open sourcing memkeys. I'm having trouble getting memkeys to run, and am leaving feedback here since github issues seem to be disabled on the official tumblr repo.

I built an RPM from the SPEC file found here: https://github.com/tumblr/memkeys/pull/1/files

I installed it on one of our production memcache boxes and started memkeys:

sudo memkeys -i eth0

Almost instantly memkeys crashes and prints out the following message:

memkeys: symbol lookup error: memkeys: undefined symbol: _ZN7pcrecpp2RE4InitEPKcPKNS_10RE_OptionsErt order (asc|desc)

It looks like something with pcrecpp, and it looks like memkeys is linked against pcre correctly:

~ ldd `which memkeys`
        linux-vdso.so.1 =>  (0x00007fffc11f9000)
        libncurses.so.5 => /lib64/libncurses.so.5 (0x00000033d3400000)
        libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x0000003d2fe00000)
        libpcrecpp.so.0 => /usr/lib64/libpcrecpp.so.0 (0x00007f309f0ca000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003005000000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f309edc4000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003005800000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003006800000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003004800000)
        libtinfo.so.5 => /lib64/libtinfo.so.5 (0x0000003006400000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000033d3800000)
        libpcre.so.0 => /lib64/libpcre.so.0 (0x000000317c400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003004400000)

PCRE version:

~ rpm -qa pcre
pcre-7.8-4.el6.x86_64

Do you have any insight into what could be going on here?

Thanks, memkeys looks really helpful!

Blake

Implement Report Class (ncurses)

Implement another report view which allows users to interact with the data via a curses UI.

Implement Report Class (CSV)

Implement a report class which is capable of outputting stats into a CSV format.

Not getting any output displayed - even though tcpdump does show data

Hi, I'll happily provide any further info or check what's needed.
Here is the the result of the log file:

INFO [20130722-15:50:21.688] [memkeys.cpp:51][getInstance] main: Starting application memkeys. PID 12067
INFO [20130722-15:50:21.756] [util/stats.cpp:48][start] stats: Starting stats engine
INFO [20130722-15:50:21.756] [net/pcap_live.cpp:38][open] pcap: Opening device eth1 for capture
INFO [20130722-15:50:21.756] [net/capture_engine.cpp:102][processPackets] capture-engine: Worker 0 starting capture processing
INFO [20130722-15:50:21.756] [net/capture_engine.cpp:102][processPackets] capture-engine: Worker 1 starting capture processing
INFO [20130722-15:50:21.757] [util/stats.cpp:181][prune] stats: Starting prune with threshold 0.00
INFO [20130722-15:50:21.757] [util/stats.cpp:149][collect] stats: Starting stats collection
INFO [20130722-15:50:21.758] [net/capture_engine.cpp:102][processPackets] capture-engine: Worker 2 starting capture processing
INFO [20130722-15:50:21.772] [net/pcap.cpp:30][setFilter] pcap: Applying filter (tcp port 5555) to pcap session
INFO [20130722-15:50:31.763] [memkeys.cpp:162][signal_cb] capture-engine: Shutting down due to signal
INFO [20130722-15:50:31.763] memkeys: packets (recv/dropped): 8205 / 0 (0.00%)
INFO [20130722-15:50:31.763] [memkeys.cpp:99][tryShutdown] memkeys: Shutting down engine
INFO [20130722-15:50:31.763] [util/stats.cpp:59][shutdown] stats: Stopping stats engine
INFO [20130722-15:50:31.763] [net/capture_engine.cpp:153][processPackets] capture-engine: Worker 2 stopped processing packets
INFO [20130722-15:50:31.765] [net/capture_engine.cpp:153][processPackets] capture-engine: Worker 1 stopped processing packets
INFO [20130722-15:50:31.765] [net/capture_engine.cpp:153][processPackets] capture-engine: Worker 0 stopped processing packets
INFO [20130722-15:50:32.181] [util/stats.cpp:174][collect] stats: Stats collect thread stopped
INFO [20130722-15:50:32.758] [util/stats.cpp:61][shutdown] stats: Repear thread dead
INFO [20130722-15:50:32.758] [util/stats.cpp:63][shutdown] stats: Poller thread dead
INFO [20130722-15:50:32.758] [report/report.cpp:14][shutdown] cursesReport: Shutting down
INFO [20130722-15:50:32.758] [memkeys.cpp:104][tryShutdown] memkeys: Stopping packet capture
INFO [20130722-15:50:32.762] [memkeys.cpp:83][run] memkeys: Finished packet capture
INFO [20130722-15:50:32.762] [main.cpp:50][main] main: Shutting down
INFO [20130722-15:50:32.762] [net/pcap_live.cpp:19][~PcapLive] pcap: Shutting down
INFO [20130722-15:50:32.762] [net/pcap.cpp:114][close] pcap: Closing pcap session
INFO [20130722-15:50:32.767] [net/capture_engine.cpp:36][~CaptureEngine] capture-engine: Capture engine successfully shut down
INFO [20130722-15:50:33.259] [util/stats.cpp:39][~Stats] stats: Stats successfully shut down
INFO [20130722-15:50:33.259] [net/capture_engine.cpp:46][~CaptureEngine] capture-engine: Worker thread 0 dead
INFO [20130722-15:50:33.259] [net/capture_engine.cpp:46][~CaptureEngine] capture-engine: Worker thread 1 dead
INFO [20130722-15:50:33.259] [net/capture_engine.cpp:46][~CaptureEngine] capture-engine: Worker thread 2 dead

I'm seeing packet being captured (packets (recv/dropped): 8205 / 0 (0.00%)), but the main window remains empty.
Running this on :

~/memkeys$ uname -a
Linux memcached08 3.2.0-24-virtual #37-Ubuntu SMP Wed Apr 25 10:17:19 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Using plain _sudo tcpdump -i eth1 port 11111 -w - | strings_ I do get the output I want.

Finish CLI Handler

Things we still need for CLI handling:

Help text
Bounds testing
Exception handling
Destructor on Config class

Allow pcap settings to be user specified

Sane defaults were picked for pcap_open_live but it would be nice if users could specify these values as command line flags.

Refactor LoggerPtr

Should use RAII pattern.

Update Docs

Usage, caveats, build requirements, etc.

Logger Updates

We should probably change the logfile config semantics such that logging will always be done to a file unless -v is specified or something similar. Write to a tmp file unless --logfile is specified.

Segfault on Debian Stretch+

Hi! While testing memkeys on Debian Stretch/Buster, I got the following segfault right after starting:

__memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
440	../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:440
#1  0x00007ffff7c94998 in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x00007ffff7c85b34 in std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x0000555555565b36 in std::operator<< <char, std::char_traits<char>, std::allocator<char> > (__str=..., __os=...)
    at /usr/include/c++/8/bits/basic_string.h:6323
#4  mckeys::CursesReport::renderHeader (this=0x555555598b80) at report/curses.cpp:99
#5  0x0000555555567d51 in mckeys::CursesReport::render (this=0x555555598b80) at report/curses.cpp:65
#6  0x00007ffff7c33b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007ffff7d03fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007ffff79114cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Some info:

vagrant@stretch:~/memkeys$ ldd /usr/local/bin/memkeys
	linux-vdso.so.1 (0x00007fff715f4000)
	libncurses.so.6 => /lib/x86_64-linux-gnu/libncurses.so.6 (0x00007ffbd03b7000)
	libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007ffbd0389000)
	libpcap.so.0.8 => /usr/lib/x86_64-linux-gnu/libpcap.so.0.8 (0x00007ffbd0147000)
	libpcrecpp.so.0 => /usr/lib/x86_64-linux-gnu/libpcrecpp.so.0 (0x00007ffbd013b000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffbd011a000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffbcff96000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffbcfe11000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffbcfdf7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffbcfc36000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffbcfc31000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007ffbcfbbd000)
	/lib64/ld-linux-x86-64.so.2 (0x00007ffbd041e000)

I only added -g to the CPPFLAGS to get a proper stacktrace. I understand that new libraries are untested (also gcc versions etc..) but I am wondering if anything can be done to make memkeys work on newer platforms :)

Implement Sniffer Class v2

Update the sniffer class to use the new stats class, instead of just outputting everything to stdout.

Unit Tests First Pass

Need unit tests for Config class.

Should this be a singleton? Maybe a reset() method would do...
Test boundary cases.

Implement Sniffer Class v1

Need to implement a class to handle sniffing based on the port and interface specified via the config.

Use the specified interface (and fail if not found)
Use the port as a pcap filter
Allow pcap_open_live params to be mostly configurable: snaplen/capture size, promisc or no, error timeout
Implement call back handler to print appropriate data to stdout

Good article here. Also here.

Implement Stats Class v1

Implement a stats class to handle new memcache requests. This might look something like:

case class Stat(key: Symbol, size: Long, created: Long = System.currentTimeMillis) {
  private[this] val _calls = new java.util.concurrent.atomic.AtomicLong();

  def elapsed: Long = System.currentTimeMillis - created

  def bandwidth(e: Long = elapsed): Double = (size * reqRate(e) * 8) / 1000
  def calls(): Long = _calls.get
  def reqRate(e: Long = elapsed): Double = {
    _calls.get / e
  }
  def incrementCalls() {
    _calls.incrementAndGet
  }
}
trait Stats {
  import scala.collection.mutable.{Map => MutableMap}
  import java.util.concurrent.ConcurrentHashMap

  val capStats = MutableMap[Symbol, Int](
    'recv -> 0,
    'drop -> 0
  )

  val stats: ConcurrentHashMap[Symbol, Stat] =
    new ConcurrentHashMap[Symbol,Stat]()

  def increment(key: Symbol, size: Long) {
    val stat = stats.putIfAbsent(key, Stat(key, size))
    stat.incrementCalls()
  }

  def prune(threshold: Double) {
    val now = System.currentTimeMillis
    stats.keys.foreach { k => // Not valid, but you get the point
      if (stats.get(k).reqRate(now) < threshold) {
        stats.remove(k)
      }
    }
  }
}

bmatheny / memkeys Goto Github PK

memkeys's People

Contributors

Stargazers

Watchers

Forkers

memkeys's Issues

Recommend Projects

Recommend Topics

Recommend Org