androa / statpipe

Poor mans splunk. Command line statistics real time statistcs while tailing logs

Statpipe

#fisk

NAME

statpipe - swiss knife statistics

DESCRIPTION

statpipe is a excellent little tool to analyse logfiles, or any file for that matter, and produce percentage of hits, hits per second and other cool stuff. It's supposed to be a better way of doing something similar to tail -f | awk | cut| sort | unique -c |sort -g | whatever.

SYNOPSIS

tail -f some.log | statpipe [options] [regex] ... [regex]

Regex is a perl regex, if the regex has a group 'something\.(.*)' the match will be used as a key instead of the regexp itself.

If no regex and no --field argument is given. It will be as '^(.*)$' was given. Meaning that it will count all unique lines in the file/pipe.

Options:
 --field|f         What field top use as key (default all fields)
 --delimiter|d     What delimiter to use for fields (spaces)
 --timefreq|-t     Frequency of output in seconds (5)
 --linefreq        Frequency of output in lines (none)
 --maxtime         Time before closing the pipe in seconds (60)
 --maxlines        Maximum numbers of lines to parse (unlimited)
 --multi|m         Match multiple times per line (no)
 --limit           Limit output of keys (30)
 --maxkeys         Max number of unique keys (50000)
 --not|n           Exclude lines with regex
 --case|s          Be casesensetive
 --clear           Clear screen between updates
 --relative|r      Show relative percentages (no)
 --keysize|k       Length of keys (output)
 --(no)hits        Show hits per second (yes)
 --help            Show help
 --version         Show version

EXAMPLES

#Show top 30 visited urls. Update it every 5 seconds for 60 seconds (default)
$ tail -f /var/log/httpd/access.log | statpipe -f 7

#Seperate fields by " and show field two
$ tail -f /var/log/httpd/access.log | statpipe -d \" -f 2

#Group jpeg and jpg differently
$ tail -f /var/log/httpd/access.log | statpipe 'jpe?g' png gif

#Group jpeg and jpg into one key
$ tail -f /var/log/httpd/access.log | statpipe '(jpe?g)' png gif --not gift

#Count all words in a file
$ cat file | statpipe --multi '(\w)'

#List top 20 articles the last 10 seconds
$ tail -f /var/log/httpd/access.log | statpipe 'artid=(\d+)' --maxtime=10 --limit 20 --time=0

BUGS

Probably plenty.

TODO

TODO: Merge ($1) ($2) etc. TODO: Name change: PMS? (Poor mans Splunk) (Pipe measure system), statpipe TODO: Read defaultsfile from .statpipe?

COPYRIGHT

Audun Ytterdal [email protected] http://github.com/auduny/statpipe/