manos / varnish-statsd-graphite-logger Goto Github PK
View Code? Open in Web Editor NEWPython script that reads varnishncsa log files, using pyinotify, and sends metrics to statsd / graphite.
Python script that reads varnishncsa log files, using pyinotify, and sends metrics to statsd / graphite.
Update: we stopped using this. varnishncsa sucks. This is better: https://github.com/jib/libvmod-statsd This is how we (at Krux) get varnish requests/s, hit / miss, and response time metrics in graphite. The script reads from a logfile (argument #1), using pyinotify. Via inotify, it supports both truncation and log rotation by means of rename + re-creation of the file. For every file IN_MODIFY event, the script reads the varnishncsa log, and sends a metric. FAQ: Q: why not use Etsy's Logster? A: it doesn't use inotify, and this was very important to us, due to dealing with > 15K req/s. Some things are pretty specific, like the use of kruxstatsd and our metrics layout - adjust to taste. Options: -h, --help show this help message and exit --cluster=CLUSTER_NAME The cluster_name stats will be grouped in --environment=ENVIRONMENT The (krux) environment the script is running in --stdin Instead of tailing a log file, read from stdin Usage: Run a varnishncsa daemon outputting the data this script expects: /usr/bin/varnishncsa -a -w /data/var/log/varnish/statsd_requests.log -F %U %{Varnish:time_firstbyte}x %{X-Request-Backend}o %{Varnish:hitmiss}x Run the script (via supervisor): python varnish_statsd_send.py --cluster=apiservices-a --environment=prod /data/var/log/varnish/statsd_requests.log Requires: pyinotify kruxstatsd (or implement your own mechanism for sending to statsd/graphite in the run() method.. see send_graphite.py for an example) Notes: varnishncsa won't log every request, if they are served by pipelining (in varnish, I mean, where it supresses multiple backend requests for the same HTTP pipelined request). This sucks, when using 'ab' with a high concurrency to test.. but in production, we never see the same exact request from the same HTTP client pipelined, so it's really not a big deal. Something to be aware of, though. TODO: currently at log rotation, we (correctly) catch-up with missed events based upon the file handle cursor. But this leads to big spikes in graphs, at high traffic rates. Once varnish supports strftime() formatting (support is in master now, but not in a release), we should start logging that too, and send catch-up events directly to graphite with the correct timestamp.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.