adamhadani / logtools Goto Github PK
View Code? Open in Web Editor NEWCommand-line utilities for working with log files - filtering, sampling, IP-based geolocation, summary statistics etc.
License: Other
Command-line utilities for working with log files - filtering, sampling, IP-based geolocation, summary statistics etc.
License: Other
Example 12 in the README.md
file uses a --format
argument which is not recognized by qps
. Apparently -F
must be used instead, although --format
as an alias would be desirable.
Hi,
where can I download "bots_hosts.txt" and "bots_useragents.txt" file ?
Can you append this information in logtools REAME ?
Thanks for your help,
Regards,
Stephane
Hi,
Great package. I have hit one small problem:
using logtools==0.8
got this exception:
File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/_parse.py", line 100, in logparse
yield key_func(line)
File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/parsers.py", line 47, in multiindex_getter
data = parser(line.strip())
File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/parsers.py", line 69, in __call__
return self.parse(line)
File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/parsers.py", line 183, in parse
raise ValueError("Could not parse log line: '%s'" % logline)
ValueError: Could not parse log line: '122.118.199.146 - \"car5941 \" [09/Apr/2012:14:35:48 -0400] "GET /favicon.ico HTTP/1.1" 200 125 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.151 Safari/535.19"'
Editing the input file and removing the space character, e.g. "car5941 " to "car5941" fixes the problem.
Hi,
Interesting package, which I adapted to my use cases. In this process, moved to Python3, added database related extensions...
To figure out the details, see extensions https://github.com/AlainLich/logtools/blob/ALPython3/PYTHON3-ADDITIONS.md. On my site https://github.com/AlainLich/logtools use branch ALPython3 (which should be the default)
Current plan is to enter a PR once more testing has been done, could do it sooner if interest; will take remarks into account if this facilitates integration.
Sorry, this is not modular, but I discovered the interesting things (in my view) late, so I did not want to undo or make modular.
I get lines like this from qps:
2012-02-03 13:43:59 2012-02-03 13:44:14 1241 82.7333333333
A truncated value like 82.73
would look nicer in the rightmost column (queries per second).
I see the following output:
$ percentiles.sh -f "accessToken.txt"
Thu Oct 10 10:47:57 2010 - Number of samples used: 118684
Thu Oct 10 10:47:57 2010 - 10th Percentile (11868th Sample): 23
Thu Oct 10 10:47:57 2010 - 20th Percentile (23736th Sample): 40
Thu Oct 10 10:47:57 2010 - 30th Percentile (35605th Sample): 58
...
Thu Oct 10 10:47:59 2010 - 99th Percentile (117497th Sample): 1000
my first guess is that "percentiles" code:
in other words, this shows that
10% of requests were completed in under 23 msec
20% of requests were completed in under 40 msec
etc
99% of requests were completed in under 1000 msec
is this correct?
if yes, maybe it makes sense to update documentation to include this info?
SQLSoup has been refactored out of latest SQLAlchemy as a separate module. This means the module path (if that's the correct terminology) has changed and using logtools with SQLAlchemy 0.8 gives an ImportError
.
I was able to workaround for the time being by downgrading to SQLAlchemy 0.7.9.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.