adamhadani / logtools Goto Github PK

Example 12 in the README.md file uses a --format argument which is not recognized by qps. Apparently -F must be used instead, although --format as an alias would be desirable.

I see the following output:

$ percentiles.sh -f "accessToken.txt"
Thu Oct 10 10:47:57 2010 - Number of samples used: 118684
Thu Oct 10 10:47:57 2010 - 10th Percentile (11868th Sample): 23
Thu Oct 10 10:47:57 2010 - 20th Percentile (23736th Sample): 40
Thu Oct 10 10:47:57 2010 - 30th Percentile (35605th Sample): 58
...
Thu Oct 10 10:47:59 2010 - 99th Percentile (117497th Sample): 1000

my first guess is that "percentiles" code:

• opens the given file
• strips all data in all lines except for response time
• sorts the lines
• goes through the lines and prints response time for several "percentiles": 10,20,30, etc.

in other words, this shows that
10% of requests were completed in under 23 msec
20% of requests were completed in under 40 msec
etc
99% of requests were completed in under 1000 msec

is this correct?
if yes, maybe it makes sense to update documentation to include this info?

Hi,

Interesting package, which I adapted to my use cases. In this process, moved to Python3, added database related extensions...

To figure out the details, see extensions https://github.com/AlainLich/logtools/blob/ALPython3/PYTHON3-ADDITIONS.md. On my site https://github.com/AlainLich/logtools use branch ALPython3 (which should be the default)

Current plan is to enter a PR once more testing has been done, could do it sooner if interest; will take remarks into account if this facilitates integration.

Sorry, this is not modular, but I discovered the interesting things (in my view) late, so I did not want to undo or make modular.

SQLSoup has been refactored out of latest SQLAlchemy as a separate module. This means the module path (if that's the correct terminology) has changed and using logtools with SQLAlchemy 0.8 gives an ImportError.

I was able to workaround for the time being by downgrading to SQLAlchemy 0.7.9.

Hi,

where can I download "bots_hosts.txt" and "bots_useragents.txt" file ?

Can you append this information in logtools REAME ?

Thanks for your help,
Regards,
Stephane

Hi,

Great package. I have hit one small problem:

using logtools==0.8

got this exception:

  File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/_parse.py", line 100, in logparse
    yield key_func(line)
  File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/parsers.py", line 47, in multiindex_getter
    data = parser(line.strip())
  File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/parsers.py", line 69, in __call__
    return self.parse(line)
  File "/home/bkc/Python_Environments/weblog/lib/python2.7/site-packages/logtools/parsers.py", line 183, in parse
    raise ValueError("Could not parse log line: '%s'" % logline)
  ValueError: Could not parse log line: '122.118.199.146 - \"car5941 \" [09/Apr/2012:14:35:48 -0400] "GET /favicon.ico HTTP/1.1" 200 125 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.151 Safari/535.19"'

Editing the input file and removing the space character, e.g. "car5941 " to "car5941" fixes the problem.

I get lines like this from qps:

2012-02-03 13:43:59 2012-02-03 13:44:14 1241    82.7333333333

A truncated value like 82.73 would look nicer in the rightmost column (queries per second).