benchmark for python2.7 about pyperformance

[vstinner]

Hi,

First of all, pyperformance is just an helper running "python -m performance". You can run: " bin/python2.7 -m performance ..." as you use " bin/python2.7 -m pip ..." instead of just "pip". For pyperfomrance, it should be in the same bin directory than the installed python2.7. No?

It seems like you compiled and installed Python manually. FYI performance has builtin tools to do that: compile and install Python, install performance and then run performance.
http://pyperformance.readthedocs.io/usage.html#compile-python-to-run-benchmarks

It uses a configuration file where you put your directories, configure options, etc.

[Pintu79]

Hi @Haypo,
Thanks for the detailed explanation. yes, I have compiled and installed the python-2.7. I run the command as follows based on your feedback.
ci@t2ctl:~/Python-2.7.13.install$ bin/python2.7 -m performance run -o py2.json
Python benchmark suite 0.5.5

[ 1/50] 2to3...
INFO:root:Running /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/bin/python -u /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/lib/python2.7/site-packages/performance/benchmarks/bm_2to3.py --output /tmp/tmpYHoSuV
.....................
2to3: Mean +- std dev: 929 ms +- 2 ms
[ 2/50] chameleon...
INFO:root:Running /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/bin/python -u /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/lib/python2.7/site-packages/performance/benchmarks/bm_chameleon.py --output /tmp/tmpdYRT49
.....................
chameleon: Mean +- std dev: 30.8 ms +- 0.1 ms
[ 3/50] chaos...
INFO:root:Running /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/bin/python -u /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/lib/python2.7/site-packages/performance/benchmarks/bm_chaos.py --output /tmp/tmpsCVS1H
.....................
chaos: Mean +- std dev: 325 ms +- 0 ms
[ 4/50] crypto_pyaes...
INFO:root:Running /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/bin/python -u /home/ci/Python-2.7.13.install/venv/cpython2.7-69770f75f1ef/lib/python2.7/site-packages/performance/benchmarks/bm_crypto_pyaes.py --output /tmp/tmpadLtWX
.....................

can you please explain me this output. I was looking to measure python performance in terms of throughput and response time. But, this output I am not able to understand. So, can you please elaborate the output in detail.
Kind Regards,
Pintu

[Pintu79]

I think I have run the command halfway ;-) I went through the documentation and run the furthur command as well.
bin/python2.7 -m perf show py2.json
bin/python2.7 -m perf check py2.json
bin/python2.7 -m perf metadata py2.json
bin/python2.7 -m perf stats py2.json
I have captured the output of the last command run. I have some query. So, it has 64 benchamrks result. If we take an example, say 2to3 benchmark, so can you explain me Raw value minimum, Raw value maximum, 95th percentile, Total number of values etc.
Kind Regards,
Pintu
performance_result.txt

[vstinner]

can you please explain me this output. I was looking to measure python performance in terms of throughput and response time.

At the end, you get a JSON file which can be displayed using

python3 -m performance show file.json

("python3 -m perf show file.json" works too, but it doesn't have the header with performance metadata)

It displays number of seconds for benchmarks. See the documentation of benchmarks:
http://pyperformance.readthedocs.io/benchmarks.html

If we take an example, say 2to3 benchmark, so can you explain me Raw value minimum, Raw value maximum, 95th percentile, Total number of values etc.

Since you use the perf module, please read the perf documentation: http://perf.readthedocs.io/en/latest/

Especially the Analyze benchmark results section: http://perf.readthedocs.io/en/latest/analyze.html

For perf stats, see: http://perf.readthedocs.io/en/latest/cli.html#stats-cmd -- it contains links to percentiles for example.

[vstinner]

Sorry, but I don't understand well what you are trying to do? If you want to compare Python2 to Python3, the first section of performance gives you the 3 commands:

pyperformance run --python=python2 -o py2.json
pyperformance run --python=python3 -o py3.json
pyperformance compare py2.json py3.json

Did you use the compare command?

[Pintu79]

Hi @Haypo,
tons of thanks for sharing the information in detail. I want to compare python2 and python3 performance. I will build python3 as well and run the benchmark.

[vstinner]

I suggest you to let performance compile Python for you, since it will compile it with the "right" option in a more reproductible way.

[Pintu79]

So, to compile python2, should I modify the script? I am also interested to build with custom gcc. I build it with gcc-7.1, so to build it from performance, how should I keep the path of this custom gcc binary.

[vstinner]

You can use "python3 -m performance compile bench.conf 2.7 2.7" to compile the 2.7 development branch, or you can put the following config in your bench.conf file and then use "python3 -m performance compile_all":

[compile_all]
branches = 2.7

Or you can specify a specific Git commit, a Git tag, etc.

http://pyperformance.readthedocs.io/usage.html#compile-python-to-run-benchmarks

[vstinner]

I build it with gcc-7.1, so to build it from performance, how should I keep the path of this custom gcc binary.

Oh, I was sure that there was a "configure_args" option in the configuration file, but no, I didn't implement this feature. I should add such option :-)

[Pintu79]

thanks for the explanation everything in detail. So, I think I should use my own way of building python.

[Pintu79]

Is this bnechmark suite (performance) specific to CPython?. I used this benchmark suite for benchmarking of python-2.7.12 downloaded from python.org. So, is it correct to use this benchmark tool for this flavor of python. should I rely on the result what I got from running it?

[vstinner]

Please don't use this bug tracker to ask questions. Please use https://mail.python.org/pipermail/speed/ mailing list to ask how to run benchmarks.