python / pyperformance Goto Github PK
View Code? Open in Web Editor NEWPython Performance Benchmark Suite
Home Page: http://pyperformance.readthedocs.io/
License: MIT License
Python Performance Benchmark Suite
Home Page: http://pyperformance.readthedocs.io/
License: MIT License
I am running pyperformance run -r -m
with a self built CPython 3.7.0 on Ubuntu 18.04.2 LTS. The command stalls at tornado_http
.
I believe this was introduced with the last version, as version 0.7.0 works without problem. Without --track-memory
the problem does not occur either.
Not sure if this belongs here but right now running bench.py with run
brings up this error. It seems like there's an active issue for this: https://bugs.python.org/issue14365
> ..\python_d.exe bench.py run
usage: bench.py [-h] {run,compare,run_compare,list,list_groups} ...
bench.py: error: argument action: ambiguous choice: 'run' could match run, run_compare
As a workaround I changed my run
to run_local
in https://github.com/python/benchmarks/blob/master/benchmark/cli.py#L84
and
https://github.com/python/benchmarks/blob/master/benchmark/cli.py#L150
and
https://github.com/python/benchmarks/blob/master/benchmark/run.py#L445
pyperformance run -m
simply fails:
[ 1/50] 2to3...
INFO:root:Running/tmp/venv/bin/python -u /usr/lib/python2.8/site-packages/performance-0.6.1-py2.8.egg/performance/benchmarks/bm_2to3.py --track-memory --output /tmp/tmpmT5Q1F
.ERROR: --worker requires --loops=N or --calibrate-loops
Traceback (most recent call last):
File "/usr/lib/python2.8/site-packages/performance-0.6.1-py2.8.egg/performance/benchmarks/bm_2to3.py", line 30, in
runner.bench_func('2to3', bench_2to3, command, devnull_out)
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_runner.py", line 485, in bench_func
return self._main(task)
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_runner.py", line 420, in _main
bench = self._master()
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_runner.py", line 543, in _master
bench = Master(self).create_bench()
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_master.py", line 221, in create_bench
worker_bench, run = self.create_worker_bench()
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_master.py", line 120, in create_worker_bench
suite = self.create_suite()
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_master.py", line 114, in create_suite
suite = self.spawn_worker(0, 0)
File "/usr/lib/python2.8/site-packages/perf-1.4-py2.8.egg/perf/_master.py", line 97, in spawn_worker
% (cmd[0], exitcode))
RuntimeError: /tmp/venv/bin/python failed with exit code 1
ERROR: Benchmark 2to3 failed: Benchmark died
Traceback (most recent call last):
File "/usr/lib/python2.8/site-packages/performance-0.6.1-py2.8.egg/performance/run.py", line 132, in run_benchmarks
bench = func(cmd_prefix, options)
File "/usr/lib/python2.8/site-packages/performance-0.6.1-py2.8.egg/performance/benchmarks/init.py", line 88, in BM_2to3
return run_perf_script(python, options, "2to3")
File "/usr/lib/python2.8/site-packages/performance-0.6.1-py2.8.egg/performance/run.py", line 98, in run_perf_script
run_command(cmd, hide_stderr=not options.verbose)
File "/usr/lib/python2.8/site-packages/performance-0.6.1-py2.8.egg/performance/run.py", line 66, in run_command
raise RuntimeError("Benchmark died")
RuntimeError: Benchmark died
pyperformance version is 0.6.1, perf version is 1.4, kernel version is 3.10.0.
https://speed.python.org/comparison/ has no 3.10 branch.
I tried to investigate a way to fix this but I'm not too sure where the branch configurations are stored. Does anyone know if they're at https://github.com/zware/codespeed or somewhere else?
I don't known if this is an issue here, but it may have something to do with the default branch of CPython's got repo being renamed to "main". There seem to be some remaining references to "master" here.
E.g. on Windows the binary is named python.exe
-- same on macOS -- and the binary is stored in a Scripts
directory instead of bin
(#2 tries to solve this). os.execv()
also doesn't quite operate the way you think on (at least) Windows 10 as it immediately returns to PowerShell/cmd.exe and the process runs in the background of the prompt, seeming to pause on I/O unless the user interacts with the terminal, e.g. hitting enter (if you type exit
at the prompt then the new process takes over the terminal window, although if you kill that process then you kill the entire terminal window instead of returning, including the end of the benchmark run).
The problem
I tried to compare the performance results between different Python versions and implementations. While the comparison between CPython 3.6 and CPython 2.7 works as expected, I get an exception when comparing the results obtained with CPython 2.7.13 and PyP 2.7.13.
Exact versions:
CPython:
Python 2.7.13 |Continuum Analytics, Inc.| (default, Dec 20 2016, 23:05:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
PyPy:
(Python 2.7.13 (c925e7381036, Jun 05 2017, 20:53:58)
[PyPy 5.8.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
How to reproduce the issue
Run from CPython 2 environment:
python -m performance run -o py27.json
Run from PyPy environment:
pypy -m performance run -o pypy27.json
Compare:
pyperformance compare -O table py27.json pypy27.json
What you expected to happen
I expected table like this:
+-------------------------+-----------+-------------+-----------------+-------------------------+
| Benchmark | py27.json | pypy27.json | Change | Significance |
+=========================+===========+=============+=================+=========================+
| 2to3 | 767 ms | 1.63 sec | 2.13x slower | Significant (t=-142.45) |
+-------------------------+-----------+-------------+-----------------+-------------------------+
| chaos | 215 ms | 5.58 ms | 38.62x faster | Significant (t=204.35) |
+-------------------------+-----------+-------------+-----------------+-------------------------+
What actually happens
I get this exception:
compare.py", line 212, in __init__
raise RuntimeError("base and changed don't have "
RuntimeError: base and changed don't have the same number of values
Note: The line number my have change do to my debug prints.
Cause
The number of values for the benchmarks python_startup
and python_startup_no_site
are different, i.e. 200 for CPython and 60 for PyPy (same numbers for both benchmarks)
My work around
I just skipped python_startup
and python_startup_no_site
with:
if name in ('python_startup', 'python_startup_no_site'):
continue
in compare.compare_results:
def compare_results(options):
base_label, changed_label = get_labels(options.baseline_filename,
options.changed_filename)
base_suite = perf.BenchmarkSuite.load(options.baseline_filename)
changed_suite = perf.BenchmarkSuite.load(options.changed_filename)
results = []
common = set(base_suite.get_benchmark_names()) & set(
changed_suite.get_benchmark_names())
for name in sorted(common):
print(name)
if name in ('python_startup', 'python_startup_no_site'):
continue
base_bench = base_suite.get_benchmark(name)
changed_bench = changed_suite.get_benchmark(name)
result = BenchmarkResult(base_bench, changed_bench)
results.append(result)
Suggested better solution
Either:
This is a feature request.
Would it make sense to add a benchmark for code that heavily uses annotations, generics, and other typing
features? I even have a suggestion for such benchmark: running mypy
on itself (on its own code). The run currently takes around 10-20 sec. (depending on machine). If this is too long, other option would be to run mypy on one of typeshed stubs, this will take less than a second.
It would be nice to print out more version details in the compare output to act as a double-check that the results are using data from interpreters that are expected.
During the compile_all stage, the script tries to print the version on the screen.
If install=True in the configuration, make install
is run. Darwin does compile the CPython binary as python.exe
but once you run make install
it corrects it to the right name.
I can see the fault in the logic here.
https://github.com/python/performance/blob/f9ebee09652f77ca01736f9b034a0bb0d8826a45/performance/compile.py#L301
2018-03-31 20:20:25,352: Installed Python version:
2018-03-31 20:20:25,353: + /Users/anthonyshaw/repo/python_comparison/bench_tmpdir/prefix/bin/python3.exe --version
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/__main__.py", line 2, in <module>
performance.cli.main()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/cli.py", line 226, in main
_main()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/cli.py", line 190, in _main
cmd_compile(options)
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 921, in cmd_compile
bench.main()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 656, in main
failed = self.compile_bench()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 641, in compile_bench
self.compile_install()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 459, in compile_install
self.python.compile_install()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 372, in compile_install
self.get_version()
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 312, in get_version
self.run(self.program, '--version')
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 63, in run
self.app.run(*cmd, cwd=self.cwd, **kw)
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 195, in run
exitcode = self.run_nocheck(*cmd, **kw)
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 174, in run_nocheck
proc = self.create_subprocess(cmd, **kwargs)
File "/Users/anthonyshaw/repo/python_comparison/env/lib/python3.6/site-packages/performance/compile.py", line 158, in create_subprocess
return subprocess.Popen(cmd, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 707, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/subprocess.py", line 1326, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/anthonyshaw/repo/python_comparison/bench_tmpdir/prefix/bin/python3.exe'
2018-03-31 20:20:25,405: Command /Users/anthonyshaw/repo/python_comparison/env/bin/python3.6 -m performance compile /Users/anthonyshaw/repo/python_comparison/testing.conf 3.6 3.6 --no-update --no-tune failed with exit code 1
The function returns True if the module is not accelerated. Either it should be renamed, or the sense of the return value should be inverted; either way both call sites should be updated, because currently they read backwards.
File "lib/python3.8/site-packages/genshi/template/text.py", line 137, in __init__
Template.__init__(self, source, filepath=filepath, filename=filename,
File "lib/python3.8/site-packages/genshi/template/base.py", line 418, in __init__
self._stream = self._parse(source, encoding)
File "lib/python3.8/site-packages/genshi/template/text.py", line 181, in _parse
for kind, data, pos in interpolate(text, self.filepath, lineno,
File "lib/python3.8/site-packages/genshi/template/interpolation.py", line 77, in interpolate
expr = Expression(chunk.strip(), pos[0], pos[1],
File "lib/python3.8/site-packages/genshi/template/eval.py", line 93, in __init__
self.code = _compile(node, self.source, mode=self.mode,
File "lib/python3.8/site-packages/genshi/template/eval.py", line 470, in _compile
return build_code_chunk(code, filename, name, lineno)
File "lib/python3.8/site-packages/genshi/compat.py", line 94, in build_code_chunk
return CodeType(0, code.co_nlocals, code.co_kwonlyargcount,
an integer is required (got type bytes)
ERROR: Package 'mercurial' requires a different Python: 3.8.0 not in '~= 2.7'
See https://www.mercurial-scm.org/repo/hg/file/5.0.2/setup.py#l10
HGALLOWPYTHON3 needs to be set as well so it can be installed with setuptools.
Hi all,
Recently I've been trying to benchmark a commit in cpython master
branch against another commit in the same branch. The docs provide an example doc/benchmark.conf.sample
file and describe it as
[compile_all_revisions]
list of 'sha1=' (default branch: 'master') or 'sha1=branch'
I spent a few hours digging through the compile.py
code to figure out what I was doing wrong before I finally realized I misinterpreted the docs. So I'd like to submit a PR to update the docs to make it slightly less confusing.
Thanks for your time!
can it be used for performance testing of python-2.7.
I build the python from the source code as folows -
/home/ci/Python-2.7.13/configure CC=/home/gcc/gcc-7.1.0.install/bin/gcc --build=powerpc64le-linux-gnu --host=powerpc64le-linux-gnu --target=powerpc64le-linux-gnu --enable-shared --with-ensurepip="install" --enable-optimizations --prefix=/home/ci/Python-2.7.13.install --enable-unicode=ucs4
make -j 64
make altinstall
after building python in the Python-2.7.13.install directoy, I went to this install directory and run the command as follows -
bin/python2.7 -m pip install performance
after triggering the above command, I dont see any binary for pyperformance. So this benchamrk is only applicable to python3 or i am doing some mistakes to install them properly. can you please help me out.
Kind Regards,
Pintu
On PyPy, I don't think that performance currently measures correctly performance after the code has been optimized by the JIT compiler. I started a thread on the PyPy mailing list:
https://mail.python.org/pipermail/pypy-dev/2017-April/015101.html
performance 0.7.0 fails on the master branch of Python (future Python 3.8), because Chameleon has no handler for AST node type ast.Constant.
performance 0.7.0 uses Chameleon 3.4. I see two options:
Chameleon project homepage: https://github.com/malthe/chameleon/
Full traceback:
2018-10-16 15:28:43,656: [ 2/47] chameleon...
2018-10-16 15:28:43,656: INFO:root:Running `/home/haypo/bench_tmpdir/venv/bin/python -u /home/haypo/performance/performance/benchmarks/bm_chameleon.py --verbose --output /tmp/tmp85spznrk`
2018-10-16 15:28:43,842: Traceback (most recent call last):
2018-10-16 15:28:43,842: File "/home/haypo/performance/performance/benchmarks/bm_chameleon.py", line 36, in <module>
2018-10-16 15:28:43,843: main()
2018-10-16 15:28:43,843: File "/home/haypo/performance/performance/benchmarks/bm_chameleon.py", line 26, in main
2018-10-16 15:28:43,843: tmpl = PageTemplate(BIGTABLE_ZPT)
2018-10-16 15:28:43,843: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/zpt/template.py", line 192, in __init__
2018-10-16 15:28:43,843: super(PageTemplate, self).__init__(body, **config)
2018-10-16 15:28:43,843: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/template.py", line 128, in __init__
2018-10-16 15:28:43,843: self.write(body)
2018-10-16 15:28:43,843: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/template.py", line 221, in write
2018-10-16 15:28:43,843: self.cook(body)
2018-10-16 15:28:43,843: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/template.py", line 158, in cook
2018-10-16 15:28:43,843: program = self._cook(body, digest, names)
2018-10-16 15:28:43,843: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/template.py", line 231, in _cook
2018-10-16 15:28:43,844: source = self._compile(body, builtins)
2018-10-16 15:28:43,844: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/template.py", line 265, in _compile
2018-10-16 15:28:43,844: compiler = Compiler(
2018-10-16 15:28:43,844: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/compiler.py", line 949, in __init__
2018-10-16 15:28:43,844: generator = TemplateCodeGenerator(module, source)
2018-10-16 15:28:43,844: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/codegen.py", line 116, in __init__
2018-10-16 15:28:43,844: super(TemplateCodeGenerator, self).__init__(tree)
2018-10-16 15:28:43,844: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 226, in __init__
2018-10-16 15:28:43,844: self.visit(tree)
2018-10-16 15:28:43,844: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/codegen.py", line 201, in visit
2018-10-16 15:28:43,844: super(TemplateCodeGenerator, self).visit(node)
2018-10-16 15:28:43,845: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 286, in visit
2018-10-16 15:28:43,845: ret = visitor(node)
2018-10-16 15:28:43,845: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/codegen.py", line 119, in visit_Module
2018-10-16 15:28:43,845: super(TemplateCodeGenerator, self).visit_Module(node)
2018-10-16 15:28:43,845: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 293, in visit_Module
2018-10-16 15:28:43,845: self.visit(n)
2018-10-16 15:28:43,845: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/codegen.py", line 201, in visit
2018-10-16 15:28:43,845: super(TemplateCodeGenerator, self).visit(node)
2018-10-16 15:28:43,845: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 286, in visit
2018-10-16 15:28:43,845: ret = visitor(node)
2018-10-16 15:28:43,845: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 387, in visit_Assign
2018-10-16 15:28:43,846: self.visit(node.value)
2018-10-16 15:28:43,846: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/codegen.py", line 201, in visit
2018-10-16 15:28:43,846: super(TemplateCodeGenerator, self).visit(node)
2018-10-16 15:28:43,846: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 286, in visit
2018-10-16 15:28:43,846: ret = visitor(node)
2018-10-16 15:28:43,846: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 829, in visit_Call
2018-10-16 15:28:43,846: self.visit(arg)
2018-10-16 15:28:43,846: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/codegen.py", line 201, in visit
2018-10-16 15:28:43,846: super(TemplateCodeGenerator, self).visit(node)
2018-10-16 15:28:43,846: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/chameleon/astutil.py", line 284, in visit
2018-10-16 15:28:43,846: raise Exception('No handler for ``%s`` (%s).' % (
2018-10-16 15:28:43,846: Exception: No handler for ``Constant`` (<_ast.Constant object at 0x7f42c52f8b80>).
2018-10-16 15:28:43,859: ERROR: Benchmark chameleon failed: Benchmark died
While I discover Sam's project, I noticed that there were several benchmarks that we do not currently use.
https://github.com/colesbury/nogil/tree/nogil/benchmarks
IMHO, the following lists are good candidates to be added. If other core devs agree to add those tests.
I will try to keep in touch with Sam about this issue :)
Regards,
Dong-hee
The richard's benchmark included with pyperformance seems to be a C -> C++ -> Java -> Python port. It makes heavy use of object oriented programming and doesn't look anything like the original C code.
I've been playing with the python2 version distributed from their website. I ported it to python3. It looks more like the C version, which has some good things and a few bad things:
+ Covers non-OO programming use case
- Overloads variables as in Union[int, Type1, Type2]
I'm in the process of improving it to Optional[Type1]
and Optional[Type2]
My goal is to explore a dialect of statically typed python using modern pythonic constructs:
The code is here: https://github.com/adsharma/richards-benchmark. Let me know if this sounds interesting and if you'd like to update the variant in the repo at some point in the future. I hear some of the faster cpython work is using these benchmarks.
Due to a current lack of representative macrobenchmarks, it is very difficult to decide on whether complex accelerators for some parts of typing are worth implementing in the future. Hence, I'm trying to upstream some benchmarks into pyperformance.
IMO, there are three main areas:
For case 2, I plan to use one of pydantic's benchmarks here https://github.com/samuelcolvin/pydantic/tree/master/benchmarks, installed without compiled binaries.
Case 3 is very tricky because there are so many ways to use typing. I don't know how often people use certain features, whether they type-hint inside tight loops, etc. So I'm struggling to find a good benchmark. An idea: grabbing one of the existing pyperformance benchmarks, fully type-hinting it, then comparing the performance delta may work.
CC @JelleZijlstra, I would greatly appreciate hearing your opinion on this (especially for case 3). Maybe I can post this on typing-sig too if I need more help.
Afterword:
All 3 cases benefit from general CPython optimizations. But usually only 3. benefits greatly from typing module-only optimizations (with 1. maybe not improving much if at all, depending on implementation).
performance 0.7.0 fails on the master branch of Python (future Python 3.8) because genshi uses _ast.Str type which is gone from Python 3.8.
visit_Name() at genshi/template/eval.py:616:
strarg = _new(_ast.Str, node.id)
performance 0.7.0 uses Genshi 0.7.1. I see two options:
Genshi project homepage: https://genshi.edgewall.org/
Full traceback:
2018-10-16 15:31:58,023: [10/47] genshi...
2018-10-16 15:31:58,024: INFO:root:Running `/home/haypo/bench_tmpdir/venv/bin/python -u /home/haypo/performance/performance/benchmarks/bm_genshi.py --verbose --output /tmp/tmpace4xllo`
2018-10-16 15:31:58,177: Traceback (most recent call last):
2018-10-16 15:31:58,177: File "/home/haypo/performance/performance/benchmarks/bm_genshi.py", line 68, in <module>
2018-10-16 15:31:58,178: runner.bench_time_func(name, bench_genshi, tmpl_cls, tmpl_str)
2018-10-16 15:31:58,178: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_runner.py", line 458, in bench_time_func
2018-10-16 15:31:58,178: return self._main(task)
2018-10-16 15:31:58,178: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_runner.py", line 423, in _main
2018-10-16 15:31:58,178: bench = self._worker(task)
2018-10-16 15:31:58,178: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_runner.py", line 397, in _worker
2018-10-16 15:31:58,179: run = task.create_run()
2018-10-16 15:31:58,179: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_worker.py", line 293, in create_run
2018-10-16 15:31:58,179: self.compute()
2018-10-16 15:31:58,179: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_worker.py", line 331, in compute
2018-10-16 15:31:58,179: WorkerTask.compute(self)
2018-10-16 15:31:58,179: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_worker.py", line 280, in compute
2018-10-16 15:31:58,180: self.calibrate_loops()
2018-10-16 15:31:58,180: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_worker.py", line 243, in calibrate_loops
2018-10-16 15:31:58,180: self._compute_values(self.warmups, nvalue,
2018-10-16 15:31:58,180: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_worker.py", line 76, in _compute_values
2018-10-16 15:31:58,180: raw_value = self.task_func(self, self.loops)
2018-10-16 15:31:58,180: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/perf/_runner.py", line 454, in task_func
2018-10-16 15:31:58,181: return time_func(loops, *args)
2018-10-16 15:31:58,181: File "/home/haypo/performance/performance/benchmarks/bm_genshi.py", line 29, in bench_genshi
2018-10-16 15:31:58,181: tmpl = tmpl_cls(tmpl_str)
2018-10-16 15:31:58,181: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/text.py", line 137, in __init__
2018-10-16 15:31:58,181: Template.__init__(self, source, filepath=filepath, filename=filename,
2018-10-16 15:31:58,181: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/base.py", line 418, in __init__
2018-10-16 15:31:58,181: self._stream = self._parse(source, encoding)
2018-10-16 15:31:58,182: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/text.py", line 181, in _parse
2018-10-16 15:31:58,182: for kind, data, pos in interpolate(text, self.filepath, lineno,
2018-10-16 15:31:58,182: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/interpolation.py", line 77, in interpolate
2018-10-16 15:31:58,182: expr = Expression(chunk.strip(), pos[0], pos[1],
2018-10-16 15:31:58,182: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/eval.py", line 93, in __init__
2018-10-16 15:31:58,182: self.code = _compile(node, self.source, mode=self.mode,
2018-10-16 15:31:58,183: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/eval.py", line 451, in _compile
2018-10-16 15:31:58,183: tree = xform().visit(node)
2018-10-16 15:31:58,183: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/astutil.py", line 794, in visit
2018-10-16 15:31:58,183: return visitor(node)
2018-10-16 15:31:58,183: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/astutil.py", line 816, in _clone
2018-10-16 15:31:58,183: value = self.visit(value)
2018-10-16 15:31:58,184: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/astutil.py", line 794, in visit
2018-10-16 15:31:58,184: return visitor(node)
2018-10-16 15:31:58,184: File "/home/haypo/bench_tmpdir/venv/lib/python3.8/site-packages/genshi/template/eval.py", line 616, in visit_Name
2018-10-16 15:31:58,184: strarg = _new(_ast.Str, node.id)
2018-10-16 15:31:58,184: AttributeError: module '_ast' has no attribute 'Str'
See also issue #74.
2019-12-17 01:04:41,423: [42/47] tornado_http...
2019-12-17 01:04:41,424: INFO:root:Running `/home/haypo/bench_tmpdir/venv/bin/python -u /home/haypo/pyperformance/pyperformance/benchmarks/bm_tornado_http.py --verbose --output /tmp/tmp8_0ujs9g`
2019-12-17 01:04:41,537: Traceback (most recent call last):
2019-12-17 01:04:41,537: File "/home/haypo/pyperformance/pyperformance/benchmarks/bm_tornado_http.py", line 15, in <module>
2019-12-17 01:04:41,537: from tornado.httpclient import AsyncHTTPClient
2019-12-17 01:04:41,537: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/tornado/httpclient.py", line 50, in <module>
2019-12-17 01:04:41,537: from tornado import gen, httputil, stack_context
2019-12-17 01:04:41,537: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/tornado/httputil.py", line 107, in <module>
2019-12-17 01:04:41,537: class HTTPHeaders(collections.MutableMapping):
2019-12-17 01:04:41,537: AttributeError: module 'collections' has no attribute 'MutableMapping'
and
2019-12-17 00:43:47,774: [ 6/47] django_template...
2019-12-17 00:43:47,775: INFO:root:Running `/home/haypo/bench_tmpdir/venv/bin/python -u /home/haypo/pyperformance/pyperformance/benchmarks/bm_django_template.py --verbose --output /tmp/tmp32v4efa1`
2019-12-17 00:43:47,925: Traceback (most recent call last):
2019-12-17 00:43:47,925: File "/home/haypo/pyperformance/pyperformance/benchmarks/bm_django_template.py", line 38, in <module>
2019-12-17 00:43:47,925: django.setup()
2019-12-17 00:43:47,925: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/__init__.py", line 18, in setup
2019-12-17 00:43:47,925: from django.urls import set_script_prefix
2019-12-17 00:43:47,925: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/urls/__init__.py", line 1, in <module>
2019-12-17 00:43:47,926: from .base import (
2019-12-17 00:43:47,926: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/urls/base.py", line 11, in <module>
2019-12-17 00:43:47,926: from .exceptions import NoReverseMatch, Resolver404
2019-12-17 00:43:47,926: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/urls/exceptions.py", line 3, in <module>
2019-12-17 00:43:47,926: from django.http import Http404
2019-12-17 00:43:47,926: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/http/__init__.py", line 5, in <module>
2019-12-17 00:43:47,926: from django.http.response import (
2019-12-17 00:43:47,926: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/http/response.py", line 13, in <module>
2019-12-17 00:43:47,926: from django.core.serializers.json import DjangoJSONEncoder
2019-12-17 00:43:47,926: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/core/serializers/__init__.py", line 23, in <module>
2019-12-17 00:43:47,926: from django.core.serializers.base import SerializerDoesNotExist
2019-12-17 00:43:47,927: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/core/serializers/base.py", line 4, in <module>
2019-12-17 00:43:47,927: from django.db import models
2019-12-17 00:43:47,927: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/db/models/__init__.py", line 5, in <module>
2019-12-17 00:43:47,927: from django.db.models.deletion import (
2019-12-17 00:43:47,927: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/db/models/deletion.py", line 5, in <module>
2019-12-17 00:43:47,927: from django.db.models import signals, sql
2019-12-17 00:43:47,927: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/db/models/sql/__init__.py", line 2, in <module>
2019-12-17 00:43:47,927: from django.db.models.sql.query import * # NOQA
2019-12-17 00:43:47,927: File "/home/haypo/bench_tmpdir/venv/lib/python3.9/site-packages/django/db/models/sql/query.py", line 11, in <module>
2019-12-17 00:43:47,927: from collections import Counter, Iterator, Mapping, OrderedDict
2019-12-17 00:43:47,927: ImportError: cannot import name 'Iterator' from 'collections' (/home/haypo/bench_tmpdir/prefix/lib/python3.9/collections/__init__.py)
2019-12-17 00:43:47,941: ERROR: Benchmark django_template failed: Benchmark died
When using performance
in my environment, it fails immediately due to an error:
Traceback (most recent call last):
File "/tmp/venv/lib/python2.7/site-packages/performance/benchmarks/bm_2to3.py", line 30, in
runner.bench_func('2to3', bench_2to3, command, devnull_out)
File "/tmp/venv/lib/python2.7/site-packages/perf/_runner.py", line 485, in bench_func
return self._main(task)
File "/tmp/venv/lib/python2.7/site-packages/perf/_runner.py", line 415, in _main
bench = self._worker(task)
File "/tmp/venv/lib/python2.7/site-packages/perf/_runner.py", line 389, in _worker
run = task.create_run()
File "/tmp/venv/lib/python2.7/site-packages/perf/_worker.py", line 293, in create_run
self.compute()
File "/tmp/venv/lib/python2.7/site-packages/perf/_worker.py", line 318, in compute
WorkerTask.compute(self)
File "/tmp/venv/lib/python2.7/site-packages/perf/_worker.py", line 285, in compute
metadata2 = self.collect_metadata()
File "/tmp/venv/lib/python2.7/site-packages/perf/_worker.py", line 350, in collect_metadata
return collect_metadata()
File "/tmp/venv/lib/python2.7/site-packages/perf/_collect_metadata.py", line 416, in collect_metadata
collect_cpu_metadata(metadata)
File "/tmp/venv/lib/python2.7/site-packages/perf/_collect_metadata.py", line 405, in collect_cpu_metadata
collect_cpu_config(metadata, all_cpus)
File "/tmp/venv/lib/python2.7/site-packages/perf/_collect_metadata.py", line 291, in collect_cpu_config
nohz_full = parse_cpu_list(nohz_full)
File "/tmp/venv/lib/python2.7/site-packages/perf/_cpu_utils.py", line 97, in parse_cpu_list
cpus.append(int(part))
ValueError: invalid literal for int() with base 10: ''
Tracing the problem I find the content of /sys/devices/system/cpu/nohz_full
is "\x00\n"
. So int("\x00")
fails.
I tried running the benchmark with python3 -m performance run --python ../cpython/3.6/python.exe --rigorous -b all -o ../3.6-perf.json
, but 3.6-perf.json
says "python_version":"3.5.2 (64-bit)"
, but running ../cpython/3.6/python.exe
says it's 3.6.0b1+
.
I think there is a typo in the assignment of the member Strength.STONG_PREFERRED
(which should probably be Strength.STRONG_PREFERRED
) in the deltablue benchmark script.
diff --git a/performance/benchmarks/bm_deltablue.py b/performance/benchmarks/bm_deltablue.py
index 798ef00..197beb4 100644
--- a/performance/benchmarks/bm_deltablue.py
+++ b/performance/benchmarks/bm_deltablue.py
@@ -81,7 +81,7 @@ class Strength(object):
# This is a terrible pattern IMO, but true to the original JS implementation.
Strength.REQUIRED = Strength(0, "required")
-Strength.STONG_PREFERRED = Strength(1, "strongPreferred")
+Strength.STRONG_PREFERRED = Strength(1, "strongPreferred")
Strength.PREFERRED = Strength(2, "preferred")
Strength.STRONG_DEFAULT = Strength(3, "strongDefault")
Strength.NORMAL = Strength(4, "normal")
I don't think this is a problem as that symbol is not used anywhere, but I think it would be nice to fix.
Thanks!
I can't use Python to download this repository or do anything useful with it. Please create a Python program so I can interact with this repository without having to use bizarre binaries written in C.
My workflow for PyPy is
upload
, which goes through compile.py
, but should it?.The cmd_upload
function in compile.py
assumes BenchmarkRevision.update_metadata
from compile.py
has been run to add commit information (commit_id
, commit_branch
, commit_date
) and possibly patch_file
to the json benchmark info. But because I am not running compile
this info is lacking. Possible solutions:
download_prebuilt
option to compile.py
and the Python
class to support this workflowupdate_metadata
method from BenchmarkRevision
to become part of every benchmark runAny thoughts? Should there be an upload.py
separate from compile.py
?
How to reproduce the issue
pyperformance compile_all test.conf
, where test.conf
contains something like:
[run_benchmark]
benchmarks = -tornado_http # single negative filter
What you expected to happen
All benchmarks except the specified one run.
What actually happens
...error!
... after pgo and a lot of stuffs ...
2021-10-21 01:34:06,050: The virtual environment /root/bench_tmpdir/venv has been created
2021-10-21 01:34:06,063: + /root/bench_tmpdir/prefix/bin/python3 -u -m pyperformance run --verbose --output /root/json/2021-10-17_23-20-base/main-54a4e1b53a18.json.gz --benchmarks -tornado_http --venv /root/bench_tmpdir/venv
2021-10-21 01:34:06,123: usage: __main__.py run [-h] [-r] [-f] [--debug-single-value] [-v] [-m]
2021-10-21 01:34:06,123: [--affinity CPU_LIST] [-o FILENAME] [--append FILENAME]
2021-10-21 01:34:06,123: [-b BM_LIST] [--inherit-environ VAR_LIST]
2021-10-21 01:34:06,123: [--inside-venv] [-p PYTHON] [--venv VENV]
2021-10-21 01:34:06,123: __main__.py run: error: argument -b/--benchmarks: expected one argument
2021-10-21 01:34:06,132: Command /root/bench_tmpdir/prefix/bin/python3 -u -m pyperformance run --verbose --output /root/json/2021-10-17_23-20-base/main-54a4e1b53a18.json.gz --benchmarks -tornado_http --venv /root/bench_tmpdir/venv failed with exit code 2
2021-10-21 01:34:06,132: Benchmark completed in 0:26:33.493049
2021-10-21 01:34:06,132: Benchmark failed but results written into /root/json/2021-10-17_23-20-base/main-54a4e1b53a18.json.gz
2021-10-21 01:34:06,145: Command /usr/bin/python3 -m pyperformance compile /root/pyperformance.conf 54a4e1b53a18f0c7420ba03de9608194c4413fc2 base/main --no-update --no-tune failed with exit code 12
2021-10-21 01:34:06,145: Benchmark exit code: 12
2021-10-21 01:34:06,145: FAILED: base/main-54a4e1b53a18f0c7420ba03de9608194c4413fc2
This looks trivial and I'll submit a PR to resolve this.
File "/home/haypo/performance/performance/benchmarks/bm_tornado_http.py", line 60, in bench_tornado
sock = make_http_server(make_application())
File "/home/haypo/performance/performance/benchmarks/bm_tornado_http.py", line 54, in make_http_server
server.add_sockets(sockets)
File "lib/python3.8/site-packages/tornado/tcpserver.py", line 157, in add_sockets
self._handlers[sock.fileno()] = add_accept_handler(
File "lib/python3.8/site-packages/tornado/netutil.py", line 268, in add_accept_handler
io_loop.add_handler(sock, accept_handler, IOLoop.READ)
File "lib/python3.8/site-packages/tornado/platform/asyncio.py", line 76, in add_handler
raise ValueError("fd %s added twice" % fd)
fd 3 added twice
In bm_chaos.py, in the definition of function GetKnots, I think that adding a list with a range object raises a TypeError in Python 3.x:
def GetKnots(points, degree):
knots = [0] * degree + range(1, len(points) - degree)
knots += [len(points) - degree] * degree
return knots
This function isn't called in the actual benchmark due to a test in Spline init. Could we change the first line of GetKnots with
knots = [0] * degree + list(range(1, len(points) - degree))
See http://bugs.python.org/issue26436 for details.
Proposed patch adds the regex-dna benchmark from The Computer Language Benchmarks Game (http://benchmarksgame.alioth.debian.org/). This is artificial but well known benchmark.
The patch is based on regex-dna Python 3 #5 program and fasta Python 3 #3 program (for generating input).
http://bugs.python.org/file42028/bm_regex_dna.patch
I have closed http://bugs.python.org/issue26436
I tried to upgrade tornado dependency from version 4.5.3 to 5.1.1, but it broke the benchmark:
File "performance/benchmarks/bm_tornado_http.py", line 61, in bench_tornado
sock = make_http_server(loop, make_application())
File "performance/benchmarks/bm_tornado_http.py", line 51, in make_http_server
server = HTTPServer(request_handler, io_loop=loop)
File "/home/vstinner/prog/python/performance/.tox/py3/lib/python3.6/site-packages/tornado/util.py", line 312, in __new__
instance.initialize(*args, **init_kwargs)
TypeError: initialize() got an unexpected keyword argument 'io_loop'
INFO:root:Running `C:\Python38\venv\cpython3.8-2459c2ec2c3d\Scripts\python.exe -u C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_tornado_http.py --output C:\Users\xy\AppData\Local\Temp\tmpdrgssiij`
Traceback (most recent call last):
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_tornado_http.py", line 96, in <module>
runner.bench_time_func('tornado_http', bench_tornado)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_runner.py", line 458, in bench_time_func
return self._main(task)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_runner.py", line 423, in _main
bench = self._worker(task)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_runner.py", line 397, in _worker
run = task.create_run()
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_worker.py", line 293, in create_run
self.compute()
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_worker.py", line 331, in compute
WorkerTask.compute(self)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_worker.py", line 280, in compute
self.calibrate_loops()
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_worker.py", line 243, in calibrate_loops
self._compute_values(self.warmups, nvalue,
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_worker.py", line 76, in _compute_values
raw_value = self.task_func(self, self.loops)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperf\_runner.py", line 454, in task_func
return time_func(loops, *args)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_tornado_http.py", line 60, in bench_tornado
server, sock = make_http_server(make_application())
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_tornado_http.py", line 54, in make_http_server
server.add_sockets(sockets)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\tornado\tcpserver.py", line 157, in add_sockets
self._handlers[sock.fileno()] = add_accept_handler(
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\tornado\netutil.py", line 268, in add_accept_handler
io_loop.add_handler(sock, accept_handler, IOLoop.READ)
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\tornado\platform\asyncio.py", line 79, in add_handler
self.asyncio_loop.add_reader(
File "C:\Python38\lib\asyncio\events.py", line 501, in add_reader
raise NotImplementedError
NotImplementedError
Hi,
I'm trying to run the bm_tornado_http.py with "tornado 6.0.3" and I got the following errors:
"
Traceback (most recent call last):
File "bm_tornado_http.py", line 17, in
from tornado.gen import coroutine, Task
ImportError: cannot import name 'Task'
"
I think it's caused by the wrong version of the tornado but I have no idea how to fix it.
Thanks
$ ./bin/python3.11 -m pyperformance run -o ~/python/pgo_lto_pyperformance_macros.json -v
(...)
creating build/temp.linux-x86_64-3.11/src/greenlet
gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/vstinner/python/main/install/venv/cpython3.11-527ffb5582f0/include -I/home/vstinner/python/main/install/include/python3.11 -c src/greenlet/greenlet.c -o build/temp.linux-x86_64-3.11/src/greenlet/greenlet.o
src/greenlet/greenlet.c: In function ‘g_switchstack’:
src/greenlet/greenlet.c:508:44: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
508 | current->recursion_depth = tstate->recursion_depth;
| ^~~~~~~~~~~~~~~
| recursion_limit
src/greenlet/greenlet.c:509:38: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
509 | current->top_frame = tstate->frame;
| ^~~~~
| cframe
src/greenlet/greenlet.c:544:17: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
544 | tstate->recursion_depth = target->recursion_depth;
| ^~~~~~~~~~~~~~~
| recursion_limit
src/greenlet/greenlet.c:545:17: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘frame’; did you mean ‘cframe’?
545 | tstate->frame = target->top_frame;
| ^~~~~
| cframe
src/greenlet/greenlet.c: In function ‘g_initialstub’:
src/greenlet/greenlet.c:821:50: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘recursion_depth’; did you mean ‘recursion_limit’?
821 | self->recursion_depth = PyThreadState_GET()->recursion_depth;
| ^~~~~~~~~~~~~~~
| recursion_limit
error: command '/usr/bin/gcc' failed with exit code 1
pyperformance venv create
has started failing for me recently, with the following message:
Execute: venv/cpython3.6-a7d80a339c76/bin/python -m pip install
ERROR: You must give at least one requirement to install (see "pip help install")
Command venv/cpython3.6-a7d80a339c76/bin/python -m pip install failed with exit code 1
I think this may be due to a behaviour change in pip 10.
I'm using performance v0.6.1 .
Full output below.
Creating the virtual environment venv/cpython3.6-a7d80a339c76
Execute: /home/mjw/opt/python3.6/bin/python3.6 -m venv --without-pip venv/cpython3.6-a7d80a339c76
Execute: venv/cpython3.6-a7d80a339c76/bin/python -c 'import sys; print(sys.hexversion)'
Python hexversion: 30602f0
Execute: venv/cpython3.6-a7d80a339c76/bin/python -m ensurepip --verbose
Ignoring indexes: https://pypi.python.org/simple
Collecting setuptools
0 location(s) to search for versions of setuptools:
Skipping link /tmp/tmpqak284a0 (from -f); not a file
Skipping link file:///tmp/tmpqak284a0/pip-9.0.1-py2.py3-none-any.whl; wrong project name (not setuptools)
Found link file:///tmp/tmpqak284a0/setuptools-28.8.0-py2.py3-none-any.whl, version: 28.8.0
Local files found: /tmp/tmpqak284a0/setuptools-28.8.0-py2.py3-none-any.whl
Using version 28.8.0 (newest of versions: 28.8.0)
Collecting pip
0 location(s) to search for versions of pip:
Found link file:///tmp/tmpqak284a0/pip-9.0.1-py2.py3-none-any.whl, version: 9.0.1
Skipping link file:///tmp/tmpqak284a0/setuptools-28.8.0-py2.py3-none-any.whl; wrong project name (not pip)
Local files found: /tmp/tmpqak284a0/pip-9.0.1-py2.py3-none-any.whl
Using version 9.0.1 (newest of versions: 9.0.1)
Installing collected packages: setuptools, pip
changing mode of /home/mjw/pyperf/venv/cpython3.6-a7d80a339c76/bin/easy_install-3.6 to 775
changing mode of /home/mjw/pyperf/venv/cpython3.6-a7d80a339c76/bin/pip3 to 775
changing mode of /home/mjw/pyperf/venv/cpython3.6-a7d80a339c76/bin/pip3.6 to 775
Successfully installed pip-9.0.1 setuptools-28.8.0
Cleaning up...
Execute: venv/cpython3.6-a7d80a339c76/bin/python -m pip --version
pip 9.0.1 from /home/mjw/pyperf/venv/cpython3.6-a7d80a339c76/lib/python3.6/site-packages (python 3.6)
Execute: venv/cpython3.6-a7d80a339c76/bin/python -m pip install -U 'setuptools>=18.5' 'pip>=6.0'
Collecting setuptools>=18.5
Using cached https://files.pythonhosted.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl
Collecting pip>=6.0
Using cached https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools, pip
Found existing installation: setuptools 28.8.0
Uninstalling setuptools-28.8.0:
Successfully uninstalled setuptools-28.8.0
Found existing installation: pip 9.0.1
Uninstalling pip-9.0.1:
Successfully uninstalled pip-9.0.1
Successfully installed pip-10.0.0 setuptools-39.0.1
Execute: venv/cpython3.6-a7d80a339c76/bin/python -m pip install -U wheel
Collecting wheel
Using cached https://files.pythonhosted.org/packages/1b/d2/22cde5ea9af055f81814f9f2545f5ed8a053eb749c08d186b369959189a8/wheel-0.31.0-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.31.0
Execute: venv/cpython3.6-a7d80a339c76/bin/python -m pip install
ERROR: You must give at least one requirement to install (see "pip help install")
Command venv/cpython3.6-a7d80a339c76/bin/python -m pip install failed with exit code 1
Remove directory venv/cpython3.6-a7d80a339c76
https://speed.python.org/about/ "About the benchmarks" has a link to https://hg.python.org/benchmarks whereas it should point to https://github.com/python/performance
Currently you have to modify this repo if you want to run a custom benchmark. It would be nice to have a mechanism by which a custom benchmark could be plugged in externally.
(This isn't a priority.)
Changes in Python 3.9 break html5lib, django_template, and tornado_http tests with this error:
File "/usr/local/src/pyperf-tot-no-venv/lib/python3.9/site-packages/html5lib/_tokenizer.py", line 16, in <module> from ._trie import Trie File "/usr/local/src/pyperf-tot-no-venv/lib/python3.9/site-packages/html5lib/_trie/__init__.py", line 3, in <module> from .py import Trie as PyTrie File "/usr/local/src/pyperf-tot-no-venv/lib/python3.9/site-packages/html5lib/_trie/py.py", line 6, in <module> from ._base import Trie as ABCTrie File "/usr/local/src/pyperf-tot-no-venv/lib/python3.9/site-packages/html5lib/_trie/_base.py", line 3, in <module> from collections import Mapping ImportError: cannot import name 'Mapping' from 'collections' (/root/py3-tot-no/lib/python3.9/collections/__init__.py) ERROR: Benchmark html5lib failed: Benchmark died
How to reproduce the issue
I've been building Python 3 from source and encountered it through the standard build process.
Build Python from source. I produced with git commit befa032d8869e0fab4732d910f3887642879d644 from cpython GitHub.
Run pyperformance with one of the following parameter sets:
/py3buildpath/bin/pyperformance run --python=/py3buildpath/bin/python3 --venv venvpath -b html5lib -o output.json
/py3buildpath/bin/pyperformance run --python=/py3buildpath/bin/python3 --venv venvpath -b django_template -o output.json
/py3buildpath/bin/pyperformance run --python=/py3buildpath/bin/python3 --venv venvpath -b tornado_http -o output.json
What you expected to happen
I expected these pyperformance benchmark to complete successfully and produce a result (without error).
What actually happens
Benchmark dies trying to import from collections.
I believe with Python 3.9 these packages need to be directly imported from collections.abc.
See this commit: python/cpython@ef092fe
Here's a patch that was backported to the Python 3 vendored library: pypa/pip@ef7ca14#diff-2496ad1eedee846e323ed2916d6c2d24
This library probably needs an official release so that the vendored patch set in Python 3 doesn't become bloated.
2019-06-06 08:14:37,403: [46/47] unpickle_pure_python...
2019-06-06 08:14:37,404: INFO:root:Running `/home/haypo/bench_tmpdir/venv/bin/python -u /home/haypo/pyperformance/pyperformance/benchmarks/bm_pickle.py --pure-python unpickle --verbose --output /tmp/tmp2jgtqf2c`
2019-06-06 08:14:37,471: Traceback (most recent call last):
2019-06-06 08:14:37,471: File "/home/haypo/pyperformance/pyperformance/benchmarks/bm_pickle.py", line 287, in <module>
2019-06-06 08:14:37,471: import pickle
2019-06-06 08:14:37,471: File "/home/haypo/bench_tmpdir/prefix/lib/python3.9/pickle.py", line 39, in <module>
2019-06-06 08:14:37,471: from _pickle import PickleBuffer
2019-06-06 08:14:37,471: ModuleNotFoundError: import of _pickle halted; None in sys.modules
2019-06-06 08:14:37,477: ERROR: Benchmark unpickle_pure_python failed: Benchmark died
Upstream issue closed as WONTFIX: https://bugs.python.org/issue37210
While looking for performance differences between CPython 2.7 and CPython 3.6, I noticed the same benchmark would exercise different parts of the interpreter.
pybench/Calls.py (PythonMethodCalls), pybench/Lookups.py (SpecialClassAttribute, NormalClassAttribute, SpecialInstanceAttribute, and NormalInstanceAttribute):
These benchmarks are using the syntax class c:
to create classes. This syntax creates old-styles classes in Python 2 and new-style classes in Python 3. Should we create separate tests for old-style classes (disabled in Python 3) and new-style classes?
pybench/Strings.py:
Due to the str
changes in Python 3, it is hard to use these benchmarks to compare between Python 2 and Python 3. While we could use bytes
in Python 3 (similar to how unicode
is being used Unicode.py), not use if that is the correct approach. Because, at that point, it would more of an ascii string benchmark rather than a str
.
Thanks!
With caching turn on in the .travis.yml
file, the base set of benchmarks should complete for a single run before Travis times out and kills the VM. That would be enough to make sure simple things like the django
-> django_template
rename breakage don't occur again.
If you run python bench.py
you trigger an exception:
Traceback (most recent call last):
File ".\bench.py", line 10, in <module>
benchmark.cli.main()
File "C:\Users\brcan\Documents\Repositories\benchmarks\benchmark\c
if not options.inside_venv:
AttributeError: 'Namespace' object has no attribute 'inside_venv'
INFO:root:Running `C:\Python38\venv\cpython3.8-2459c2ec2c3d\Scripts\python.exe -u C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_sqlalchemy_imperative.py --output C:\Users\xy\AppData\Local\Temp\tmpidjg6d0q`
Traceback (most recent call last):
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_sqlalchemy_imperative.py", line 4, in <module>
from sqlalchemy import Column, ForeignKey, Integer, String, Table, MetaData
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\sqlalchemy\__init__.py", line 8, in <module>
from . import util as _util # noqa
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\sqlalchemy\util\__init__.py", line 14, in <module>
from ._collections import coerce_generator_arg # noqa
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\sqlalchemy\util\_collections.py", line 16, in <module>
from .compat import binary_types
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\sqlalchemy\util\compat.py", line 331, in <module>
time_func = time.clock
AttributeError: module 'time' has no attribute 'clock'
ERROR: Benchmark sqlalchemy_imperative failed: Benchmark died
Steps to reproduce:
> pyperformance run --python="C:\User\GitHub\cpython\PCbuild\amd64\python.exe"
Creating the virtual environment venv\cpython3.10-3bfab7affa2d
...
... <installing packages>
...
INFO:root:Running `C:\User\GitHub\cpython\PCbuild\amd64\python.exe -u D:\User\Downloads\python\venv\cpython3.10-3bfab7affa2d\Lib\site-packages\pyperformance\benchmarks\bm_xml_etree.py --output C:\Users\User\AppData\Local\Temp\tmpo_57rqul`
Traceback (most recent call last):
File "D:\User\Downloads\python\venv\cpython3.10-3bfab7affa2d\Lib\site-packages\pyperformance\benchmarks\bm_xml_etree.py", line 17, in <module>
import pyperf
ModuleNotFoundError: No module named 'pyperf'
ERROR: Benchmark xml_etree failed: Benchmark died
Traceback ...
RuntimeError: Benchmark died
For some reason, pyperformance runs the benchmark by calling C:\User\GitHub\cpython\PCbuild\amd64\python.exe
. I think it should be calling the venv's python (venv\cpython3.10-3bfab7affa2d\Scripts\python.exe
) instead.
INFO:root:Running `C:\Python38\venv\cpython3.8-2459c2ec2c3d\Scripts\python.exe -u C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_mako.py --output C:\Users\xy\AppData\Local\Temp\tmpdezyn81_`
Traceback (most recent call last):
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\pyperformance\benchmarks\bm_mako.py", line 20, in <module>
from mako.template import Template # noqa
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\mako\template.py", line 10, in <module>
from mako.lexer import Lexer
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\mako\lexer.py", line 11, in <module>
from mako import parsetree, exceptions, compat
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\mako\parsetree.py", line 9, in <module>
from mako import exceptions, ast, util, filters, compat
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\mako\exceptions.py", line 11, in <module>
from mako import util, compat
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\mako\util.py", line 11, in <module>
from mako import compat
File "C:\Python38\venv\cpython3.8-2459c2ec2c3d\lib\site-packages\mako\compat.py", line 123, in <module>
time_func = time.clock
AttributeError: module 'time' has no attribute 'clock'
ERROR: Benchmark mako failed: Benchmark died
Bars on hystograms on https://speed.python.org/comparison/ are ordered in strange order: 2.7-3.6-3.5-master. Legends are ordered in inversed order: master-3.5-3.6-2.7. Both are misleading.
It would be better to support the order 2.7-3.5-3.6-master in both cases.
pyperformance/pyperformance/venv.py
Line 223 in a041ab2
hg_startup failed on the py3 job of Travis CI. I had to disable again this benchmark on Python 3. It's an issue related to virtual environment which is not specific to Mercurial.
Travis CI runs Python in a virtual environment created by the "virtualenv" command which overrides the site.py module.
runtests.py then creates a second virtual environment using "python3 -m venv" but this second venv inherits modules of the first venv, whereas pyperformance requires a virtual environment isolated from the system to get reproducible results.
Example without pyperformance:
# create venv1 and venv2
vstinner@apu$ virtualenv -p python3 venv1
vstinner@apu$ venv1/bin/python -m venv venv2
# install mercurial in venv1
vstinner@apu$ venv1/bin/python -m pip install mercurial # in venv1
# ... it's available in venv2!!!
vstinner@apu$ venv2/bin/python -c 'import mercurial; print(mercurial.__file__)' # in venv2
/home/vstinner/venv1/lib/python3.7/site-packages/mercurial/__init__.py
# venv1 uses /home/vstinner/venv1/lib/python3.7/site-packages path
vstinner@apu$ venv1/bin/python -m site
sys.path = [
'/home/vstinner',
'/usr/share/qa-tools/python-modules',
'/home/vstinner/venv1/lib64/python37.zip',
'/home/vstinner/venv1/lib64/python3.7',
'/home/vstinner/venv1/lib64/python3.7/lib-dynload',
'/usr/lib64/python3.7',
'/usr/lib/python3.7',
'/home/vstinner/venv1/lib/python3.7/site-packages',
]
USER_BASE: '/home/vstinner/.local' (exists)
USER_SITE: '/home/vstinner/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: False
# venv2 uses /home/vstinner/venv1/lib/python3.7/site-packages path as well!!!
vstinner@apu$ venv2/bin/python -m site
sys.path = [
'/home/vstinner',
'/usr/share/qa-tools/python-modules',
'/home/vstinner/venv1/lib64/python37.zip',
'/home/vstinner/venv1/lib64/python3.7',
'/home/vstinner/venv1/lib64/python3.7/lib-dynload',
'/usr/lib64/python3.7',
'/usr/lib/python3.7',
'/home/vstinner/venv1/lib/python3.7/site-packages',
]
USER_BASE: '/home/vstinner/.local' (exists)
USER_SITE: '/home/vstinner/.local/lib/python3.7/site-packages' (exists)
ENABLE_USER_SITE: False
# mercurial is installed in venv1...
vstinner@apu$ venv1/bin/python -m pip list
Package Version
---------- -------
mercurial 5.0.2
pip 19.2.1
setuptools 41.0.1
wheel 0.33.4
# ... but it's also "installed" in venv2!!!
vstinner@apu$ venv2/bin/python -m pip list
Package Version
---------- -------
mercurial 5.0.2
pip 19.2.1
setuptools 41.0.1
wheel 0.33.4
# it's even considered as a "local" install in venv2
vstinner@apu$ venv2/bin/python -m pip list --local
Package Version
---------- -------
mercurial 5.0.2
pip 19.2.1
setuptools 41.0.1
wheel 0.33.4
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.