anupli / running-ng Goto Github PK

View Code? Open in Web Editor NEW

4.0 5.0 7.0 2.53 MB

Running next generation

Home Page: https://anupli.github.io/running-ng/

License: Apache License 2.0

Python 100.00%

methodology benchmarking benchmark-framework garbage-collection

running-ng's Introduction

`running-ng`

running-ng is a collection of scripts that help people run workloads in a methodologically sound settings.

Installation

pip3 install running-ng

There are two extras available.

zulip: dependencies for the Zulip runbms plugin, useful for users.
tests: dependencies for running tests, useful for package developers.

Development setup

virtualenv env
source env/bin/activate
pip install -e .[zulip,tests]
pip install -U pip setuptools build[virtualenv] twine # extra packages for building releases

To make a distribution archives, run python -m build within the virtual environment.
To install to user site-packages, run pip install dist/running_ng-<VERSION>-py3-none-any.whl outside the virtual environment.
To update to PyPI, run twine upload --repository running-ng dist/*<VERSION>*.

Documentation

Please refer to this site for up-to-date documentations.

License

This project is licensed under the Apache License, Version 2.0.

running-ng's People

Contributors

Stargazers

Watchers

Forkers

tianleq k-sareen angussidney qinsoon wks shamouda enesgunes1

running-ng's Issues

Use a singleton class for global states like dry_run

Support initial values/guesses for minheap

If we already know the ballpark minheap for a benchmark, we should be able to determine the minheap much quicker.

cc @udesou

Zulip integration

Potentially add Zulip integration [1, 2] such that you get a private message when a benchmark or run is completed.

[1]: https://zulip.com/api/
[2]: https://zulip.com/api/send-message

Add an option to run a command to check the execution environment before running

Support specifying precision for minheap

For example, during the bisect, if low and high are close enough, the algorithm can terminate early. For backward compatibility, the default resolution is 1M.

cc @udesou

Currently we support wrapper commands. It might make sense to also add companion commands that are launched before the main program, and are SIGINTed after the main program exits. This is useful, e.g., for running eBPF tracing.

Use `shlex.quote`

https://docs.python.org/3/library/shlex.html#shlex.quote to replace https://github.com/caizixian/running-ng/blob/112f2cbf5606f0b46a825da99622479b60b42cf7/running/util.py#L46

Audit default settings of `runbms`

Return code is 0 for ValueError

For example, with the following error, we should expect running to return a non-zero value. When we use running for CI, a zero return value is considered as pass, and can easily be neglected unless we look into the logs.

Run id: linux-x86_64-batik-fastdebug-fv-az477-706-2023-09-13-Wed-024258
Traceback (most recent call last):
  File "/home/runner/.local/bin/running", line 8, in <module>
    sys.exit(main())
  File "/home/runner/.local/lib/python3.10/site-packages/running/__main__.py", line 48, in main
    if m.run(args):
  File "/home/runner/.local/lib/python3.10/site-packages/running/command/runbms.py", line 383, in run
    configuration.resolve_class()
  File "/home/runner/.local/lib/python3.10/site-packages/running/config.py", line [46](https://github.com/mmtk/mmtk-openjdk/actions/runs/6167075597/job/16738052124?pr=241#step:15:47), in resolve_class
    self.__items[cls_name] = load_class(
  File "/home/runner/.local/lib/python3.10/site-packages/running/config.py", line 14, in load_class
    return {k: cls.from_config(k, v) for (k, v) in config.items()}
  File "/home/runner/.local/lib/python3.10/site-packages/running/config.py", line 14, in <dictcomp>
    return {k: cls.from_config(k, v) for (k, v) in config.items()}
  File "/home/runner/.local/lib/python3.10/site-packages/running/suite.py", line [47](https://github.com/mmtk/mmtk-openjdk/actions/runs/6167075597/job/16738052124?pr=241#step:15:48), in from_config
    return BenchmarkSuite.CLS_MAPPING[config["type"]](name=name, **config)
  File "/home/runner/.local/lib/python3.10/site-packages/running/suite.py", line 109, in __init__
    raise ValueError(
ValueError: DaCapo release rc not recongized

Fix the hardcoded timeout in `minheap`

Instead, use get_timeout on the JavaBenchmarkSuite

Refactoring out DaCapo hacks into the DaCapo class

When running DaCapo with callbacks (by overriding the callback of an instance of DaCapo), it uses -cp instead of -jar to run DaCapo, which bypasses the MANIFEST file in the jar, causing issues like below

java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.dacapo.harness.Cassandra.prepare(Cassandra.java:124)
    at org.dacapo.harness.Benchmark.run(Benchmark.java:205)
    at org.dacapo.harness.TestHarness.runBenchmark(TestHarness.java:225)
    at org.dacapo.harness.TestHarness.main(TestHarness.java:170)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at Harness.main(Unknown Source)
Caused by: java.lang.ExceptionInInitializerError
    at org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1193)
    at org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:550)
    at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:350)
    at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:178)
    at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:162)
    at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:816)
    at org.apache.cassandra.service.EmbeddedCassandraService.start(EmbeddedCassandraService.java:49)
    ... 13 more
Caused by: java.lang.RuntimeException: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@78fbff54/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @158da8e)
    at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:107)
    ... 20 more
Caused by: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@78fbff54/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @158da8e)
    at java.base/java.lang.invoke.MemberName.makeAccessException(MemberName.java:942)
    at java.base/java.lang.invoke.MethodHandles$Lookup.checkAccess(MethodHandles.java:2201)
    at java.base/java.lang.invoke.MethodHandles$Lookup.checkMethod(MethodHandles.java:2141)
    at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(MethodHandles.java:2285)
    at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(MethodHandles.java:2278)
    at java.base/java.lang.invoke.MethodHandles$Lookup.unreflect(MethodHandles.java:1747)
    at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:96)

The current workaround changes JavaBenchmark instead of DaCapo, which can be confusing for people using other Java benchmark suites. This DaCapo specific hack should be moved to DaCapo

running-ng/src/running/benchmark.py

Line 208 in 4a8a9a0

if isinstance(runtime, OpenJDK):

This DaCapo specific hack should be moved to DaCapo, and be conditional on whether -jar is used.

running-ng/src/running/suite.py

Line 182 in 4a8a9a0

if self.callback:

Code comment and documentation need to be added.

runbms should collect image timestamp when running on moma machines

Support time estimation for the hfac/benchmark once the first invocation has finished

Introduce a version check

Allow config files to declare the minimum running-ng version required

Deprecate some builtin modifiers

Some of the flags, especially the initial set of OpenJDK related flags are purely artifacts of the limitation of early versions of the MMTk OpenJDK binding.

This is now becoming a headache as unsuspecting people using these default modifier can introduce methodological errors (such as triggering GC before the benchmark timing iteration or not).
Many of the numerical values are arbitrary and should either be removed or turned into modifiers with value arguments.

This is potentially a breaking change if people rely on the behaviors of these modifiers.

Support specifying a config matrix similar to GitHub Actions matrix

Overriding invocations does not affect dry run.

I tried to have the following:

overrides:
  invocations: 5

while the runbms defines invocations: 10.

An actual run does 5 invocations, but a dry run still does 10.

Implement Zulip plugin for minheap

Support mixing wildcard wrapper and benchmark specific wrapper

Some benchmarks require specific wrapper (e.g. chart requires xfvb-run).
Wildcard wrapper (taskset or perf stat) needs to be compatible with benchmark specific wrappers instead of overwriting it

Add slice/mega mode

Sometimes we want to specify a particular hfac (e.g. 1500 or 1.5x, 9999 or 9.999x) to, say have a very large heap so that we will only perform nursery collection.
Previously, it's supported as -s and -m through the command lines.

Common modifiers for configs

For all the 'configs' in a run, usually there are some common modifiers that ensure each config is measured with the same methodology. For example, using the same probe, using the same command line arguments, etc. It would be good if we can set common modifiers for configs.

Disable OpenJDK GC weak references processing

MMTk GCs (with OpenJDK binding) do not support weak-ref processing at the moment. For a fair comparison when running benchmarks, we should find a way to (effectively) disable OpenJDK GC's weak reference processing and make them treat weak-refs as strong.

minheap: use `INFO` to tell users when benchmarks are skipped when the results already exist in the output file

Improve error handling

Add more perf sanity check

Check people logged in. If anyone other than the current user is logged in, emit a warning.
Before each invocation, dump the output from w, which contains the users logged in and the recent loads on the machine
Print the current CPU usage from vmstat. https://askubuntu.com/questions/274349/getting-cpu-usage-realtime
Print the top processes wrt the CPU usage. https://unix.stackexchange.com/questions/30129/how-do-i-get-top-command-to-wrap-its-output

Expand environment variables in companion commands

running-ng/src/running/benchmark.py

Line 130 in 28190fc

self.companion, stdout=subprocess.PIPE, stderr=subprocess.STDOUT

Line 218 in b74e3a1

return __DEFAULT_MINHEAP

with importlib.resources.path(__package__, "config") as config_path:

Also see:
python/cpython#88328
python/importlib_resources#85

Reported by @rkennke

Support more flexible heap size modifiers

Proposed by @wks

modifiers:
  heap2x:
    type: HeapSize
    multiplier: 2
  heap64m:
    type: HeapSize
    size: "64M"
  heapdyn:
    type: HeapSize
    min: "1M"
    max: "128G"
configs:
  - "build1|mmtk_gc-Immix|heap2x"
  - "build1|mmtk_gc-Immix|heap64m"
  - "build1|mmtk_gc-Immix|heapdyn"

The special modifier type HeapSize overrides the heap size setting given on the command line (if any). If the multipler: key is given, the heap size will be computed from the min heap size provided by dacapo.yml. If size: is given, it is fixed heap size. If max: and min: are given, it will be dynamic heap size.

Filter out empty modifiers in configs

Right now, if you have a config like openjdk7||foobar, you will get a confusing error message KeyError: "Modifier '' not defined". We could filter out empty modifiers.

Get rid of `--add-opens` workaround, and turn it into a modifier shipped with running-ng

It's currently not sufficient.

  h2o21:
    type: JVMArg
    val: "-Dsys.ai.h2o.debug.allowJavaVersions=21 --add-opens java.base/java.lang=ALL-UNNAMED"
    includes:
      dacapochopin-rc5:
        - h2o
  cassandra21:
    type: JVMArg
    val: "-Djava.security.manager=allow --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED"
    includes:
      dacapochopin-rc5:
        - cassandra
  trades21:
    type: JVMArg
    val: "--add-opens java.base/java.util=ALL-UNNAMED"
    includes:
      dacapochopin-rc5:
        - tradebeans
        - tradesoap

Support different invocations for different benchmarks

Also see #14

Report currently logged in users when emitting warnings

running-ng/src/running/command/runbms.py

Line 223 in f61018d

if len(get_logged_in_users()) > 1: