scala / scala-partest Goto Github PK

Each run (and jvm) test always forks a fresh JVM; this costs a lot in startup time, and we're paying that cost about 1800 times. @retronym observes that tests that actually need this level of isolation are probably relatively few. Most of the time we should be enough to stay in partest's own JVM and just isolate each test in a fresh classloader. We'll need to capture stdout and stderr and trap System.exit.

@vpetro, @dwijnand, @retronym and I wrote wip code for this today at a Lightbend engineering retreat. Our preliminary results indicate an average overall speedup of about 2x. If that holds, this will substantially speed up PR validation.

PR to follow. It may take a while to identify tests that need the old level of isolation. (We've already seen cases where test A does something later causing test B to fail, so it isn't a trivial matter of just running any failed test the old way.)

See https://scala-ci.typesafe.com/job/dotty-master-validate-partest/1128/console the tests finished with:

[error] Interrupted waiting for command to finish (java -Xms64M -Xmx1024M ...)
[info] Thread pool timeout elapsed before all tests were complete!!!
646 - run/t6253a.scala                          [non-zero exit code]
[info] 
[info] 1504/1505 passed, 1 skipped (elapsed time: 04:11:41)
[info] Test Run PASSED
[success] Total time: 15105 s, completed Feb 23, 2016 4:00:29 PM

which means that from the github ui this failure is completely silent.

The .flags file should allow

-Dscala.repl.prompt="%nevil scala> "

because otherwise it's just evil.

Why are we so terrified of parsing quoted strings? It's the third rail of parsing.

As originally pointed out by @gkossakowski, these dependencies should not come with a partest dependency, but should instead be provided by the partest user, as partest is supposed to test the provided compiler.

@retronym explains how to do this in sbt:

libraryDependencies += "net.liftweb" %% "lift-json" % "2.4" % "provided"
You can always wield pomPostProcess on the producer side: http://www.scala-sbt.org/0.13.0/docs/Detailed-Topics/Publishing#modifying-the-generated-pom

Seems (anecdotally) that maven-ant does the right thing: http://mail-archives.apache.org/mod_mbox/maven-users/200809.mbox/%[email protected]%3E

it should at least print what the delta was that exceeded the threshold

we should probably also look into any JMH-level of robustness memory profiling/benchmarking tools out there

In sbt:

set version := "1.0.0-local"
publishM2

Then, over in the scala/scala checkout,
edit versions.properties so that partest.version.number=1.0.0-local.
rm build/pack/partest.properties to have the build recreate the partest classpath, and run the test suite using test/partest.

Partest tries to use java.dll as the java command by default

I expect to provide a PR to fix that this evening

Sometimes I don't have time for the run tests to complete locally, but it would be nice to at least compile them before asking Jenkins to give it a run. I propose --dryrun to mean compile-only the executable tests, run and jvm.

at #35 @lrytz suggests: "Could we switch to https://github.com/sbt/sbt-git? Then we don't need to keep updating the version after each release. Here's how it is configured in scala-asm: https://github.com/scala/scala-asm/blob/master/build.sbt#L5-L7"

Under --show-diff, things like these happen:

!! 1 - neg/t7494-no-options                      [output differs]
% diff /Users/tschlatt/Documents/scala-js/scalalib/fetchedSources/2.11.2/test/files/neg/t7494-no-options-neg.log /Users/tschlatt/Documents/scala-js/partest-suite/target/scala-2.11/test-classes/scala/tools/partest/scalajs/2.11.2/neg/t7494-no-options.check
@@ -5,27 +5,36 @@ error: Error: ploogin takes no options
          namer   2  resolve names, attach symbols to named trees
 packageobjects   3  load package objects
          typer   4  the meat and potatoes: type the trees
-     jsinterop   5  
-        patmat   6  translate match expressions
-superaccessors   7  add super accessors in traits and nested classes
-    extmethods   8  add extension methods for inline classes
-       pickler   9  serialize symbol tables
-     refchecks  10  reference/override checking, translate nested objects
-       uncurry  11  uncurry, translate function values to anonymous classes
-     tailcalls  12  replace tail calls by jumps
-    specialize  13  @specialized-driven class and method specialization
- explicitouter  14  this refs to outer pointers
-       erasure  15  erase types, add interfaces for traits
-   posterasure  16  clean up erased inline classes
-      lazyvals  17  allocate bitmaps, translate lazy vals into lazified defs
-    lambdalift  18  move nested functions to top level
-  constructors  19  move field definitions into constructors
-       flatten  20  eliminate inner classes
-         mixin  21  mixin composition
-        jscode  22  
-       cleanup  23  platform-specific cleanups, generate reflective calls
-    delambdafy  24  remove lambdas
-         icode  25  generate portable intermediate code
-           jvm  26  generate JVM bytecode
-       ploogin  27  A sample phase that does so many things it's kind of hard...
-      terminal  28  the last phase during a compilation run
+        patmat   5  translate match expressions
+superaccessors   6  add super accessors in traits and nested classes
+    extmethods   7  add extension methods for inline classes
+       pickler   8  serialize symbol tables
+     refchecks   9  reference/override checking, translate nested objects
+       uncurry  10  uncurry, translate function values to anonymous classes
+     tailcalls  11  replace tail calls by jumps
+    specialize  12  @specialized-driven class and method specialization
+ explicitouter  13  this refs to outer pointers
+       erasure  14  erase types, add interfaces for traits
+   posterasure  15  clean up erased inline classes
+      lazyvals  16  allocate bitmaps, translate lazy vals into lazified defs
+    lambdalift  17  move nested functions to top level
+  constructors  18  move field definitions into constructors
+       flatten  19  eliminate inner classes
+         mixin  20  mixin composition
+       cleanup  21  platform-specific cleanups, generate reflective calls
+    delambdafy  22  remove lambdas
+         icode  23  generate portable intermediate code
+#partest !-optimise
+           jvm  24  generate JVM bytecode
+       ploogin  25  A sample phase that does so many things it's kind of hard...
+      terminal  26  the last phase during a compilation run
+#partest -optimise
+       inliner  24  optimization: do inlining
+inlinehandlers  25  optimization: inline exception handlers
+      closelim  26  optimization: eliminate uncalled closures
+      constopt  27  optimization: optimize null and other constants
+           dce  28  optimization: eliminate dead code
+           jvm  29  generate JVM bytecode
+       ploogin  30  A sample phase that does so many things it's kind of hard...
+      terminal  31  the last phase during a compilation run
+#partest

Note the #partest lines. They should be processed just like when comparing, before doing the diff.

That thing where it filters out non-tests and if there's nothing left, runs all tests? Really annoying when I happen to mistype. Some paths do look especially like tests, so if one of those is not an existing file, then please don't test the world.

In scala/scala#6436, scala-xml was dropped from the scala compiler build.

Turns out, when the compiler bootstrap script tries to build partest it fails to find a scala-xml version that satisfies the scalaBinaryVersion.

However, the dependency on scala-xml here is only "provided". It's not clear why it even needs to be "provided". Removing the dependency from build.sbt, it will still compile. Usually, "provided" suggests there are code paths that require it, but the user should provide it themselves. But I'm not seeing any code paths that require scala-xml. Perhaps, there is a use case in partest that requires scala-xml that's not obvious?

If not, could the provided-dependency on scala-xml just be ripped out?

Maybe, when scala-xml was part of scaladoc (as far back as yesterday), it was important to probably sprinkle the dependency everywhere, including here.

An old comment in build.sbt hints as much:

scalap depends on scala-compiler, which depends (for the scaladoc part) on scala-xml
and scala-parser-combinators more precisely, scala-compiler_2.11.0-M5 depends on
                     scala-xml_2.11.0-M4 and
      scala-parser-combinators_2.11.0-M4,
so that we get a binary version incompatibility warning
To fix this, we'll modularize scaladoc to remove the dependency from scala-compiler-core,
and use dbuild to replicate the staged build we had originally,
so that we don't mix cross-versioned artifacts.

such as by printing a little report of the top 5, in descending order by runtime, that kind of thing.

an idea of @retronym's

this release is meant for a version of scala that in-sourced scalacheck

It looks like cwd is on the classpath when partest runs. A REPL test fails because it tries to run method test imported from a previous line (under tweaked name binding). Presumably test package is introduced by ./test/files.

scala> for (i <- 1 to 8) test(i)
<console>:16: error: reference to test is ambiguous;
it is both defined in package <root> and imported subsequently by
import test
       for (i <- 1 to 8) test(i)
                         ^

the + vs - direction is the reverse of what you would expect

following the lead of scala/scala-xml#242, scala/scala-swing#91, and others

Quoting @sjrd from gkossakowski/scala#2 explaining reasons for forking partest:

At least, we had to replace
https://github.com/scala/scala-partest-interface/blob/master/src/main/scala/tools/partest/TestingInterface.scala#L75
by
https://github.com/scala-js/scala-js/blob/master/partest/src/main/scala/scala/tools/partest/scalajs/PartestInterface.scala#L97
and
https://github.com/scala/scala-partest-interface/blob/master/src/main/scala/tools/partest/TestingInterface.scala#L90
by
https://github.com/scala-js/scala-js/blob/master/partest/src/main/scala/scala/tools/partest/scalajs/PartestInterface.scala#L113

I don't remember if there were other things we needed to change.

The task would be to make those things configurable so Scala.js could tweak whatever it needs to tweak instead of forking the whole project.

Partest shouldn't pick up just any old java command from the environment.

I've always had alias partest8="JAVACMD='${SCALA_JAVACMD}' JAVA_HOME='${SCALA_JAVA_HOME}' ./test/partest" which seems a bit much.

The partest repo at the moment has the following branches

• 1.0.x
• 1.1.x
• 2.12.0
• bump-1.1.2
• runCommand-public
• topic/sbt-partest-read

It's not documented what these are for, and I suspect several of them could be deleted.