t2ym / scenarist

Class-based branching test scenario runner for mocha

• 📫 @t2ym's OpenPGP public key for [email protected] is available at t2ym/thin-hook#407

scenarist's Issues

[Safari][README] Demo Animation GIF is not shown on Safari

With Safari, the demo animation GIF (scenaristdemo.gif) on README is not shown

[test][istanbul][Edge/Safari] Tests with istanbul hang up on 400 "Bad Request"

With Edge 13/14, and Safari 7/8/9/10, tests with istanbul hang up on 400 "Bad Request", although tests themselves on browsers succeed.

Workaround:

Perform tests without istanbul

[Safari TP Release 21] Experimental support of native ES6 modules

Experimental support of native ES6 modules with Safari Technical Preview Release 21

Notes:

• test/module/Suite.js has the same content as Suite.mjs
  • polyserve does not support the .mjs extension as a script type
• bower_components/stacky/browser.js needs this patch at line#45 since global is defined

var isNode = new Function('try {return this===global && window!==global;}catch(e){return false;}');

Handle underscores in uncamel

Convert underscores into spaces in uncamel()

[Polymer 2.x][app-location] Query parameters for HTML suites disappear

[Polymer 2.x][app-location] Query parameters for HTML suites disappear

Root Cause:

iron-location in app-location cannot handle query parameters properly with Polymer 2.x

Workaround:

• Use #hash tag instead of ?query_parameter to work around the issue.
• Use the patched @t2ym/web-component-tester (bower: t2ym/web-component-tester) with #hashtag support in html suites

Notes:

• Example test suites with the workaround: https://github.com/t2ym/live-localizer
• More descriptions on the issue: t2ym/live-localizer#58

Add optional exception handler method

Add optional exception handler method

• exception(reject: function, exception: Error) instance method - [Optional] Exception handler for errors outside of test callback function
  • If it calls reject(), it must return non-null to tell the runner not to call resolve()

    class ExampleTest3 extends ExampleSuite {
      async operation() { ... }
      async checkpoint() { ... }
      exception(reject, exception) {
        // default action when exception() is not defined
        reject(exception);
        return true;
      }
    }
    try {
      await (new ExampleTest3('#example')).run();
    }
    catch (exception) {
      // Handle exception in runner
      ...
    }

    class ExampleTest4 extends ExampleSuite {
      async operation() { ... }
      async checkpoint() { ... }
      exception(reject, exception) {
        // Treat the exception as a test failure by mocha
        test('exception on scenario', function() {
          throw exception;
        });
      }
    }

[test][Suite.js][istanbul] No coverage reports for Suite.js with native async/await

With Suite.js using native async/await, istanbul cannot generate coverage reports.

Root Cause:

The current version of JS parser esprima, on which istanbul depends, cannot properly parse async/await syntax.

Workaround (for Node):

• Step 1: npm install istanbul-espree
• Step 2: Patch node_modules/istanbul-espree/lib/instrumentor.js in istanbul-espree with ecmaVersion: 8 from ecmaVersion: 7 in ESP.parse()
• Step 3: Use Node 7: nvm use 7
• Step 4: node --harmony ./node_modules/istanbul-espree/lib/cli.js cover ./node_modules/mocha/bin/_mocha test/src/test.js --report lcov -- -R spec

Workaround (for WCT): Not Verified Yet

• Install node_modules/web-component-tester-istanbul/node_modules/istanbul-espree
• Patch it as above
• Rename istanbul-espree folder as istanbul
• Run on an async/await-ready browser like Chrome 55+

[Egde/IE11] Modified exception to reject() in exception() handler is not propagated to catch

On Egde and IE11, exception() handler is not called

Add explicit error handling for invalid identifiers

Add explicit error handling for invalid identifiers

Check existing names more rigidly to avoid misleading conflicts

Class name with the same name as the corresponding mixin name can be defined only once with the last mixin in the chain as the corresponding mixin.
Class name with the same name as an existing mixin name is not allowed unless the above condition is met.

error.test = (base) => class DefinedMixinBase3 extends base {}
error.test = (base) => class DefinedMixinBase4 extends base {}
error.test = (base) => class DefinedMixinBase5 extends base {}
error.test = {
  DummyTest: {
    DefinedMixinBase3: 'DefinedMixinBase3' // no error
  }
};
error.test = {
  DummyTest: {
    DefinedMixinBase5: '' // no error; defaults to 'DefinedMixinBase5'; allowed only once
  }
};
error.test = {
  DummyTest: {
    DefinedMixinBase3: 'DefinedMixinBase4' // error
  }
};

Mixin name with another existing class name is not allowed.

error.test = class DefinedClass1 extends Suite {}
error.test = (base) => class DefinedMixinBase6 extends base {}
error.test = (base) => class DefinedMixinBase7 extends base {}
error.test = {
  '': {
    DefinedMixinBase6: {
      DefinedMixinBase7: 'DefinedClass1' // error
    }
  }
};

Unable to catch multiple exceptions on run()

Unable to catch multiple exceptions/rejections from testInstance.run() on Suite.scopes[scope].run()

Root Cause:

Promise.all([ testInstance.run(), ... ]).catch() in run() can catch only the first exception.

for (var scope in Suite.scopes) {
  Suite.scopes[scope].test.forEach(function (tests, index) {
    Suite.scopes[scope].run(index)
      .catch(e => { /* Only the first exception in each run() can be caught */ }); 
  });
}

async function runner() {
  for (let scope in Suite.scopes) {
    for (let index in Suite.scopes[scope].test) {
      try {
        await Suite.scopes[scope].run(index);
      }
      catch (e) {
        /* Only the first exception in each run() can be caught */
      }
    }
  }
}()

[Edge 15/16][Suite.js] Class constructor cannot be called without the new keyword on subclassing

On Edge 15.14986 with Suite.js, "Class constructor cannot be called without the new keyword" error is thrown on subclassing. This should be a bug for Edge 15 prerelease build 14986.

Reproducible Code:

let example = Suite('example');
example.test = class ExampleSuite extends Suite {}

Workaround:

let example = Suite('example');
let t;
example.test = t = class ExampleSuite extends Suite {}

Compatibility with NodeJS

Compatibility with NodeJS.

Notes:

• node 7.3.0 has the optional support of async/await with the flag --harmony_async_await
• node 6.x can parse Babel ES5 transpiled codes of Suite.min.js and test classes with the help of babel-polyfill

[demo][gh-pages] path/index.html is responded with 404 Not Found

On GitHub pages, demo's path/index.html are responded with 404 Not Found.

Workaround:

Use the corresponding URL path/ instead.

Support module context

Support module context, i.e., non-global Suite class

// test.js
let assert = require('chai').assert;
let Suite = require('scenarist/Suite.min.js');
require('./scope1.js')(Suite, assert);
require('./scope2.js')(Suite, assert);

for (let scope in Suite.scopes) {
  Suite.scopes[scope].test.forEach(function (tests, index) {
    Suite.scopes[scope].run(index);
  });
}

// scope1.js
module.exports = function (Suite, assert) {
  let scope1 = new Suite('scope1');
  scope1.test = ...
  return scope1; // optional since scope1 === Suite.scopes.scope1
}

[Edge 15][Suite.js] No class mixin name test fails

On Edge 15.14986 with raw Suite.js, no class mixin name test fails.

no class mixin name 
‣
Error: expected [Function] to throw an error
              AssertionError at /components/chai/chai.js:5553
  Assertion.prototype.assert at /components/chai/chai.js:206
                assertThrows at /components/chai/chai.js:1665
                   ctx[name] at /components/chai/chai.js:4192
               assert.throws at /components/chai/chai.js:3271
          Anonymous function at error.js:43

assert.throws(function () {
  error.test = (base) => class extends base {}
}, /Suite[.]error: class mixin has no name /);

Polymer.dom() is hard-coded in forEvent()

Polymer.dom() is hard-coded in forEvent()

forEvent() should be applicable to general use without Polymer.

Compatibility with IE11

Since Suite.min.js is still using dynamically generated ES6 class syntax, IE11 cannot execute the test classes.

new Function('return class ...') has to be converted to ES5 classes

Leaf test class with a default name can unexpectedly be trimmed

Leaf test class which has the same name as its corresponding mixin can unexpectedly be trimmed even if it has a unique scenario.

[Suite.min.js] anonymous class has an invalid name _class

In transpiled Suite.min.js, anonymous class has an invalid name _class.

arrowFunctionSupport is always false

SuiteObjectInstance.arrowFunctionSupport is always false due to the incorrect condition

Support subscope

Support subscopes of scopes

Suite <-- GlobalScope <-- ScopeA <-- SubscopeAB ...
                                  +- SubscopeAC ...

[NodeJS] Support bdd ui

Support bdd ui for mocha on Node.js to omit -u tdd option

mocha test.js

or

mocha -u bdd test.js

[demo] Word '0' is not shown

In the scenario tree, word '0' occurrences are not shown. This should be a bug of Google Chart for the string '0'.

Support number in the first argument of run()

In addition to a number string and csv test names, support a number in the first argument of run()

run('10') fails

run('10') fails.

[Suite.min.js] TypeError: Class constructor cannot be invoked without 'new'

With Suite.min.js, the following error is thrown.

TypeError: Class constructor cannot be invoked without 'new'

Workaround:

let scope = new Suite('scope');
scope.classSyntaxSupport = false;

Suppress debug console.log() unless Suite.debug is true

Suppress debug console.log() unless Suite.debug is true

Add Suite.esm.js for ES module version in .js extension

Add Suite.esm.js for ES module version in .js extension

Fix

• Copy Suite.mjs to Suite.esm.js

Generate ES5 class without hard-coding Babel-transpiled codes

Generate ES5 class without hard-coding Babel-transpiled codes

Update API Documentation

Update API Documentation

[Suite.js] Show test class definitions on HTML reporter

With native ES6 Suite.js, show test class definitions on HTML reporter by overriding toString() of test functions.

[test] DeprecationWarning: Unhandled promise rejections are deprecated.

When is "the future"? How can the rejection be handled?

  Test iteration error test
    ✓ iteration generator error
(node:28970) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: iteration error
(node:28970) DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

  Test scenario error test
    ✓ scenario generator error
(node:28970) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 4): Error: scenario error