Update API Documentation about scenarist

[t2ym]

API (Draft)

Suite class

Scope Object as a Suite class instance

• constructor(scope: string, description: string = scope + ' suite') - Create a scope typically in a block scope; The scope object is globally accessible via Suite.scopes[scope]

    { // "scope1" block scope
      let scope1 = new Suite('scope1', 'Description of scope1 Suite');
      scope1 === Suite.scopes.scope1; // is true
    }

• test property setter with a class expression - Define a test runner subclass or a test suite subclass in the scope

    let scope = 'example';
    let example = new Suite(scope, 'Description of Example Suite');
    // Defined test suite class is accessible via example.classes.ExampleSuite
    example.test = class ExampleSuite extends Suite {
      async setup() { await super.setup(); ... }
      async teardown() { await super.teardown(); ... }
    }
    // Defined test runner class is accessible via example.classes.ExampleTest1
    example.test = class ExampleTest1 extends example.classes.ExampleSuite {
      async operation() { ... }
      async checkpoint() { ... }
    }

• test property setter with a class expression mixin - Define a test runner class mixin in the scope

    // Defined test runner class mixin is accessible via example.mixins.ExampleTestMixin1
    example.test = (base) => class ExampleTestMixin1 extends base {
      async operation() { ... }
      async checkpoint() { ... }
    }

• test property setter with a test scenario object - Define test scenarios with a scenario object in the scope
  • Arrays in scenarios iterate over their items
  • Description can be optionally specified after ';' of the test class name
  • Default description string is generated by uncamelcasing the test class name, e.g., 'TestAAndB' -> 'test a and b'

    example.test = {
      // test class mixins
      '': {
        ExampleTestMixin1: {
          ExampleTestMixin2: 'Test1Then2' // Define Test1Then2 test class mixin
        }
      },
      // test classes
      ExampleTest1: {
        Test1Then2: 'Test1ThenTest1Then2' // Define Test1ThenTest1Then2 test class
      }
    }

    example.test = {
      // test class mixins
      '': [
        Suite.repeat('ExampleTestMixin1', 3, 'RepeatTest1_3Times'),
        Suite.repeat('ExampleTestMixin2', 4, 'RepeatTest2_4Times')
      ],
      // test classes
      ExampleTest1: {
        Test1Then2: 'Test1ThenTest1Then2; Description of the Test'
      }
    }

• testClasses(tests) instance method - Get Array of test classes
  • tests as number - List classes in CSV this.test[tests]
  • tests as CSV string - List classes in the CSV
• run(classes, target: string) instance method - Run the specified test classes in the scope; target is handed to constructors of target classes
  • classes as number - Target classes are testClasses(test[classes])
  • classes as CSV string - Target classes are testClasses(classes)
  • classes as Array of string - Target classes are classes.map((item) => self.classes[item])
  • classes as Array of classes - Target classes are classes
  • classes as object with class properties - Target classes are properties of classes

Scope Object Instance Properties other than test property setter

• scope string property - Scope name set by new Suite(scope)
• description string property - Default: scope + ' suite'; Scope description set by new Suite(scope, description)
• classes object property - Object containing the currently defined test classes
• mixins object property - Object containing the currently defined test class mixins
• classSyntaxSupport boolean property - true if ES6 class syntax is natively supported
• arrowFunctionSupport boolean property - true if ES6 arrow function syntax is natively supported
• leafClasses object property - Object containing the current leaf (non-redundant) test classes
• branchScenarios object property - Object with CSV strings of branch test scenarios as its properties
• test Array property - Array of CSV strings, each of which constitutes a group of reconnectable tests

    Suite.scopes.example.test returns
    [
      'ReconnectableTest1,ReconnectableTest2',
      'NonReconnectableTest1',
      'ReconnectableTest3,ReconnectableTest4',
      ...
    ]
    // Test page has to be reloaded for each reconnectable test group

    // In Driver page
    // example for web-component-tester
    var suites = [];
    for (var scope in Suite.scopes) {
      Suite.scopes[scope].test.forEach(function (tests, index) {
        suites.push(scope + '-test.html?TestSuites=' + index);
      });
    }
    WCT.loadSuites(suites);

Test Suite Subclass as a direct or descendent subclass of Suite class

• async setup() instance method - Overridden methods called once at the beginning of each running test via suiteSetup(); Overridden methods in subclasses should call await super.setup()
• async teardown() instance method - Overridden methods called once at the end of each running test via suiteTeardown(); Overridden methods in subclasses should call await super.teardown()

    // Define a common test suite as a direct subclass of Suite
    class CommonTestSuite extends Suite {
      async setup() { await super.setup(); ... }
      async teardown() { await super.teardown(); ... }
      ... // custom methods
    }
    { // example scope
      let example = new Suite('example', 'Example Subsuite');
      // [Optional] Define a test subsuite class in the 'example' scope 
      example.test = class ExampleSuite extends CommonTestSuite {
        async setup() { await super.setup(); ... }
        async teardown() { ...; await super.teardown(); } // may need teardown operation before super.teardown()
        ... // more custom methods for the subsuite
      }
    }

Test Runner Subclass as a subclass of Suite class

• constructor(target: string) - Instantiate a test runner; Optional for overriding
• async run() instance method - Run the test; Not for overriding
• * scenario() instance generator method - Iterate over tests in the reversed order of the prototype chain of the test; Yield { name: string, iteration: function, operation: function, checkpoint: function, ctor: function } for each test class

    // Simplest example without a test suite subclass and a scope
    class TestClass extends Suite {} // Define a test runner class
    let testRunner = new TestClass('#target'); // testRunner.target = '#target'
    testRunner.run(); // run the test

    // Example with a scope
    class ExampleSuite extends Suite { ... }
    { // example scope
      let example = new Suite('example');
      // Define a test class in the scope
      example.test = class ExampleTest1 extends ExampleSuite {
        async operation() { ... }
        async checkpoint() { ... }
      }
      // run the test
      (new example.classes.ExampleTest1('#target')).run();
    }

• async operation(parameters: optional) instance method - Perform operations of the test; parameters argument is omitted if *iteration() is not defined
• async checkpoint(parameters: optional) instance method - Perform test assertions of the test; parameters argument is omitted if *iteration() is not defined
• *iteration() instance generator method - [Optional] Provide parameters to operation and checkpoint methods

    class ExampleTest2 extends ExampleSuite {
      * iteration() { yield * [ 1, 2, 3 ]; }
      // parameters iterate through 1 to 3
      async operation(parameters) { ... }
      async checkpoint(parameters) { ... }
    }

Utility Instance Methods

• async forEvent(element: Element, type: string, trigger: function, condition: function) - Invoke the trigger() and wait for the event type for the element until condition(element: Element, type: string, event: Event) returns true

    async operation(parameters) {
      let self = this;
      // wait for 'track' event until the event state becomes 'end'
      await self.forEvent(self.dialog, 
        'track',
        () => { MockInteractions.track(self.dialog, parameters.dx, parameters.dy); },
        (element, type, event) => event.detail.state === 'end'
      );
    }

Utility Static Methods

• Suite.repeat(target: string, count: number, subclass: string/object) - Repetition operator for scenario objects

    Suite.repeat('TargetTest', 3, 'RepeatTarget3Times') returns
    { TargetTest: { TargetTest: { TargetTest: 'RepeatTarget3Times' } } }

• Suite.permute(targets: Array of string, subclass: function (scenario: Array of string)) - Permutation operator for scenario objects

    Suite.permute([ 'TestA', 'TestB', 'TestC' ],
      (scenario) => 'Test_' + scenario.map(n => n.replace(/^Test/,'')).join('_')
    ) returns
    {
      TestA: {
        TestB: { TestC: "Test_A_B_C" },
        TestC: { TestB: "Test_A_C_B" }
      },
      TestB: {
        TestA: { TestC: "Test_B_A_C" },
        TestC: { TestA: "Test_B_C_A" }
      },
      TestC: {
        TestB: { TestA: "Test_C_B_A" },
        TestA: { TestB: "Test_C_A_B" }
      }
    }

Static Properties for Suite class

• Suite.scopes static object property - Object containing scope objects as Suite.scopes[scope] such as Suite.scopes.example for 'example' scope

Static Properties for subclasses

• reconnectable static boolean read-only property - Default: true; Override the value as false to treat the test and its subclasses are not reconnectable and need page reloading to perform further tests

    class NonReconnectableSuite extends Suite {
      get reconnectable() { return false; }
      ...
    }

• skipAfterFailure static boolean read-only property - Default: false; Override the value as true to skip subsequent tests of the suite after an assertion failure

    class SkipAfterFailureSuite extends Suite {
      get skipAfterFailure() { return true; }
      ...
    }

TBD

• let scope = new Suite('scope', 'Description')
• scope.test = class TestClass extends scope.classes.BaseClass {...}
• scope.test = (base) => class TestClassMixin extends base {...}
• scope.test = {...} // scenario object
• scope.run(tests, target)
• Suite.repeat('TestClassMixin', count, 'TestClass')
• Suite.permute([ 'Test1', 'Test2', 'Test3' ], 'TestClass')
• TestClass.skipAfterFailure
• TestClass.reconnectable
• ...