mmanela / chutzpah

Chutzpah is an open source JavaScript test runner which enables you to run unit tests using QUnit, Jasmine, Mocha and TypeScript.

• 🍕 I’m Matthew Manela
• 🌱 I am a software engineering leader who is passionate about building high-performing teams and products
• 📝 I blog at http://matthewmanela.com about programming and running software teams
• 📫 You can reach out to me on 🐘 @[email protected] and 🐦 @mmanela
• 🌎 I built Idiomatically.net - A site for exploring idioms across languages and locales (and it's open source)
• 🪜 I made a word game for Android and iPhone called Anagram Ladder (and it's open source)
• 🎉 My most popular open source projects are DiffPlex - A .NET diffing library and Chutzpah - A JS Unit Test Runner
• 😄 Pronouns: he/him/his

chutzpah's Issues

Enable launching generated html test harness file in browser

mmanela[CodePlex]
Sometimes it is helpful to run the test harness in a browse to debug tests. When you run test directly from a JS file Chutzpah generates this file but keeps it hidden. A context menu option should be added to launch this file in a browser.

Command Line Runner to support Jasmine tests

benwoodcock13[CodePlex]
Enable the command line runner to work with jasmine unit tests

Add ability to scan for all test files under a folder and run them

mmanela[CodePlex]
It is common to have multiple test files and currently to run them you need to explicitly choose each using the /file command line argument which takes a file to run.

Update /file to also be able to take a folder where chutzpah will scan recursively for all test files within that folder.

Detecting a test file will be a simple heuristic where if a file contains something like test( we can assume it is a qunit test file.,

Cannot execute tests in a Windows 8 Consumer Preview Javascript Metro Style app

geyzer[CodePlex]
I downloaded Chutzpah via the VS2011 Beta Extension manager to execute my Jasmine tests for my Windows 8 JavaScript Metro-style application (named StoreMetro). It discovers my tests correctly, but when running them, it displays this error in the Test Console
Output:

------ Discover test started ------
Could not find file C:\git\StoreMetro\StoreMetro\bin\Debug\StoreMetro.
========== Discover test finished: 4 found (00:00:00.9589543) ==========
------ Run test started ------
The given key was not present in the dictionary.
========== Run test finished: 0 run (00:00:00.4989762) ==========

Tests fail if source file has the same name as test file

guilhermesalerno[CodePlex]
If you run a test file that has a dependecency with the same name all tests will fail because the dependency is not included. In the VS plugin that only happens the first time (I guess that the second time a name with a hash is generated) but on the command
line it repros consistently.

Detect and exclude qunit references

Qube[CodePlex]
If a script includes a reference to qunit.js for the sake of intellisense, do not include it in the test run. Having the script included twice (see below) causes some strange issues, and could throw off the validity of the tests (like if the version of
qunit differs between Chutzpah and the reference).

For example:

/// ltreference path=quot../scripts/lib/jquery-1.6.4-vsdoc.jsquot /gt
/// ltreference path=quot../scripts/lib/qunit.jsquot /gt
/// ltreference path=quot../scripts/my.library.jsquot /gt

Should not result in:
ltheadgt ltlink rel=quotstylesheetquot href=quotqunit.cssquot type=quottext/cssquot/gt ltscript type=quottext/javascriptquot src=quotqunit.jsquotgtlt/scriptgt ltscript type=quottext/javascriptquot src=quotmolu5nlu.diz_jquery-1.6.4-vsdoc.jsquotgtlt/scriptgt
ltscript type=quottext/javascriptquot src=quotwdmjm0hm.irn_qunit.jsquotgtlt/scriptgt ltscript type=quottext/javascriptquot src=quotla5xktlh.3rx_my.library.jsquotgtlt/scriptgt ltscript type=quottext/javascriptquot src=quotmy.library-test.jsquotgtlt/scriptgt
lt/headgt But should have the second qunit excluded like this:
ltheadgt ltlink rel=quotstylesheetquot href=quotqunit.cssquot type=quottext/cssquot/gt ltscript type=quottext/javascriptquot src=quotqunit.jsquotgtlt/scriptgt ltscript type=quottext/javascriptquot src=quotmolu5nlu.diz_jquery-1.6.4-vsdoc.jsquotgtlt/scriptgt
ltscript type=quottext/javascriptquot src=quotla5xktlh.3rx_my.library.jsquotgtlt/scriptgt ltscript type=quottext/javascriptquot src=quotmy.library-test.jsquotgtlt/scriptgt lt/headgt

Chutzpah does not run Asynchronous tests in Phantom if timeout in QUnit test is set to more than 3000 ms.

NV[CodePlex]
Chutzpah does not run Asynchronous tests in Phantom if timeout in QUnit test is set to more than 3000 ms.

This is for run not in browser, in browser works fine. Did not work even for simple test from QUnit site:

// A custom function
function ajax(successCallback) {
$.ajax({
url: 'server.php',
success: successCallback
});
}

//test
test('asynchronous test', function () {
// Pause the test
stop();

ajax(function () {
ok(true);
})

setTimeout(function () {
start();
}, 6000); //with it fails
})

Exception is:

Chutzpah Error Occured:
System.ArgumentNullException: Value cannot be null.
Parameter name: browserTestFileResult.BrowserOutput
at Chutzpah.TestResultsBuilder.Build(BrowserTestFileResult browserTestFileResult)

at Chutzpah.TestRunner.RunTestsFromHtmlFile(String headlessBrowserPath, String jsTestRunnerPath, TestContext testContext, List1 testResults, ITestMethodRunnerCallback callback)

at Chutzpah.TestRunner.RunTests(IEnumerable1 testPaths, TestOptions options, ITestMethodRunnerCallback callback)

Updated to 1.3 and asynctests no longer work

grandeconcarne[CodePlex]
I get quot0 passed, 0 failed, 0 total outputquot when running tests and quotObject reference not set to an instance of an object.quot when running tests in browser. I notice it creates a temp dir in Local/Temp but doesn't the directory does not
have any files in it.

Best regards,
Keith

Supporting running tests from the Solution Navigator

mmanela[CodePlex]
The Productivity power tools extension for Visual Studio adds a replacement for the Solution Explorer called the Solution Navigator. Chutzpah run correctly when you right click on a js test in the Solution Naviator and choose run js tests.

This behavior currently works as desired for the Solution Explorer

Console Runner hangs on html file

CraigMaslowski[CodePlex]
The attached simple html file runs fine in browser, but when run with the console runner, it just hangs.

I'm not sure why it's failing, but I did find that there should be an else condition in pageOpenHandler when checking status to exit phantom if page.open fails.

Support Jasmine-species (doesn't use it())

Dan15[CodePlex]
Chutzpah's Jasmine support seems pretty good, but it appears to detect Jasmine tests by searching for quotit(...)quot anywhere in the file (RegexPatterns.cs and JasmineDefinition.cs). Jasmine-species1
is an addin to Jasmine that adds additional test styles. Notably, it adds the ability to use BDD scenario (quotgiven, when, thenquot) syntax2.

Chutzpah doesn't detect these test files as Jasmine tests, and hence doesn't run them correctly. I managed to get it working by adding the correct references (via /// ltreference ...gt) and adding a comment like the following:

/// it('dummy')

However, this does seem like a hack. A short-term fix might be to search for quotgiven(...)quot and quotfeature(...)quot as well as quotit(...)quot.

Run in browser results in blank page for Jasmine Tests

mmanela[CodePlex]
A bug was introduced that causes quotrun test in browserquot to not work for jasmine.

Unexpected character encountered while parsing value: U. Line 1, position 1

NickJosevski[CodePlex]
When a file has a reference path link at the top, Chutzpah test runner falls over.

/// ltreference path=quot../../../f1/f2/file.jsquot /gt

Full error up on: http://pastebin.com/239vKC58

Create and publish Chutzpah as a NuGet package

mmanela[CodePlex]
Create a nuget package for chutzpah and publish it to nuget gallery

Prevent Phantom from deadlocking

mmanela[CodePlex]
There are certain situations where Phantom will hang and never die. Chutzpah should detect this and kill the process

Add /OpenInBrowser flag to console runner

mmanela[CodePlex]
Add an /OpenInBrowser flag to console runner which will in addition to running the tests also open your default browser with the test harness file.

Add support for proxy servers

mmanela[CodePlex]
There have been some reported issue of test files which reference external files (e.g
http://code.jquery.com/jquery.js) not being able to load the reference due to proxy servers.

Adding proxy support should solve this.

Fix referenced file expansion

mmanela[CodePlex]
Chutzpah 1.4.0 added supported for having referenced files in your tests that contain a references to other dependencies. However, Chutzpah is incorrectly trying to resolve these paths relative to the original test file rather than from the current referenced
file.

Capture running time of tests

mmanela[CodePlex]
Capture how long it takes to run the tests

Capture line number of each test name when you run a js file directly

mmanela[CodePlex]
Currently, Chutzpah doesn't return the line number of the passing or failing tests. This would be useful to have especially inside of Visual Studio since you can then double click on the error task and it can take you to the line in the source file.

Ideally this would take you to the failing assert but I think for simplicity this feature will map just to the beginning of the test.

For this item, only when you run chutzpah directly on the javascript file will it report the line numbers.

Running tests located in different folder than source

baxevanis[CodePlex]
Currently when I move the source to a different folder than test, all unit tests fail with message quotCan't find variable ...quot.

Upgrade to Phantom 1.2

mmanela[CodePlex]
Upgrade to phantom 1.2

Jasmine Support

Qube[CodePlex]
Add ability to run Jasmine unit tests the same way QUnit test currently run.

Add better support for RequireJS

mmanela[CodePlex]
Add better support for RequireJS for both HTML files and JS files

MSBuild Tasks

Qube[CodePlex]
Adding MSBuild tasks for Chutzpah would open the way for integration with other CI platforms such as CruiseControl.net or TFS Build.

Add ability to scan for files

mmanela[CodePlex]
Add an option to scan directories and sub directories for test files to run.

This should examine the files to see if they are test files.

Add better logging

mmanela[CodePlex]
Implement trace information throughout chutzpah to help enable better debugging.

Add doctype to test harness file

mmanela[CodePlex]
Add doctype to test harness file

Fix test output for QUnit ok assert

mmanela[CodePlex]
When a QUnit test has an quotokquot assert that fails the console runner ouputs:

Galleries.url.postAsyncWithCallback:Will execute callback [FAIL]
Test '[testname]' failed
Expected: undefined, Actual: undefined

This is less than ideal. Instead if should just says Assert Failed

Index out of range error using VS extension

sslotsky[CodePlex]
Using the Visual Studio extension, if I choose to run the tests in the browser, they work fine, but if I choose to run them (using Phantom), I get the error below. More details have been posted on StackOverflow:

http://stackoverflow.com/questions/9219236/index-out-of-range-error-using-chutzpahs-visual-studio-extension

------ Test started: File: C:\Users\U0120711\Documents\Visual Studio 2010\Projects\Behave\StaticContent\tests\Calculator\calculatorTest.js ------

Chutzpah Error Occured:
System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

Parameter name: index
at System.ThrowHelper.ThrowArgumentOutOfRangeException()
at Chutzpah.TestResultsBuilder.Build(BrowserTestFileResult browserTestFileResult)

at Chutzpah.TestRunner.RunTestsFromHtmlFile(String headlessBrowserPath, TestOptions options, TestContext testContext, List1 testResults, ITestMethodRunnerCallback callback)

at Chutzpah.TestRunner.RunTests(IEnumerable1 testPaths, TestOptions options, ITestMethodRunnerCallback callback)

While Running:C:\Users\U0120711\Documents\Visual Studio 2010\Projects\Behave\StaticContent\tests\Calculator\calculatorTest.js

========== Total Tests: 0 passed, 0 failed, 0 total ==========

test harness html that gets created removes qunit-fixture content

grandeconcarne[CodePlex]
The markup in qunit-fixture gets removed when the test.html is created by the runner.

Test discovery reports incorrect suite name for nested jasmine suites

mmanela[CodePlex]
When you perform test discovery on a jasmine file that has nested suites only the name of the deepest nested suite gets reported. However, if you perform test execution on the same test file it reports the suite name as the full name (included all parent
suites).

Update the test discovery to be consistent with the test execution naming.

Exception when run from nant

alex_razv[CodePlex]
Hi,

I want to integrate chutzpah with my build system but I got an exception bellow when try to run application from nant. Do you have any idea why it happens
[exec] Chutzpah console test runner (64-bit .NET 4.0.30319.237)
[exec] Copyright (C) 2011 Matthew Manela (http://matthewmanela.com).
[exec] chutzpah.dll: Version 2.0.0.136
[exec] Tests complete: 1
[exec] Tests complete: 2
[exec] Tests complete: 3
[exec] Tests complete: 4
[exec] System.AggregateException: One or more errors occurred. ---gt System.IO.IOException: The handle is invalid.
[exec] at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
[exec] at System.Console.GetBufferInfo(Boolean throwOnNoConsole, Booleanamp succeeded)
[exec] at Chutzpah.RunnerCallbacks.StandardConsoleRunnerCallback.ClearCounter()
[exec] at Chutzpah.RunnerCallbacks.StandardConsoleRunnerCallback.ExceptionThrown(Exception exception, String fileName)
[exec] at Chutzpah.Callbacks.ParallelRunnerCallbackAdapter.ExceptionThrown(Exception exception, String fileName)
[exec] at Chutzpah.TestRunner.ltgtc__DisplayClass2.ltProcessTestPathsgtb__1(String testFile)
[exec] at System.Threading.Tasks.Parallel.ltgtc__DisplayClass322.ltPartitionerForEachWorkergtb__30() [exec] at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask) [exec] at System.Threading.Tasks.Task.ltgtc__DisplayClass7.ltExecuteSelfReplicatinggtb__6(Object ) [exec] --- End of inner exception stack trace --- [exec] at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) [exec] at System.Threading.Tasks.Task.Wait() [exec] at System.Threading.Tasks.Parallel.PartitionerForEachWorker[TSource,TLocal](Partitioner1 source, ParallelOptions parallelOptions, Action1 simpleBody , Action2 bodyWithState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) [exec] at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body, Action2 bodyWi
thState, Action3 bodyWithStateAndIndex, Func4 bodyWithStateAndLocal, Func5 bodyWithEverything, Func1 localInit, Action1 localFinally) [exec] at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable1 source, ParallelOptions parallelOptions, Action1 body) [exec] at Chutzpah.TestRunner.ProcessTestPaths(IEnumerable1 testPaths, TestOptions options, TestRunnerMode testRunnerMode, ITestMethodRunnerCallback callbac
k)
[exec] at Chutzpah.TestRunner.RunTests(IEnumerable1 testPaths, TestOptions options, ITestMethodRunnerCallback callback) [exec] at Chutzpah.Program.RunTests(CommandLine commandLine) [exec] at Chutzpah.Program.Main(String[] args) [exec] ---gt (Inner Exception #0) System.IO.IOException: The handle is invalid. [exec] at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) [exec] at System.Console.GetBufferInfo(Boolean throwOnNoConsole, Booleanamp succeeded) [exec] at Chutzpah.RunnerCallbacks.StandardConsoleRunnerCallback.ClearCounter() [exec] at Chutzpah.RunnerCallbacks.StandardConsoleRunnerCallback.ExceptionThrown(Exception exception, String fileName) [exec] at Chutzpah.Callbacks.ParallelRunnerCallbackAdapter.ExceptionThrown(Exception exception, String fileName) [exec] at Chutzpah.TestRunner.ltgtc__DisplayClass2.ltProcessTestPathsgtb__1(String testFile) [exec] at System.Threading.Tasks.Parallel.ltgtc__DisplayClass322.ltPartitionerForEachWorkergtb__30()
[exec] at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
[exec] at System.Threading.Tasks.Task.ltgtc__DisplayClass7.ltExecuteSelfReplicatinggtb__6(Object )lt---

BUILD FAILED

Stream test results as they finish

mmanela[CodePlex]
Instead of waiting for a test file to finish before reporting the results Chutzpah should report when each test starts and finishes

Add support for Jasmine reporters to write to a file

mmanela[CodePlex]
The Jasmine-Reporters project (
https://github.com/larrymyers/jasmine-reporters) adds support to a Phantom Jasmine test runner which can write an xml report to a file. Their runner (https://github.com/larrymyers/jasmine-reporters/blob/master/test/phantomjs-testrunner.js)
accomplishes this by stashing the result in the window variable and then grabs it from the hosting page and writes it to disk once execution is finished.

To support this scenario we need to add similar logic to the jasmineRunner.js file in Chutzpah.

Use custom reporter instead of parsing HTML

Dan15[CodePlex]
jasmineRunner.js parses the HTML output from Jasmine in order to determine which tests have passed and which tests have failed. This works, but would be reliant on HTML output in a specific format.

Jasmine supports the notion of quotreportersquot, which are basically classes that take the Jasmine data and output it in a usable format. See jasmine-html.js for an example of a report (the default reporter that comes with Jasmine). Instead of parsing
HTML, a custom reporter could be used to directly exposes the required data (perhaps as a JSON object). This would be more reliable than parsing HTML.

Add options menu in Visual Studio

mmanela[CodePlex]
Initially this options menu will let the user set the timeout in milliseconds for a test file

Expose PhantomJS options

mmanela[CodePlex]
PhantomJS has several settings that can be useful for a user to toggle such as:

javascriptEnabled defines whether to execute the script in the page or not (default to true)

loadImages defines whether to load the inlined images or not

localToRemoteUrlAccessEnabled defines whether local resource (e.g. from file) can access remote URLs or not (default to false)

userAgent defines the user agent sent to server when the web page requests resources.

userName sets the user name used for HTTP authentication

password sets the password used for HTTP authentication

XSSAuditingEnabled defines whether load requests should be monitored for cross-site scripting attempts (default to false)

webSecurityEnabled defines whether web security should be enabled or not (default to `true)

These should be exposed from both the command line and the VS options page.

Support for retrieving JS dependencies through references.js

bpasero[CodePlex]
Hi,

would it be possible to allow to receive the list of dependencies of a test through 1 references.js file? In a larger project this would allow to keep dependencies in one place instead of having to update them across all tests.

It would work like this:

from a test require a references.js file (/// ltreference path=quot../references.jsquot /gt)
inside the references.js one would list all the dependencies using /// ltreference... as before

Thanks

No "test" output, no activity

nabog[CodePlex]

Downloaded and installed via VS Extension manager (Tools gtgt Extension Manager).
Restarted VS. Right-clicked on test.js file and selected quotRun JS Testsquot (file contains script: test(quotMathsquot, function () { equals(1, 4);});)
Does nothing. Checked Error List: Nothing. Checked output window: No item quottestquot in quotShow output fromquot box.
Right-clicked on test.js and selected quotRun JS Tests in Browserquot. Exception Message: quotObject reference not set to an instance of an object.quot
Re-started VS. Carried out steps above. Same result. Un-installed via Extension Manager. Installed via Nuget command line. Restarted VS. Same result. Un-installed via Nuget command line. Tried once more via VS Extension Manager. Same result. Un-installed.

I'm using VS 2010 SP1.

Upgrade to Phantom 1.3

mmanela[CodePlex]
Upgrade to the latest version of phantom

Preserve Dependency Order for referenced files

mmanela[CodePlex]
If a test references A.js. And A.js references file B.js. Then the test harness file that Chutzpah is generating places the script references in the order: A.js, B.js, followed by the test. This seems wrong, and it causes some of our scripts to throw exceptions.
The expected order in the test harness should be: B, A, test.

expected and actual result are not shown in Visual Studio

ewaldhofman[CodePlex]
When you run the tests with the chutzpah.console.exe it shows the expected and actual results on failing tests, but not in Visual Studio

Adding reference to a css file should add it to HTML test harness

mmanela[CodePlex]
If you add the following to your js file:

/// ltreference path=quotmatthew.cssquot /gt

Chutzpah should add the css to the generated test harness like:
ltlink rel=quotstylesheetquot href=quotmatthew.cssquot type=quottext/cssquot/gt

Currently it does the following:
ltscript type=quottext/javascriptquot src=quotmatthew.cssquotgtlt/scriptgt

Generated test harness does not run in FireFox 12+

mmanela[CodePlex]
I tested in FireFox 12+ and the generated test harness does not work correctly. It looks like it fails to load the test path which is a windows path (e.g. C:\somepath\file.js).

This same test harness works in IE and Chrome.

Change chutzpah.console.exe output to support VS error list integration

WizzApp[CodePlex]
Please change the output of the console to support integration in VS (see
http://blogs.msdn.com/b/msbuild/archive/2006/11/03/msbuild-visual-studio-aware-error-messages-and-message-formats.aspx). This allows chutzpah.console.exe to be used in a MSBuild Exec Task (for example to be executed on every build).

Alternatively you could also add an MSBuild task or an option for the extension to be run on every build.

Tests cannot be run from files marked ReadOnly

shadyproject[CodePlex]
When running tests from a file that is flagged as read only, the test fails because the file itself can't be copied to a temp directory.

We discovered this when running chutzpah against files that are stored in perforce. Removing the read only flag does solve this problem.

Prevent temporary folder explosion

mmanela[CodePlex]
Chutzpah creates a new temporary folder each time you run a test. This can add up quick and your temp folder can becomes quite large. Instead of using a random folder name take a hash of the test file path.

This will solve the issue as well as make the StagingFolder Test Option unnecessary so this will be removed.

Be able to run individual tests

mmanela[CodePlex]
Enable running of individual tests within a file.

Improve line number detection

mmanela[CodePlex]
Chutzpah uses a simple heuristic to try to match test's to line numbers.
For example, given a qunit test file it will scan the filing recording the line number of each instance of quottest(quot and then after tests are run match the test run to the index of each quottest(quot call.

This is a very fragile implementation that fails in a couple scenarios.

If you are using a framework that adds its own verbage for tests (like Pavalov.js) this matching completely fails.
If you wrap your test in a loop, Chutzpah will fail to give correct line numbers.

I have not thought about a more robust implementation yet, thoughts are welcome.