emilyst / vscode-test-explorer-diagnostics

A Visual Studio Code extension for Test Explorer UI which populates test outcomes as diagnostics in the Problems panel

Test Explorer Diagnostics Controller

This repository contains the source code for the Test Explorer Diagnostics Controller extension for Visual Studio Code. It requires the Test Explorer extension.

Using the test states known to the Test Explorer, it populates diagnostics in Visual Studio Code. These appear in the Problems panel. See the screenshot below as an example.

Status

This extension should be considered early alpha. It should work, but I have barely tested it in various scenarios, and much remains to do.

To do

• Handle retired test states fully.
• Add suggestions for preferences in Test Explorer to set.
• Discover proper ranges to provide diagnostics.
• Custom extension icon.
• Allow user to customize formatting of diagnostics.

vscode-test-explorer-diagnostics's Issues

show testcases with errors or exceptions

Gratz on your new extension!

It looks like your extension only shows diagnostics for failed tests and not for errored tests (I know it's fresh out of the factory, and you might already have plans for these errors).
Would be nice if they showed up as diagnostics as well.

I'm undecided if skipped testcases should show up in diagnostics.

What if there are multiple test event decorations?

vscode-test-explorer-diagnostics/src/controller.ts

Line 110 in 1d9dcb7

// TODO: What if there are multiple test event decorations?

vscode-test-explorer-diagnostics/src/controller.ts

Line 126 in 733e117

// TODO: What if there are multiple test event decorations?

support of tool-specific diagnostics

Thank you very much for your work, Emily! By coincedence this extension solves one of problems I have with my personal project. I've thought to make something similar myself, and it's really nice that I can focus on other tasks instead. Thank you very much indeed.

Would be nice if extension could also adress diagnostics specific for used unit-testing tool. It is especially useful for errored tests because, as I see it, error of test is an inherently tool-specific state. But it would be useful for other outcomes as well - tool can report warnings even if test passed.

Let me explain my use case: I work with ceedling test adapter (C language) and tests could be errored (currently they reported as failed, but I'm planning to fix it) if code syntax is wrong or if compilation or linking failed. Also I can get warnings if code is considered wrong but compiles anyway. All of these problems reported from used toolchain (GCC in my case) to stderr, which is then could be passed in TestEvent message field - as I understood from comment above this field it's intended for this exact purpose - reporting of problems.

VSC already has a mechanism for parsing such tool-specific problems - Tasks with problemMatchers. It's regex-based - output of tool is just scanned with regular expressions to find problems information - file, line, etc. Unfortunately it seems like it's not usable in this case (at least I've not figured a way to use it) - problemMatchers-related code is not accessible directly from extensions and Tasks are not not suitable because tool could be executed multiple times when running test suites - problems reported by Task would be cleared before each execution.

So, as I see it, the most proper way of implementing this would be to copy implementation of problemMatchers from VSC source code (it's in problemMatcher.ts file) to extension and make it configurable from extension settings. Algorithm is something like "if there is problem-matching patterns defined in extension settings, search for them in message field of every TestEvent and report diagnostics accordingly".

Now, why I think this should be in this extension rather than in tool-specific test adapters. First, adapters are not necessarily tool-specific - I use Ceedling tool, but I can swap underlying compiler and messages would be different. Second - if test adapters were handling their diagnostics themselves, this extension would not be needed, but it is - so it's only logical to implement this feature here.

Sorry for wall of text and thanks for your time. I intend to contribute btw.

What if TestInfo.file isn't just a string?

vscode-test-explorer-diagnostics/src/controller.ts

Line 95 in 1d9dcb7

if (info.file) { // TODO: What if TestInfo.file isn't just a string?

Allow user-customizable diagnostic message

vscode-test-explorer-diagnostics/src/controller.ts

Line 127 in 1d9dcb7

// TODO: Allow user-customizable diagnostic message

What do I do with retire events without tests defined?

https://github.com/emilyst/vscode-test-explorer-diagnostics/blob/master/src/controller.ts#L70-L70

How to get the true position of a test (event)?

vscode-test-explorer-diagnostics/src/controller.ts

Line 111 in 1d9dcb7

// TODO: How to get the true position of a test (event)?