Add project element to top-level sub-project before merging about nunit-console

[CharliePoole]

The TestPackage is actually created by the runner. The engine fills in subprojects. I haven't seen anything wrong so far in how the engine does that. In fact, in my GUI issue, I'm relying on the TestPackage structure to provide the extra level of info that is missing from the XML that is returned from the engine. IIRC, we have tests that check the package structure for various combos of assemblies and projects.

The trick is for the engine to use the package structure so as to create a project node when it aggregates the assemblies under a project. That's what it isn't doing. When I run my GUI solution file, I get back a test-run element with 11 child test-suite elements of type Assembly.I should bet back a project node in between the test run and it's children.

[CharliePoole]

Looks good. I'll take this one and also add comments for some of the esoteric bits that are liable to confuse.

[CharliePoole]

I've just been bitten by this one in TestCentric. When opening a single project, containing multiple assemblies, there is no top-level project element present. This is actuallly the "One Project" place I commented on above, which I thought was working.

See TestCentric/testcentric-gui#159

[ChrisMaddock]

@CharliePoole could you help my understand what the issue is here, in terms of TestPackage structure? i.e. What we currently have, and what would be desirable? I'm assuming when we talk about "project element's" here, that relates directly to test package structure? Or is this a separate issue?

[CharliePoole]

By "project" I mean anything the engine recognizes as a project through one of the extensions, like NUnit projects or Visual Studio projects and solutions.

Projects cannot currently be nested, so they can only appear at the top level... command-line or constructed in the GUI by adding more files to those that are open.

Whenever there is a project, there was supposed to be a <test-suite type='Project'/> node in the tree.

There is code to create a project node in the engine, but I'm not seeing any being created. I haven't explored all combinations of projects and assemblies to be sure. Back when Rob commented, it seems like we were only creating a project node when there was a single project, by itself on the command-line. I know we are not doing so now when I open a solution file.

The test package structure derives from what is entered on the command-line or may be created in other ways in a GUI. Because of that tie-in, I was able to deduce what the engine should have produced in the GUI issue and patch the test tree to have the necessary nodes. That's a real hack, of course.

[ChrisMaddock]

@CharliePoole So just to confirm - the TestPackage structure here is currently correct, but there is a failure in the xml produced?

I ask, as I'm currently looking at TestPackage structure, but not 100% clear on where the TestPackage structure is a) incorrect itself, or b) incorrectly handled.

[CharliePoole]

To be clear... every runner has a test package that it is running it may be just an assembly or a collection of assemblies. If the test package is recognized as a project, then a project node should be created on top of the returned assembly or assemblies. We need tests for this but they would be fairly high level, i.e. loading one or more assemblies.

[CharliePoole]

This continues to be an issue for TestCentric GUI. As far as I can tell, there are no circumstances in which project elements are produced.

[CharliePoole]

Currently, the only code in the engine that creates a project element is the oddly named MasterTestRunner.PrepareResult. It creates a project element IFF the top level element is a project, which in fact it never is, rather the top level element is the anonymous command-line package, which may contain one or more projects.

In principle,depending on settings, any of the TestEngineRunner results may be for a project or part of a project. So, in theory, this could be handled at a lower level or at the master runner level using more complex code.

The original change, cited at the top of this issue, was working at first for cases with a single project, by itself. That's because we used to handle the case of a top-level single project specially, replacing the anonymous command-line package. That created some pretty obscure code and at some point it got removed, breaking what is essentially an untested feature.

IMO, we need some true, separate integration tests that actually run the engine and examine the results for all these cases. I think this belongs in a separate project, so I'm not going to try to create them now for this issue.