Thoughts on switching unit test framework? about prism

[Mike-E-angelo]

I am glad to hear about xunit's adoption. I personally feel it's the best unit-testing experience that I have encountered, particularly when paired with AutoFixture. 👍

[brianlagunas]

To be honest, I am not a very experienced tester. I have always been the implementer, and left the testing to others better suited for it. So I don't feel strongly about it.

I am not apposed to it, I would just need to learn how it works, and what dependencies would be required, as well as how much work it would take to rewrite/convert the tests we do have.

What are the benefits? The tests we have now seem to work, and it's not complicated to add more tests.

If we were to make a switch would you be willing to do the leg work to convert everything over :0)

[omares]

For the sake of completeness we should also consider nunit, which i would vote for :)

[sgrassie]

There is no cross platform support for MS Test, see: dotnet/msbuild#6, which seems to me like it would be a problem for the Xamarin.Forms support. Also, as I mentioned, all the other dotnet projects which have been open sourced by Microsoft have all (or are in the process of) migrating away from MS Test for xunit. That tells it's own story, and many other OSS .net projects all use xunit.

On the face of it, xunit works much the same as MS Test, and would require only the necessary dependencies for xunit itself. Replacing it would take a little bit of leg work in removing all the MS Test references/usings, adding xunit and perhaps adding a custom attribute or two.

I think the benefits speak for themselves. The existing tests do work (except for 3 which are failing, which I assume is due to a combination of ongoing refactoring work and a lack of CI).

If the switch does happen I would be happy to lend a hand.

I would also be in favour of using nunit, although I'm not sure on the cross platform support.

Interested in see what other people's thoughts are.

[omares]

nunit is the default unit test library when creating xamarin unit tests projects. Also we have zero issues with using nunit in our xamarin project (which we develop on mac). So there should be no cross platform issues at all :)

[briannoyes]

I'll play devil's advocate here and ask where is the added value in porting the extensive existing tests to NUnit or XUnit. What kinds of tests do we need that can't be done in MS Test? Given a green field start to the project I would not argue strongly in favor of MS Test over NUnit or XUnit, but I don't see any value in migrating. Everyone has their opinions on what the best test framework is, and most of the time it boils down to "I like the way this one does X better than that one" - and opinions without added value or that do not advance the capabilties of the framework are not something we have the time pursue. Come forward with concrete arguments about how MS Test is insufficient for the unit test needs of Prism and we can consider it. And then it would require someone who is a proponent of switching to do the work of migrating all the existing tests to the new framework and submitting that as a pull request once we had agreed to move to a different test framework.

[sgrassie]

Come forward with concrete arguments about how MS Test is insufficient for the unit test needs of Prism and we can consider it.

Any further comment is really dependent on knowing whether there is an intention in supporting cross-platform development/usage of Prism on Linux/Mac with Xamarin.Forms, as I don't want to waste any time repeating what so many others have already said elsewhere about this subject.

[omares]

MSTest is part of visual studio. Looking at xamarin there will be a user base that does not work on windows and have no access to MSTest. If this is not a subject, then dont fix something thats not broken. :) but you will lock out certain contributors.

Besides that, there are alot of comparisons and blog posts of people migrating away from mstest. One of the main reasons would be that MSTest did not evolve with the new technologies.

[paulomorgado]

I think the CoreClr team has come up with a way to translate the tests.

[briannoyes]

Fair points. We'll have to discuss how far into the field of supporting development of Prism.Forms on non-Windows machines we need to go.

[bartlannoeye]

I think development of Prism on non-Windows machines will be rather limited. You're only 'allowed' to change/fix Prism.Forms libraries. You can't even touch the base Prism PCL as you don't have the required WPF (PresentationCore, ...) assemblies to run the rest of the test suite to test your changes in the base library.

[brianlagunas]

@bartlannoeye You don't need to have PresentationCore for the base Prism PCL. Only if you are dealing with the Prism.Wpf assembly.

[bartlannoeye]

@brianlagunas you're right on not needing PresentationCore for the base PCL. But my point was that changing the base PCL might break a test in the WPF assembly (it shouldn't, but ...) and you won't know because you can't run that part of the test suite when developing on a non-Windows machine.

[brianlagunas]

@bartlannoeye Ahhh... yes, good point. When we start adding tests for the Xamarin.Forms projects, we will soon discover any issues with MSTest :0)

[brianlagunas]

So MSTest doesn't support Xamarin. I will give XUnit a try for the Xamarin support.

[sgrassie]

👍

[bartlannoeye]

Keep us posted!
I was planning to do some refactoring on the tests for the different IoC containers to get some of the duplicate code out (and add tests for Ninject), but I'll wait until we have a decision on the unit test framework.

[omares]

Just repeating again, i would go with nunit as its xunit based and "newer".

Quote from the nunit website.

The current production release, version 2.6, is the seventh major release of this xUnit based unit testing tool for Microsoft .NET.

[sgrassie]

@omares How do you qualify "newer"? I don't want to argue too much over this, because I like NUnit and xUnit.net in equal measure, however to avoid confusion I feel I must point out that your statement is incorrect.

To start with, NUnit 2.6.4 was released December 16 2014, and xunit 2.0 was released March 16 2015. Both are the latest stable versions.

Additionally, xUnit.net was based on lessons learned from NUnit 2.0, see Why did we build Xunit.

I wonder if your (apparent) confusion arises from both NUnit and xUnit.net being based on the xunit family of test frameworks?

The reason I pushed xUnit.net in this issue was that most (if not all) of the CoreCLR/.net OSS projects that have landed on Github out of Microsoft all seem to have embraced xUnit.net as their test framework (as do many more OSS projects), and I thought that because Prism has come out Microsoft as well, that following suit with xUnit.net would be most appropriate.

[omares]

@sgrassie yes you are absolutely right. Sorry for my nonsense comment. :(

[brianlagunas]

We have started moving to xUnit for tests. Prism.Tests has already been converted.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.