microsoft / vs-extension-testing

Integration test harness for Visual Studio extension development

Visual Studio Extension Testing

This project allows Visual Studio extension developers to write integration tests that run inside an experimental instance of Visual Studio.

Installation and Use

Requirements

• Extension development requires Visual Studio 2017 or newer. Version 15.7 or newer is recommended for the best Test Explorer experience.
• Extensions themselves must target one or more versions of Visual Studio from the following list:
  • Visual Studio 2012
  • Visual Studio 2013
  • Visual Studio 2015
  • Visual Studio 2017
  • Visual Studio 2019
  • Visual Studio 2022
• Extensions must be deployed via one or more VSIX packages.
• Test execution and debugging is only supported for versions of Visual Studio available on the same machine as the development IDE.

Install the test harness

Install the test package for the applicable version(s) of Visual Studio

Visual Studio Version	Integration Testing Package
2022	Microsoft.VisualStudio.Extensibility.Testing.Xunit
2012 - 2019	Microsoft.VisualStudio.Extensibility.Testing.Xunit.Legacy

Configure the test framework

Classic projects

Add the following to AssemblyInfo.cs to enable the test framework:

using Xunit;

[assembly: TestFramework("Xunit.Harness.IdeTestFramework", "Microsoft.VisualStudio.Extensibility.Testing.Xunit")]

SDK projects

💡 By default, SDK projects automatically generate the required assembly attribute. Manual customization is only required if the default assembly attributes support has been disabled, or in cases where the automatic application of TestFrameworkAttribute is not desired.

To disable generation of TestFrameworkAttribute (which will require manual addition similar to classic projects), add the following to the project file:

<PropertyGroup>
  <GenerateTestFrameworkAttribute>false</GenerateTestFrameworkAttribute>
</PropertyGroup>

Configure extensions for deployment

Add the following to AssemblyInfo.cs to deploy extensions required for testing.

using Xunit.Harness;

[assembly: RequireExtension("Extension.File.Name.vsix")]

Ensure test discovery is enabled

Test projects using a customized xUnit test framework cannot currently be discovered while tests are being written. The test discovery process that runs after a build completes will detect the required tests. Ensure this feature is enabled by the following steps:

1. Open Tools → Options...
2. Select the Test page on the left
3. Ensure Additionally discover tests from built assemblies after builds is checked

Tests will be automatically discovered and Test Explorer updated after each successful build.

Write tests

Apply the [IdeFact] attribute to tests that need to run in the IDE. After building the project, the tests will appear in Test Explorer where they can be launched for running and/or debugging directly.

Contributing

Please see CONTRIBUTING.md for information about our Code of Conduct and contributing guidelines.

vs-extension-testing's Issues

Crashes when starting experimental instance aren't captured

Had a misconfiguration in an extension (binding redirect for a DLL that wasn't being deployed) which meant the experimental instance crashed immediately on startup. The error was in the Activity Log but the experience of running the tests was just that they hung because IntegrationHelper.TryLocateDteForProcess has no timeout.

Would be good to have a timeout around https://github.com/microsoft/vs-extension-testing/blob/main/src/Microsoft.VisualStudio.Extensibility.Testing.Xunit.Shared/Harness/VisualStudioInstanceFactory.cs#L148

Gather diagnostics on test failures

• Screenshots
  • Capture a screenshot at the point of test failure (#59)
  • Attach the screenshot to the test results, if/when possible (xunit/xunit#936)
• System logs (possibly blocked on xunit/xunit#1613)
  • .NET Event logs (#59)
  • Watson logs (#59)
  • In-memory activity log (#61)

In-memory activity log is not captured

The in-memory activity log (added in 6eed3c6 )is not being captured in the EqualExceptionLegacy test.

Update README for new users

• Add license section
• Add Code of Conduct reference
• Add link(s) to the Visual Studio product where applicable

/cc @richlander

Consider using WCF instead of .NET Remoting

WCF is a more recent technology intended to replace .NET Remoting, which is considered obsolete. Switching to WCF would also address cases where .NET Remoting channels can interfere with one another, such as dotnet/roslyn#28513.

GitHub CI fails to attach test results

The test step reports this:

Results File: D:\a\vs-extension-testing\vs-extension-testing\src\Microsoft.VisualStudio.Extensibility.Testing.Xunit.IntegrationTests\TestResults\runneradmin_fv-az177-418_2021-08-02_15_11_36.trx

...

Results File: D:\a\vs-extension-testing\vs-extension-testing\src\Microsoft.VisualStudio.Extensibility.Testing.Xunit.Legacy.IntegrationTests\TestResults\runneradmin_fv-az177-418_2021-08-02_15_11_39.trx

But the Upload testResults step reports this:

Error: Path does not exist D:\a\vs-extension-testing\vs-extension-testing\obj\_artifacts\testResults
Error: Exit code 1 returned from process: file name 'c:\runners\2.279.0\bin\Runner.PluginHost.exe', arguments 'action "GitHub.Runner.Plugins.Artifact.PublishArtifact, Runner.Plugins"'.

/cc @AArnott this might be a problem with Library.Template

Remove hard-coded dependency paths

See #1 (comment)

Related to #8

Allow more granular skipping of tests via VS Version

Today the IDEFact will let you set a minimum and maximum VS version, but the granularity is only to the level of "2022, 2019, etc". It would be helpful in some instances to have more granular control than that. Ideally we'd be able to specify a specific build as Min or Max.

Capturing additional Logs on failure is difficult

By default lots of useful information is collected on failure (Screenshots, ActivityLogs, IDE state), but if a particular test or set of tests needs to collect additional info (in my case I'd like to capture the output from the Razor Language Server Output pane) it's not clear how that would be accomplished.

ACTION REQUIRED: Microsoft needs this private repository to complete compliance info

There are open compliance tasks that need to be reviewed for your vs-extension-testing repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your microsoft GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/microsoft/repos/vs-extension-testing/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim.

FYI: current admins at Microsoft include @wade0016, @Michael-Eng, @sharwell, @StephanieSu, @AlexEyler, @tinaschrepfer, @madskristensen, @SelmaI

Specify nuget feed on README

The README file tells us the package ID, but not the feed the package comes from. Since it isn't on nuget.org, this is crucial information to consuming the test framework.

Add instructions for use

Set up CI

• Set up project build
• Run tests during build
• Make test NuGet package available via MyGet (or equivalent)

Avoid starting a new instance every test?

Is there a way to avoid starting a new instance of devenv.exe every test? When migrating from VsixTesting.Xunit, my CI went from 10 minutes to 30. I suspect this is the reason.

As another side effect of this, I occasionally get the following error when running all tests.

System.InvalidOperationException : Test execution was skipped due to a prior exception in the harness.
---- System.InvalidOperationException : VSIX installer failed with exit code: -532462766

Unhandled Exception: Microsoft.VisualStudio.ExtensionManager.AlreadyInstalledException: Visual Studio Integration Test Service is already installed.
  at Microsoft.VisualStudio.ExtensionManager.InstallModule.CheckIfExtensionRequiresAdditionalOperationForInstall(IExtension extension, IInstalledExtensionList modifiedInstalledExtensionsList, Boolean installingAsNestedExtension, Boolean allowReplace, Boolean& performExtensionUpdate)
  at Microsoft.VisualStudio.ExtensionManager.InstallModule.ExtensionRequiresAdditionalOperationForInstall(InstallableExtensionImpl extension, IInstalledExtensionList modifiedInstalledExtensionsList, Boolean isNestedExtension, InstallFlags flags, Boolean& performExtensionUpdate)
  at Microsoft.VisualStudio.ExtensionManager.InstallModule.InstallInternal(InstallableExtensionImpl extension, InstallFlags installFlags, IDictionary`2 extensionsInstalledSoFar, List`1 extensionsUninstalledSoFar, IInstalledExtensionList modifiedInstalledExtensionsList, AsyncOperation asyncOp, IProgress`1 progress, Version targetedVsVersion)
  at Microsoft.VisualStudio.ExtensionManager.InstallModule.BeginInstall(IInstallableExtension installableExtension, InstallFlags installFlags, AsyncOperation asyncOp, Version targetedVsVersion)
  at Microsoft.VisualStudio.ExtensionManager.ExtensionEngineImpl.Install(IInstallableExtension extension, InstallFlags installFlags, Version targetedVsVersion)
  at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.Install(IInstallableExtension extension, Boolean perMachine)
  at Microsoft.VisualStudio.VsixInstaller.Installer.InstallImpl(IEnumerable`1 vsixFiles, String rootSuffix, String installationPath)
  at Microsoft.VisualStudio.VsixInstaller.Installer.Install(IEnumerable`1 vsixFiles, String installationPath, String rootSuffix)
  at Microsoft.VisualStudio.VsixInstaller.Installer.Main(String[] args)

Allow assembly-level [IdeSettings]

It's currently only allowed on test methods and classes. It would be nice to specify assembly-wide values:

[assembly: IdeSettings(RootSuffix = "Test")]

Automatically generate [RequireExtension] for referenced projects

Having to manually add it is rough--especially if you want to use the current Configuraiton (Debug or Release).

using Xunit.Harness;

#if DEBUG
[assembly: RequireExtension(@"..\..\..\..\..\VS.Data.Sqlite\bin\Debug\VisualStudio.Data.Sqlite.vsix")]
#else
[assembly: RequireExtension(@"..\..\..\..\..\VS.Data.Sqlite\bin\Release\VisualStudio.Data.Sqlite.vsix")]
#endif

...doing it in MSBuild is a bit better but still not great.

<ItemGroup>
  <AssemblyAttribute Include="Xunit.Harness.RequireExtension">
    <_Parameter1>..\..\..\..\..\VS.Data.Sqlite\bin\$(Configuration)\VisualStudio.Data.Sqlite.vsix</_Parameter1>
  </AssemblyAttribute>
</ItemGroup>

Consider using string[] instead of Array in interop types

See #1 (comment)

Investigate whether test collection messages can be passed through from child processes

See #1 (comment)

Exception thrown when debugging tests on latest Main

If I attempt to debug tests on latest VSMain I get the following error which causes the test to fail before I can see anything "interesting":

System.Runtime.InteropServices.COMException: 'Exception from HRESULT: 0x8971001E'

This is thrown from https://github.com/microsoft/vs-extension-testing/blob/main/src/Microsoft.VisualStudio.Extensibility.Testing.Xunit.Shared/Harness/VisualStudioInstance.cs#L45. This replicates both against the tests in this repo and those in razor-tooling.

Provide an extension mechanism for extensions installed by the test harness

See #1 (comment)

Investigate the `appDomain: denied` requirement

See #1 (comment).

If appDomain: denied is a requirement, it should be documented in the usage notes for the repository. Otherwise, the current use of this property should be removed in favor of the xUnit.net default.

Remove build dependency on Visual Studio 2012-2015

See #1 (comment)

Set up versioning

• Apply versioning policy to the repository
• Use strong names for assemblies

Cascading test failures due to a harness failure are not clearly indicated

When the test harness fails, remaining tests all fail with the same exception. This makes it difficult to distinguish the test that originally triggered the failure.

Custom logs are not (always?) collected when Initialize method fails

The Razor Integration tests currently have a failure that seems to happen only on the CI that's been hard to diagnose because it happens within AbstractRazorEditorTest.InitializeAsync and even though we make our RegisterCustomLogger calls at the top of the method they don't seem to get hit. I assume that has something to do with
this code seemingly special-casing exceptions thrown from Initialize, but I'm not clear on the precise behaviors here.

Here's one of the failures which is lacking the expected logs from our RegisterCustomLogger's.

Update IntegrationTestService extension to disallow publishing

See #1 (comment)

Can't find Xunit.Harness dependency

Hello,

I have done all the above steps (from docs), but VS does not find the dependency Xunit.Harness and related dependencies.

using Xunit.Harness;
[assembly: RequireExtension("Extension.File.Name.vsix")]
...

[IdeFact]

Could you please help? Maybe I miss some NuGet package or something? I added Xunit and Microsoft.VisualStudio.Extensibility.Testing.Xunit packages.

Thanks!

Best regards,
Alex

Doesn't work for Visual Studio 17.4.0 and 17.5 Preview

It seems that the testing is broken for just released VS 17.4.

From a short debug, something is going wrong inside

vs-extension-testing/src/Microsoft.VisualStudio.VsixInstaller.Shared/Installer.cs

Line 102 in 0158b24

if (extensionManager.IsInstalled(extension))

VsixInstaller is just hanged here.

During debug I can see the following exception:
System.Runtime.InteropServices.COMException: 'Element not found. (Exception from HRESULT: 0x80070490)'
in

vs-extension-testing/src/Microsoft.VisualStudio.VsixInstaller.Shared/Installer.cs

Line 96 in 0158b24

var extensionManager = new ExtensionManagerService(settingsManager);

Also, during assembly resolving inside the HandleAssemblyResolve method it's failed to find the
Microsoft.VisualStudio.ExtensionEngine.XmlSerializers, Version=17.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
assembly.

I can reproduce the VsixInstaller.17 hang when running the TestOpenAndCloseIDE test inside IntegrationTests.

Things was working well for 17.3 and for at least some versions of 17.4 Preview(didn't run integrations tests for a latest Preview update).

Could not load System.Collections.Immutable version 5.0.0

I'm getting this error on VS 17.6 Preview 3.

System.InvalidOperationException : Test did not run due to a harness failure.
---- System.IO.FileNotFoundException : Could not load file or assembly 'System.Collections.Immutable, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
Stack Trace:
  
  ----- Inner Stack Trace -----
     at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.Start[TStateMachine](TStateMachine& stateMachine)
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
  --- End of stack trace from previous location where exception was thrown ---
     at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
     at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()

Manually adding a reference to it from the test project makes it go away.

<PackageReference Include="System.Collections.Immutable" Version="5.0.0" />