dotnet / roslyn-tools

Tools used in Roslyn based repos

dotnet/dotnet - Home of the .NET VMR

This repository is a Virtual Monolithic Repository (VMR) which includes all the source code and infrastructure needed to build the .NET SDK.

What this means:

• Monolithic - a join of multiple repositories that make up the whole product, such as dotnet/runtime or dotnet/sdk.
• Virtual - a mirror (not replacement) of product repos where sources from those repositories are synchronized into.

In the VMR, you can find:

• source files of each product repository which are mirrored inside of their respective directories under src/,
• tooling that enables building the whole .NET product from source on Linux platforms,
• small customizations, in the form of patches, applied on top of the original code to make the build possible,
• [in future] E2E tests for the whole .NET product.

Just like the development repositories, the VMR will have a release branch for every feature band (e.g. release/8.0.1xx). Similarly, VMR's main branch will follow main branches of product repositories (see Synchronization Based on Declared Dependencies).

More in-depth documentation about the VMR can be found in VMR Design And Operation. See also dotnet/source-build for more information about our whole-product source-build.

Goals

• The main purpose of the dotnet/dotnet repository is to have all source code necessary to build the .NET product available in one repository and identified by a single commit.
• The VMR also aims to become the place from which we release and service future versions of .NET to reduce the complexity of the product construction process. This should allow our partners and and 3rd parties to easily build, test and modify .NET using their custom infrastructure as well as make the process available to the community.
• Lastly, we hope to solve other problems that the current multi-repo setup brings:
  • Enable the standard down-/up-stream open-source model.
  • Fulfill requirements of .NET distro builders such as RedHat or Canonical to natively include .NET in their distribution repositories.
  • Simplify scenarios such as client-run testing of bug fixes and improvements. The build should work in an offline environment too for certain platforms.
  • Enable developers to make and test changes spanning multiple repositories.
  • More efficient pipeline for security fixes during the CVE pre-disclosure process.

We will achieve these goals while keeping active coding work in the separate repos where it happens today. For example: ASP.NET features will continue to be developed in dotnet/aspnetcore and CLR features will be continue to be developed in dotnet/runtime. Each of these repos have their own distinct communities and processes, and aggregating development into a true mono-repo would work against that. Hence, the "virtual" monolithic repo: the VMR gives us the simplicity of a mono-repo for building and servicing the product, while active development of components of that product stays in its various existing repos. The day to day experience for typical contributors will not change.

Limitations

This is a work-in-progress. There are considerable limitations to what is possible at the moment. For an extensive list of current limitations, please see Temporary Mechanics.
See the Unified Build roadmap for more details.

Supported platforms

• 8.0
  • source-build configuration on Linux
• 9.0+ (WIP)
  • source-build configuration on Linux
  • non-source-build configuration on Linux, Mac, and Windows

For the latest information about Source-Build support for new .NET versions, please check our GitHub Discussions page for announcements.

Code flow

For the time being, the source code only flows one way - from the development repos into the VMR. More details on this process:

• Source Synchronization Process
• Synchronization Based on Declared Dependencies
• Moving Code and Dependencies between the VMR and Development Repos

We expect the code flow to start working both ways at the completion of the Unified Build project. See the Unified Build roadmap for more details.

Contribution

At this time, the VMR will not accept any changes and is a read-only mirror of the development repositories only. Please, make the changes in the respective development repositories (e.g., dotnet/runtime or dotnet/sdk) and they will get synchronized into the VMR automatically.

Dev instructions

Please note that this repository is a work-in-progress and there are some usability issues connected to this. These can be nuisances such as some checked-in files getting modified by the build itself and similar. For the latest information about Source-Build support, please watch for announcements posted on our GitHub Discussions page.

Prerequisites

The dependencies for building can be found here. In case you don't want to / cannot prepare your environment per the requirements, consider using Docker.

Building

1. Clone the repository

   git clone https://github.com/dotnet/dotnet dotnet-dotnet
   cd dotnet-dotnet

2. Build the .NET SDK

   Choose one of the following build modes:

   • Microsoft based build

     For Unix:

     ./build.sh --clean-while-building

     For Windows:

     .\build.cmd -cleanWhileBuilding

   • Building from source

     # Prep the source to build on your distro.
     # This downloads a .NET SDK and a number of .NET packages needed to build .NET from source.
     ./prep-source-build.sh
     
     # Build the .NET SDK
     ./build.sh -sb --clean-while-building

   The resulting SDK is placed at artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz (for Unix) or artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].zip (for Windows).

3. (Optional) Unpack and install the .NET SDK

   For Unix:

   mkdir -p $HOME/dotnet
   tar zxf artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz -C $HOME/dotnet
   ln -s $HOME/dotnet/dotnet /usr/bin/dotnet

   For Windows:

   mkdir %userprofile%\dotnet
   tar -xf artifacts/assets/Release/dotnet-sdk-9.0.100-[your RID].zip -C %userprofile%\dotnet
   set "PATH=%userprofile%\dotnet;%PATH%"

   To test your built SDK, run the following:

   dotnet --info

Note

Run ./build.sh --help (for Unix) or .\build.cmd -help (for Windows) to see more information about supported build options.

Building using Docker

You can also build the repository using a Docker image which has the required prerequisites inside. The example below creates a Docker volume named vmr and clones and builds the VMR there.

docker run --rm -it -v vmr:/vmr -w /vmr mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream9
git clone https://github.com/dotnet/dotnet .

# - Microsoft based build
./build.sh --clean-while-building

# - Building from source
./prep-source-build.sh && ./build.sh -sb --clean-while-building

mkdir -p $HOME/.dotnet
tar -zxf artifacts/assets/Release/dotnet-sdk-9.0.100-centos.9-x64.tar.gz -C $HOME/.dotnet
ln -s $HOME/.dotnet/dotnet /usr/bin/dotnet

Codespaces

You can also utilize GitHub Codespaces where you can find preset containers in this repository.

Building from released sources

You can also build from sources (and not from a context of a git repository), such as the ones you can acquire from a dotnet/dotnet release. In this case, you need to provide additional information which includes the original repository and commit hash the code was built from so that the SDK can provide a better debugging experience (think the Step into.. functionality). Usually, this means the dotnet/dotnet repository together with the commit the release tag is connected to.

In practice, this means that when calling the main build script, you need to provide additional arguments when building outside of a context of a git repository.
Alternatively, you can also provide a manifest file where this information can be read from. This file (release.json) can be found attached with the dotnet/dotnet release.

Synchronizing code into the VMR

Sometimes you want to make a change in a repository and test that change in the VMR. You could of course make the change in the VMR directly (locally, as the VMR is read-only for now) but in case it's already available in your repository, you can synchronize it into the VMR (again locally).

To do this, you can either start a dotnet/dotnet Codespace - you will see instructions right after it starts. Alternatively, you can clone the repository locally and use the vmr-sync.sh or vmr-sync.ps1 script to pull your changes in. Please refer to the documentation in the script for more details.

List of components

The full list of components synchronized into the VMR is here (Components.md).

The repository also contains a JSON manifest listing all components in a machine-readable format.

Filing Issues

This repo does not accept issues as of now. Please file issues to the appropriate development repos. For issues with the VMR itself, please use the source-build repository.

Useful Links

• Design documentation for the VMR - a set of documents describing the high-level design and the why's and how's
  • Design and Operation
  • Upstream/Downstream Relationships
  • Code and Build Workflow
  • Strategy for Managing External Source Dependencies
• .NET Source-Build
• What is .NET

.NET Foundation

.NET Runtime is a .NET Foundation project.

License

.NET is licensed under the MIT license.

roslyn-tools's Issues

Artifact layout and new SDK

The proposed artifact layout has the following structure:

artifacts
  $(Configuration)
    obj
       $(MSBuildProjectName)

I do not believe that will work with the new SDK and project.assets.json. The NuGet model uses a single project.assets.json for all configurations. In this layout there doesn't appear to be anywhere to put that file.

Roslyn originally had this model but we had to modify it to support project.assets.json to the following:

Binaries
  Obj
    $(MSBuildProjectName)
      $(Configuration) 

Insertion tool should only try to insert standalone builds

Right now the insertion tool is trying to insert builds kicked off as part of the overall product construction build. It should only be trying to insert builds that actually have artifacts on our drop share.

Packing fails when repo path has spaces in it

  Packaging ...
MSBUILD : error MSB1008: Only one project can be specified. [C:\Users\micodoba\.nuget\packages\roslyntools.repotoolset\1.0.0-beta-62512-02\tools\Build.proj]
  Switch: spaces\msbuild\

  For switch syntax, type "MSBuild /help"
C:\Users\micodoba\.nuget\packages\roslyntools.repotoolset\1.0.0-beta-62512-02\tools\Build.proj(191,5): error MSB3073: The command ""C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\MSBuild\15.0\bin\MSBuild.exe" /nodeRe
use:false "E:\projects\with spaces\msbuild\MSBuild.sln" /nologo /m /v:minimal /t:Pack /p:NoBuild=true /bl:"E:\projects\with spaces\msbuild\artifacts\Debug\log\Pack.binlog" /p:Configuration=Debug /p:CIBuild=True /p:RepoRoot=E:\projects\
with spaces\msbuild\ /p:VersionsPropsPath=E:\projects\with spaces\msbuild\build\Versions.props /p:DotNetPackageVersionPropsPath= /p:__BuildPhase=SolutionBuild" exited with code 1.

Root path is e:\projects\with spaces\msbuild

Arguments to the failing Exec task

"C:\Program Files (x86)\Microsoft Visual Studio\Preview\Enterprise\MSBuild\15.0\bin\MSBuild.exe" /nodeReuse:false "E:\projects\with spaces\msbuild\MSBuild.sln" /nologo /m /v:minimal /t:Pack /p:NoBuild=true /bl:"E:\projects\with spaces\msbuild\artifacts\Debug\log\Pack.binlog" /p:Configuration=Debug /p:CIBuild=True /p:RepoRoot=E:\projects\with spaces\msbuild\ /p:VersionsPropsPath=E:\projects\with spaces\msbuild\build\Versions.props /p:DotNetPackageVersionPropsPath= /p:__BuildPhase=SolutionBuild

RepoToolset fails on macOS

I'm trying to use repotoolset to compile the dotnet/arcade repo on macOS, and I get the following error

/Users/namc/.nuget/packages/roslyntools.repotoolset/1.0.0-beta-62614-02/tools/RepoLayout.props(37,5): error MSB4185: The function "GetFolderPath" on type "System.Environment" is not available for execution as an MSBuild property function.

It looks like the problem is here:

roslyn-tools/sdks/RepoToolset/tools/RepoLayout.props

Line 37 in 5780611

<NuGetPackageRoot Condition="'$(NuGetPackageRoot)' == '' AND '$(OS)' != 'Windows_NT'">$([System.Environment]::GetFolderPath(SpecialFolder.Personal))\.nuget\packages\</NuGetPackageRoot>

Using: dotnet 2.1.300-preview2-008251, macOS 10.13

cc @tmat

Insertion branches for pull requests need to be cleaned up automatically

It's not uncommon we forget to delete merged insertion branches, or forget to merge them entirely. The insertion tool should probably clean up old branches after a few weeks since we accumulate hundreds of them.

Remove references to VS assemblies from VSIXExpInstaller

Use package references instead.
This is blocking VS-less build of this repo.

https://github.com/dotnet/roslyn-tools/blob/master/src/VSIXExpInstaller/VSIXExpInstaller.csproj#L17-L56

SignTool needs to be packaged into a NuGet and uploaded for consumption.

Rename GetBuildVersion target to GetVsixVersion

The Nerdbank.GitVersioning package defines a GetBuildVersion target: https://github.com/AArnott/Nerdbank.GitVersioning/blob/d398833fcdcb857fe44c3e8287f25b4058921181/src/Nerdbank.GitVersioning.NuGet/build/Nerdbank.GitVersioning.targets#L62

This conflicts with the GetBuildVersion target which repo toolset defines if UsingToolVSSDK is set to true. The result is that if you try to enable the VS SDK in a project, then Nerdbank.GitVersioning will stop working.

PR #132 allows this to be worked around by setting the DefineGetBuildVersionTargetForVsix property to false. However, a better fix would be to rename GetBuildVersion to GetVsixVersion in repo toolset.

This would be a breaking change for consumers of repo toolset- they'd have to update their vsix definitions to use the new target name.

Insertion tool needs to respect updateRevisions.txt

The insertion tool right now (I believe) won't respond to updateRevisions.txt files, since it does the commit via libgit2sharp which won't do the commit hooks. This is bad, because it means we won't update revision.txt files that we need to do in some cases, like assembly version changes.

We could fix this in two ways:

1. Shell out to git commit which will run the hooks.
2. Shell out to the hook directly.
3. Reimplement the support in the insertion tool.

Insertion tool should warn if it's skipping over newer builds

The insertion tool's logic is:

1. Find the oldest passing build.
2. Insert that.
3. If it turns out no files changed, then simply don't create a PR and call it a "skip".

This can backfire if the oldest passing build is the build that's already inserted, and the reason it picked that build is because the newer builds were all failing. If that's the case, it should treat that as failure, not as a false "success".

repo-toolset does not work with live-unit testing

None of the repositories that use repotoolset work with live-unit-testing. Talking to @KevinH-MS it appears to be related to how repo-toolset and live-unit-testing interact when manipulating the intermediate output path.

@tmat

Create a dashboard to show insertion latency

We need a dashboard to show the last time F#, Live Unit Testing, Project System, Roslyn, and CLI were inserted compared to their current latest source to ensure we don't have something breaking the flow process.

Empty.targets needs to specify DefaultTargets="Build"

When we build a solution in build.proj via

<MSBuild Projects="X.sln" Targets="Build"/>

msbuild generates a metaproj whose Build target builds the contained projects using their default targets, not their Build target.

If the projects do not specify default target the first target in the project is selected instead.

This is an issue when we are building in source build projects that should be excluded from source build, in which case the default targets are not defined anywhere.

Remove support for `build` directory

The directory has been renamed to eng. See dotnet/arcade#33

Build.cmd doesn't show failing tests inline

We used to show the actual failed tests inline in the console output so you could quickly see the test that failed, now build.cmd it just tells you the dll:

  [...]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.Un
  itTests.dll [net461|x86]
XUnit : error : E:\project-system2\artifacts\Debug\TestResults\Microsoft.VisualStudio.ProjectSystem.Managed.TestService
s.UnitTests_net461_x86.html [net461|x86] [E:\project-system2\src\Microsoft.VisualStudio.ProjectSystem.Managed.TestServi
ces.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.VS.UnitTe
  sts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.VS.Unit
  Tests.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.CSharp.VS.Unit
  Tests.dll [net461|x86]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.UnitTests
  .dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.FSharp.UnitTes
  ts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.UnitTe
  sts.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.VS.Uni
  tTests.dll [net461|x86]
  Running tests: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.VS.U
  nitTests.dll [net461|x86]
  Tests succeeded: E:\project-system2\artifacts\Debug\bin\UnitTests\Microsoft.VisualStudio.ProjectSystem.VisualBasic.VS
  .UnitTests.dll [net461|x86]

Build FAILED.

XUnit : error : E:\project-system2\artifacts\Debug\TestResults\Microsoft.VisualStudio.ProjectSystem.Managed.TestService
s.UnitTests_net461_x86.html [net461|x86] [E:\project-system2\src\Microsoft.VisualStudio.ProjectSystem.Managed.TestServi
ces.UnitTests\Microsoft.VisualStudio.ProjectSystem.Managed.TestServices.UnitTests.csproj]
    0 Warning(s)

Restore.cmd creates NuGet packages

Didn't specify it should build packages but it is

C:\Users\jaredpar\code\roslyn-tools> .\Restore.cmd -msbuildversion "14.0"

Starting restore...
The restore completed successfully.

Skipping build...

Starting package...

Running: nuget.exe pack C:\Users\jaredpar\code\roslyn-tools\src/NuGet\SignTool.nuspec -BasePath "C:\Users\jaredpar\code\roslyn-tools\Artifacts\Debug\bin\SignTool" -OutputDirectory "C:\Users\jaredpar\code\roslyn-tools\Artifacts\Debug\bin\SignTool\NuGet\PreRelease" -prop licenseUrl="http://go.microsoft.com/fwlink/?LinkId=529443" -prop version="0.1.0-beta" -prop authors=Microsoft -prop projectURL="http://msdn.com/roslyn" -prop tags="Roslyn Signing" -prop thirdPartyNoticesPath="C:\Users\jaredpar\code\roslyn-tools\build/NuGetAdditionalFiles\ThirdPartyNotices.rtf" -prop netCompilersPropsPath="C:\Users\jaredpar\code\roslyn-tools\build/NuGetAdditionalFiles\Microsoft.Net.Compilers.props" -prop emptyDirPath="C:\Users\jaredpar\AppData\Local\Temp\2\u1kbmzib.muv" -prop NewtonsoftJsonVersion="9.0.1" -prop SystemCollectionsImmutableVersion="1.2.0" -prop SystemReflectionMetadataVersion="1.4.1-beta-24322-03"
Attempting to build package from 'SignTool.nuspec'.
Successfully created package 'C:\Users\jaredpar\code\roslyn-tools\Artifacts\Debug\bin\SignTool\NuGet\PreRelease\RoslynTools.Microsoft.SignTool.0.1.0-beta.nupkg'.
The package task completed successfully.
C:\Users\jaredpar\code\roslyn-tools>

Support ExcludeFromSourceBuild when using PackAsTool

In dotnet/sdk/src/Tests/Directory.Build.targets, PackAsTool is being set to true. This causes the Pack to use PackTool, instead of Pack.

In RepoToolset, if ExcludeFromSourceBuild is set to true, which it is for these tests, instead of importing the real Pack target, it imports an empty Pack target which basically noops for the project. But, since the SDK test projects switched to using PackTool, it isn’t noop’d. The easy solution is to add PackTool to the list of empty imports, but the other thing that happens when setting PackAsTool to true is that TargetsForTfmSpecificContentInPackage gets PackTool included, which then gets set as a DependsOnTarget for some other targets that get included and then fail because they expect certain artifacts to be present.

Build retention tool should verify symbols were archived for inserted builds

Right now we kick off archiving for inserted builds. But if that job fails, or nobody ever ran the insertion tool (and it was inserted manually), we won't try again. The retention tool already has what's needed to do this, so we should just update it to check too.

WPF inputs are not added to UpToDateCheckInput

Steps to reproduce:

1. Build a project containing a XAML page
2. Make a change to a XAML page (in a *.xaml file)
3. Build the project again

Expected results:

The project builds.

Actual results:

The project does not build, reporting that it is already up to date.

Resolution steps:

Add the following somewhere in the toolset targets:

<ItemGroup>
  <UpToDateCheckInput Include="@(Page);@(Resource)" />
</ItemGroup>

WorkaroundIncorrectSatelliteDeployment target can probably be removed from workarounds

The WorkaroundIncorrectSatelliteDeployment has a comment on it:

Work around bug in Microsoft.NET.Sdk < v2.0 where satellites were deployed on top of each other in root. dotnet/sdk#1360

The SDK bug has been fixed, so this target can probably be removed. The workaround is causing issues when trying to pack a project:

C:\git\dotnet-sdk\artifacts.dotnet\2.1.300-preview3-008616\sdk\2.1.300-preview3-008616\Sdks\NuGet.Build.Tasks.Pack\build\NuGet.Build.Tasks.Pack.targets(198,5): error NU5118: File 'C:\git\dotnet-sdk\artifacts\Debug\bin\Tests\Microsoft.NET.Build.Tests\publish\de\Microsoft.DotNet.Cli.Utils.resources.dll' is not added because the package already contains file 'tools\netcoreapp2.1\any\de\Microsoft.DotNet.Cli.Utils.resources.dll' [C:\git\dotnet-sdk\src\Tests\Microsoft.NET.Build.Tests\Microsoft.NET.Build.Tests.csproj]

Running build.cmd without VS installed shows "Cannot bind argument to parameter 'Path' because it is null."

Not sure if this is projectsystem problem or repo toolset problem, but we used to show a friendly error if you didn't have the right VS installed and you ran build.cmd, now we show:

E:\project-system2>build
Cannot bind argument to parameter 'Path' because it is null.
System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'Path' because it is null.
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
at LocateVisualStudio, E:\project-system2\build\Build.ps1: line 96
at Build, E:\project-system2\build\Build.ps1: line 104
at <ScriptBlock>, E:\project-system2\build\Build.ps1: line 155
at <ScriptBlock>, <No file>: line 1

Insertion tool should grant force push and delete permissions to PR branch

The insertion tool creates a branch dev/vslsnap/... which it uses as the source of the pull request. It should also grant permissions to the swat group to be able to force push and delete this branch. This would allow swat members to delete the branches after merging, and would allow changes to be made when necessary.

Change 'build' folder name

RepoToolset is using the $(RepoRoot)build folder location for build stuff. There's discussion in dotnet/arcade#33, where it was decided to change the folder structure to something like

\eng
   \core
   \tools

Arcade cannot make this change until RepoToolset is updated.

Internal bug dashboard needs updating

@Pilchie, to track the work you're doing for @MattGertz, which might already be done. (This is mostly for tracking kudos down the road.)

Update Roslyn Insertion Tool to include satellite assembly Willow packages

These packages won't exist until dotnet/roslyn#23196 is resolved.

RepoToolset fails to load the correct compiler in WPF projects

The toolset uses a <PackageReference> to reference Microsoft.Net.Compilers:

roslyn-tools/sdks/RepoToolset/tools/Compiler.props

Line 10 in f093fd2

<PackageReference Include="Microsoft.Net.Compilers" Version="$(MicrosoftNetCompilersVersion)" />

This feature assumes that build extensions included in the package will be used during the build. However, for the .tmp_proj created for WPF projects, this is not the case. The underlying bug is NuGet/Home#5894.

The workaround is to explicitly import the NuGet build files when building the WPF temporary project, as shown in dotnet/roslyn-sdk#94.

Insertion tool creates pull requests if there are only whitespace changes

The insertion tool creates pull requests whenever it updates any file on disk. If we're trying to insert a build that is already the inserted build, and this implicitly changes whitespace in a file (because we didn't round-trip it), we end up creating pull requests. This gets confusing because it looks like something is being updated but it isn't.

The fix (I think) is before creating the commit, look at all modified files and unpend any changes that are whitespace only changes.

Restore doesn't restore Microsoft.VSSDK.BuildTools

The version of build tools currently used is 14.3.2.5407. That is not properly restored by using Restore.cmd.

Restore fails on machine without MSBuild 15.0

Get the following error:

Starting toolset restore...
Locating MSBuild install path...
System.DllNotFoundException: Unable to load DLL 'Microsoft.VisualStudio.Setup.Configuration.Native.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E)
   at LocateVS.Interop.GetSetupConfiguration(ISetupConfiguration& setupConfiguration)
   at LocateVS.Instance.GetSetupConfiguration()
   at LocateVS.Instance.<EnumerateVisualStudioInstances>d__1.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
   at LocateVS.Instance.LocateVisualStudioInstance(String vsProductVersion, HashSet`1 requiredPackageIds)
   at LocateVS.Instance.GetInstallPath(String vsProductVersion, String[] requiredPackageIds)
   at CallSite.Target(Closure , CallSite , Type , Object , Object )

RepoToolset overrides DeployExtension set in the project file

DeployExtension is a standard VSSDK property that controls extension deployment. Here is an example usage:
https://github.com/DotNetAnalyzers/StyleCopAnalyzers/blob/d18b15f2342b52800c86a9540f95c1a51b0ddbbb/StyleCop.Analyzers/StyleCop.Analyzers.Vsix/StyleCop.Analyzers.Vsix.csproj#L21-L24

This approach does not work with RepoToolset because the DeployExtension property is overwritten with the value from DeployProjectOutput. This is unintuitive and results in example VS extension projects whose behavior is highly context-dependent and will not work if other users try to take the same approach.

Locate-VS should support pre-Dev15

Doing this will allow us to route all locating of VS through a single call to LocateVS and will additionally allow people to differentiate between errors in the LocateVS or ManagedQuery APIs and errors in locating VS itself.

Automatic GitHub merges should merge automatically if everything is green

Add a feature to NuGetRepack to block generating release packages for certain pre-release NuGet packages

Roslyn-analyzers repo currently has an experimental NuGet package for dataflow based analyzers Microsoft.CodeQuality.Analyzers.Exp.nupkg. We want to ship pre-release version of this package to NuGet to allow dogfooding, but never want to ship a release package (once we are comfortable with the DFA based rules to make it release, these rules will be moved to other existing analyzer packages and we will delete Microsoft.CodeQuality.Analyzers.Exp). It would be good if NuGetRepack can support blocking generation of release packages for such experimental packages - essentially address dotnet/roslyn-analyzers#1622 (comment).

Automatic GitHub merges should run every 10 minutes

Right now we are kicking off new inter-branch merges every midnight Pacific Time, but we'd like to ramp this up to every few minutes so we can flow things immediately when they're merged.

SignTool crashes if 'exclude' section is missing

InternalsVisibleTo not hooked up

It should be possible to generate an IVT via the following:

  <ItemGroup>
    <InternalsVisibleTo Include="SignTool.UnitTests" />
  </ItemGroup>

Presently it is not.

RepoToolset Package Version is formatted incorrectly.

Currently the repo toolset package has a version formatted as <Major>.<Minor>.<Patch>-<Suffix><number> (ex.: 1.0.0-alpha9 or 1.0.0-alpha40).

However, this does not follow the Package Versioning guidelines from NuGet (https://docs.microsoft.com/en-us/nuget/reference/package-versioning) and causes older packages to be reported as the 'latest version' (ex: 1.0.0-alpha9 is reported as being newer than 1.0.0-alpha40).

The package version should be updated to either place the postfix number (i.e. 9 or 40) in the slot or place a dot between the Suffix and postfix number (do: 1.0.9-alpha or 1.0.0-alpha.9, the former is probably "better", since that is the format we use elsewhere).

SignTool crashes if there are no nested VSIX

Signtool support for certificate files

I am interested in using signtool for our own projects. But since we build everything inside docker containers we can only use certificate files and not from a store (as far as I know).
It would be great to be able to specify a certificate file and password or private key file.

RepoToolset needs to support alternative strong name keys

Some of the repositories, such as the sdk/cli (and related) repos, are using the ASP.NET strong name key, rather than the .NET strong name key.

We need to, at a minimum, support selecting which of these two keys are used and support IVT generation for each.

Update RepoToolset to have a 'UsingDotNetCli' property

@tmat suggested that we update repo toolset to have a single UsingDotNetCli property, rather than the current auto-detection plus explicit switches.

RepoToolset does not declare `IsTestProject` earlier enough to use as a TargetFramework condition

As per the title, IsTestProject is declared late enough that it cannot be used as a condition on the TargetFramework property.

It would be great if this was possible, since some repos are targeting net46 for shipping binaries and net461 for test binaries.

csc.exe bundled in RepoToolset exit code 8 on macOS

I'm trying to compile this on macOS: dotnet/arcade#35

After working around #172, I get this error:

/Users/namc/dev/dotnet/arcade/packages/microsoft.net.compilers/2.7.0-beta2-62330-02/tools/Microsoft.CSharp.Core.targets(84,5): error MSB6006: "csc.exe" exited with code 8

dotnet --info:

.NET Command Line Tools (2.1.100-preview-007366)

Product Information:
 Version:            2.1.100-preview-007366
 Commit SHA-1 hash:  8265c551de

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  10.13
 OS Platform: Darwin
 RID:         osx.10.12-x64
 Base Path:   /Users/namc/.dotnet/sdk/2.1.100-preview-007366/

Microsoft .NET Core Shared Framework Host

  Version  : 2.1.0-preview2-26131-06
  Build    : b13a0d5c331f374afd35ded57b9a4b4ab128864c

msbuild.binlog.zip

cc @tmat

Versioning does not allow per project overrides

Versioning currently defines version properties before analyzing the project. This means that the project can't define a project specific major / minor number. In some repos, like Arcade, version numbers are not consistent across artifacts produced by the repo.

Providing the version suffix as a separate property would allow a project to override the generated version.

RepoToolset: Add Visibility="public" attribute to resx generator task

Existing resx generator has an option to generate public resource class. Our generator should have the same option.

This is needed by VSUnitTesting repo.

Insertion tool requires a very large number of manual changes for new CoreFX dependencies

The worst part is that some of these changes need to already be in the branch when the tool runs, so it will fail automated insertion unless these changes are added separately.

As far as I can tell, here's a complete list of things that need to be updated when a new CoreFX dependency gets added:

• A new line for the version in src/ProductData/AssemblyVersions.tt
• Two new lines for each dependency in src/ProductData/versions.xml
• New entries in every *.config.tt file below src/appid
• A new binding redirect in src/VSSDK/VSIntegration/ISOShell/Templates/VSShellTemplate/VSShellIso/VSShellStubExe/Stub.exe.config

It's not clear how many of these are actually necessary, but the insertion tool should probably do all of them automatically.

Build.cmd doesn't stop the build if there are restore errors

If there are restore errors, build will continue and run the build. It should stop the build if there are restore errors.

QuietRestore=true does not work with paths that contain a space

Currently a repository using repo toolset that also specifies /p:QuietRestore=true cannot build if the enlistment path contains a space. Instead, the build fails with an error similar to the following:

MSBUILD : error MSB1008: Only one project can be specified. [C:\Users\tagoo\.nuget\packages\RoslynTools.RepoToolset\1.0.0-beta-62615-02\tools\Build.proj]
C:\Users\tagoo\.nuget\packages\RoslynTools.RepoToolset\1.0.0-beta-62615-02\tools\Build.proj(152,5): error MSB3073: The command ""C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\MS
Build.exe" /nodeReuse:false "C:\Users\tagoo\.nuget\packages\RoslynTools.RepoToolset\1.0.0-beta-62615-02\tools\Tools.proj" /bl:"D:\project system\artifacts\Debug\log\RestoreRepoTools.binlog" /nologo /m /v:quiet
/t:Restore /p:Configuration=Debug /p:CIBuild=False /p:RepoRoot=D:\project system\ /p:VersionsPropsPath=D:\project system\build\Versions.props /p:DotNetPackageVersionPropsPath= /p:BaseIntermediateOutputPath=D:\p
roject system\artifacts\toolset\Common /p:ExcludeRestorePackageImports=true /p:UsingPipeBuildPublishing=false" exited with code 1.

This failure looks to be because repo-toolset is calling Exec and not properly quoting the command line arguments.

Quoting the arguments is fairly trivial. However, there is an additional issue where, even if quoted, the build still fails for any paths that end with \.

The recommended fix would be to trim the trailing \ from any arguments and ensure that it is re-added, where required, in the props files themselves.

SignTool needs tests.

SignTool error message shouldn't use part

The word "part" isn't resonating with end users in the error message. Use a different word to describe items contained inside a VSIX.