dotnet / api-docs-sync

Tools for porting documentation from intellisense xmls to api-docs, or from api-docs to triple slash comments.

dotnet/dotnet - Home of the .NET VMR

This repository is a Virtual Monolithic Repository (VMR) which includes all the source code and infrastructure needed to build the .NET SDK.

What this means:

• Monolithic - a join of multiple repositories that make up the whole product, such as dotnet/runtime or dotnet/sdk.
• Virtual - a mirror (not replacement) of product repos where sources from those repositories are synchronized into.

In the VMR, you can find:

• source files of each product repository which are mirrored inside of their respective directories under src/,
• tooling that enables building the whole .NET product from source on Linux platforms,
• small customizations, in the form of patches, applied on top of the original code to make the build possible,
• [in future] E2E tests for the whole .NET product.

Just like the development repositories, the VMR will have a release branch for every feature band (e.g. release/8.0.1xx). Similarly, VMR's main branch will follow main branches of product repositories (see Synchronization Based on Declared Dependencies).

More in-depth documentation about the VMR can be found in VMR Design And Operation. See also dotnet/source-build for more information about our whole-product source-build.

Goals

• The main purpose of the dotnet/dotnet repository is to have all source code necessary to build the .NET product available in one repository and identified by a single commit.
• The VMR also aims to become the place from which we release and service future versions of .NET to reduce the complexity of the product construction process. This should allow our partners and and 3rd parties to easily build, test and modify .NET using their custom infrastructure as well as make the process available to the community.
• Lastly, we hope to solve other problems that the current multi-repo setup brings:
  • Enable the standard down-/up-stream open-source model.
  • Fulfill requirements of .NET distro builders such as RedHat or Canonical to natively include .NET in their distribution repositories.
  • Simplify scenarios such as client-run testing of bug fixes and improvements. The build should work in an offline environment too for certain platforms.
  • Enable developers to make and test changes spanning multiple repositories.
  • More efficient pipeline for security fixes during the CVE pre-disclosure process.

We will achieve these goals while keeping active coding work in the separate repos where it happens today. For example: ASP.NET features will continue to be developed in dotnet/aspnetcore and CLR features will be continue to be developed in dotnet/runtime. Each of these repos have their own distinct communities and processes, and aggregating development into a true mono-repo would work against that. Hence, the "virtual" monolithic repo: the VMR gives us the simplicity of a mono-repo for building and servicing the product, while active development of components of that product stays in its various existing repos. The day to day experience for typical contributors will not change.

Limitations

This is a work-in-progress. There are considerable limitations to what is possible at the moment. For an extensive list of current limitations, please see Temporary Mechanics.
See the Unified Build roadmap for more details.

Supported platforms

• 8.0
  • source-build configuration on Linux
• 9.0+ (WIP)
  • source-build configuration on Linux
  • non-source-build configuration on Linux, Mac, and Windows

For the latest information about Source-Build support for new .NET versions, please check our GitHub Discussions page for announcements.

Code flow

For the time being, the source code only flows one way - from the development repos into the VMR. More details on this process:

• Source Synchronization Process
• Synchronization Based on Declared Dependencies
• Moving Code and Dependencies between the VMR and Development Repos

We expect the code flow to start working both ways in the .NET 9 timeframe. See the Unified Build roadmap for more details.

Contribution

At this time, the VMR will not accept any changes and is a read-only mirror of the development repositories only. Please, make the changes in the respective development repositories (e.g., dotnet/runtime or dotnet/sdk) and they will get synchronized into the VMR automatically.

Dev instructions

Please note that this repository is a work-in-progress and there are some usability issues connected to this. These can be nuisances such as some checked-in files getting modified by the build itself and similar. For the latest information about Source-Build support, please watch for announcements posted on our GitHub Discussions page.

Prerequisites

The dependencies for building can be found here. In case you don't want to / cannot prepare your environment per the requirements, consider using Docker.

Building

1. Clone the repository

   git clone https://github.com/dotnet/dotnet dotnet-dotnet
   cd dotnet-dotnet

2. Build the .NET SDK

   Choose one of the following build modes:

   • Microsoft based build

     For Unix:

     ./build.sh --clean-while-building

     For Windows:

     .\build.cmd -cleanWhileBuilding

   • Building from source

     # Prep the source to build on your distro.
     # This downloads a .NET SDK and a number of .NET packages needed to build .NET from source.
     ./prep-source-build.sh
     
     # Build the .NET SDK
     ./build.sh -sb --clean-while-building

   The resulting SDK is placed at artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz (for Unix) or artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].zip (for Windows).

3. (Optional) Unpack and install the .NET SDK

   For Unix:

   mkdir -p $HOME/dotnet
   tar zxf artifacts/assets/Release/dotnet-sdk-9.0.100-[your-RID].tar.gz -C $HOME/dotnet
   ln -s $HOME/dotnet/dotnet /usr/bin/dotnet

   For Windows:

   mkdir %userprofile%\dotnet
   tar -xf artifacts/assets/Release/dotnet-sdk-9.0.100-[your RID].zip -C %userprofile%\dotnet
   set "PATH=%userprofile%\dotnet;%PATH%"

   To test your built SDK, run the following:

   dotnet --info

Note

Run ./build.sh --help (for Unix) or .\build.cmd -help (for Windows) to see more information about supported build options.

Building using Docker

You can also build the repository using a Docker image which has the required prerequisites inside. The example below creates a Docker volume named vmr and clones and builds the VMR there.

docker run --rm -it -v vmr:/vmr -w /vmr mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8
git clone https://github.com/dotnet/dotnet .

# - Microsoft based build
./build.sh --clean-while-building

# - Building from source
./prep-source-build.sh && ./build.sh -sb --clean-while-building

mkdir -p $HOME/.dotnet
tar -zxf artifacts/assets/Release/dotnet-sdk-9.0.100-centos.8-x64.tar.gz -C $HOME/.dotnet
ln -s $HOME/.dotnet/dotnet /usr/bin/dotnet

Codespaces

You can also utilize GitHub Codespaces where you can find preset containers in this repository.

Building from released sources

You can also build from sources (and not from a context of a git repository), such as the ones you can acquire from a dotnet/dotnet release. In this case, you need to provide additional information which includes the original repository and commit hash the code was built from so that the SDK can provide a better debugging experience (think the Step into.. functionality). Usually, this means the dotnet/dotnet repository together with the commit the release tag is connected to.

In practice, this means that when calling the main build script, you need to provide additional arguments when building outside of a context of a git repository.
Alternatively, you can also provide a manifest file where this information can be read from. This file (release.json) can be found attached with the dotnet/dotnet release.

Synchronizing code into the VMR

Sometimes you want to make a change in a repository and test that change in the VMR. You could of course make the change in the VMR directly (locally, as the VMR is read-only for now) but in case it's already available in your repository, you can synchronize it into the VMR (again locally).

To do this, you can either start a dotnet/dotnet Codespace - you will see instructions right after it starts. Alternatively, you can clone the repository locally and use the vmr-sync.sh or vmr-sync.ps1 script to pull your changes in. Please refer to the documentation in the script for more details.

List of components

The full list of components synchronized into the VMR is here (Components.md).

The repository also contains a JSON manifest listing all components in a machine-readable format.

Filing Issues

This repo does not accept issues as of now. Please file issues to the appropriate development repos. For issues with the VMR itself, please use the source-build repository.

Useful Links

• Design documentation for the VMR - a set of documents describing the high-level design and the why's and how's
  • Design and Operation
  • Upstream/Downstream Relationships
  • Code and Build Workflow
  • Strategy for Managing External Source Dependencies
• .NET Source-Build
• What is .NET

.NET Foundation

.NET Runtime is a .NET Foundation project.

License

.NET is licensed under the MIT license.

api-docs-sync's Issues

What is ECMAXml?

From the description of this repo:

Tool that can port intellisense triple slash comments to ECMAXml files from an api-docs repo. For example: from dotnet/runtime to dotnet-api-docs.

The only reference I was able to find about ECMAXml format is in this article: https://docs.microsoft.com/en-us/teamblog/announcing-unified-dotnet-experience-on-docs#built-with-open-source-in-mind, which seems to be broken: https://docs.go-mono.com/index.aspx?link=man:mdoc(5).

There is nothing on official website either: https://www.ecma-international.org/?s=ecmaxml

So what is ECMAXml and is there any published spec? :)

Unit tests fail when running against net6.0

Running unit tests against the main branch with the projects targeting net6.0 is leading to a failed test.

 Libraries.Tests.PortToTripleSlashTests.Port_Basic
   Source: PortToTripleSlashTests.cs line 15
   Duration: 1 sec

  Message: 
    System.Exception : Diagnostic errors found.
  Stack Trace: 
    ToTripleSlashPorter.CheckDiagnostics(MSBuildWorkspace workspace, String stepName) line 479
    ToTripleSlashPorter.GetOrAddProject(MSBuildWorkspace workspace, Dictionary`2 projects, String projectPath) line 420
    ToTripleSlashPorter.GetProjectInfo(String projectPath, Boolean isMono) line 369
    ToTripleSlashPorter.Port() line 142
    ToTripleSlashPorter.Start(Configuration config) line 127
    PortToTripleSlashTests.PortToTripleSlash(String testDataDir, Boolean save, Boolean skipInterfaceImplementations, String assemblyName, String namespaceName, String typeName) line 54
    PortToTripleSlashTests.Port_Basic() line 17

The diagnostic error is:

"Failure - Msbuild failed when processing the file 'C:\\Users\\jeffhand\\AppData\\Local\\Temp\\0shltoqo.fn2\\Project\\MyAssembly.csproj' with message:
The expression \"[MSBuild]::GetTargetFrameworkIdentifier(net5.0)\" cannot be evaluated. MSB0001: Internal MSBuild Error: A required NuGet assembly was not found. 
Expected Path: C:\\Program Files\\dotnet\\sdk\\6.0.100-preview.2.21158.2  C:\\Program Files\\dotnet\\sdk\\6.0.100-preview.2.21158.2\\Sdks\\Microsoft.NET.Sdk\\targets\\Microsoft.NET.TargetFrameworkInference.targets"

For types that live in a difference assembly than the one that ships it, make sure the MSBuild task will move them to the appropriate intellisense xml

For example, String lives in System.Private.CoreLib.xml, move its docs to System.Runtime.xml

Pragmas before types and members get dropped when porting docs into triple slash comments

When porting API docs into triple slash comments, classes that have #if def pragmas wrapped around their access modifiers lose the first line of the pragma.

Given the following example, The #if USEPUBLIC pragma line is dropped.

#if USEPUBLIC
    public
#else
    internal
#endif
    class Foo { }

Expected

    /// <summary>Foo summary</summary>
#if USEPUBLIC
    public
#else
    internal
#endif
    class Foo { }

Actual

    /// <summary>Foo summary</summary>
    public
#else
    internal
#endif
    class Foo { }

This occurs with other types of pragmas as well. An example from Utf8Formatter.Guid.cs shows a scenario of an #endregion getting dropped.

Before

namespace System.Buffers.Text
{
    public static partial class Utf8Formatter
    {
        #region Constants

        private const byte OpenBrace = (byte)'{';
        private const byte CloseBrace = (byte)'}';

        private const byte OpenParen = (byte)'(';
        private const byte CloseParen = (byte)')';

        private const byte Dash = (byte)'-';

        #endregion Constants

        /// <summary>
        /// Formats a Guid as a UTF8 string.
        /// </summary>
        /// <param name="value">Value to format</param>
        /// <param name="destination">Buffer to write the UTF8-formatted value to</param>
        /// <param name="bytesWritten">Receives the length of the formatted text in bytes</param>
        /// <param name="format">The standard format to use</param>
        /// <returns>
        /// true for success. "bytesWritten" contains the length of the formatted text in bytes.
        /// false if buffer was too short. Iteratively increase the size of the buffer and retry until it succeeds.
        /// </returns>
        /// <remarks>
        /// Formats supported:
        ///     D (default)     nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn
        ///     B               {nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn}
        ///     P               (nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn)
        ///     N               nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
        /// </remarks>
        /// <exceptions>
        /// <cref>System.FormatException</cref> if the format is not valid for this data type.
        /// </exceptions>
        public static bool TryFormat(Guid value, Span<byte> destination, out int bytesWritten, StandardFormat format = default)

After

namespace System.Buffers.Text
{
    /// <summary>Provides static methods to format common data types as Utf8 strings.</summary>
    public static partial class Utf8Formatter
    {
        #region Constants

        private const byte OpenBrace = (byte)'{';
        private const byte CloseBrace = (byte)'}';

        private const byte OpenParen = (byte)'(';
        private const byte CloseParen = (byte)')';

        private const byte Dash = (byte)'-';

        /// <summary>Formats a <see cref="System.Guid" /> as a UTF8 string.</summary>
        /// <param name="value">The value to format.</param>
        /// <param name="destination">The buffer to write the UTF8-formatted value to.</param>
        /// <param name="bytesWritten">When the method returns, contains the length of the formatted text in bytes.</param>
        /// <param name="format">The standard format to use.</param>
        /// <returns><see langword="true" /> if the formatting operation succeeds; <see langword="false" /> if <paramref name="buffer" /> is too small.</returns>
        /// <remarks>Formats supported:
        /// |Format string|Result string|
        /// |--|--|
        /// |D (default)|nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn|
        /// |B|{nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn}|
        /// |P|(nnnnnnnn-nnnn-nnnn-nnnn-nnnnnnnnnnnn)|
        /// |N|nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn|
        /// If the method fails, iteratively increase the size of the buffer and retry until it succeeds.</remarks>
        public static bool TryFormat(Guid value, Span<byte> destination, out int bytesWritten, StandardFormat format = default)

Tool fails to run with System.InvalidOperationException:

I tried to run the tool on a dotnet/runtime library and hit the following error:

Specified Docs xml locations:
  -  c:\src\dotnet\dotnet-api-docs\xml
Included assemblies:
 - System.IO.Pipelines
Included namespaces:
 - System.IO
Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Libraries.ToTripleSlashPorter.LoadVSInstance() in C:\src\carlossanlop\DocsPortingTool\Libraries\ToTripleSlashPorter.cs:line 439
   at Libraries.ToTripleSlashPorter.Start(Configuration config) in C:\src\carlossanlop\DocsPortingTool\Libraries\ToTripleSlashPorter.cs:line 112
   at DocsPortingTool.DocsPortingTool.Main(String[] args) in C:\src\carlossanlop\DocsPortingTool\Program\DocsPortingTool.cs:line 22

@carlossanlop

Missing mapping with ToDocs command

When I worked on dotnet/dotnet-api-docs#7129 the DocsPortingTool didn't map some methods to xml files. I had to do it manually.

• M:System.Net.Http.HttpRequestOptions.TryGetValue``1(System.Net.Http.HttpRequestOptionsKey{``0},``0@)
• M:System.Net.Http.HttpRequestOptions.Set``1(System.Net.Http.HttpRequestOptionsKey{``0},``0)

The command I used was DocsPortingTool -Direction ToDocs -Docs c:\Dev\dotnet-api-docs\xml -IntelliSense C:\dev\runtime\artifacts\bin\System.Net.Http\ -IncludedAssemblies System.Net.Http -IncludedNamespaces System.Net.Http -Save true on this code https://github.com/dotnet/runtime/pull/58896/files

Add `##remarks`

Preserve xml file encoding to prevent non-ascii chars from disappearing

For example, System/Math.xml and MathF.xml have several unicode characters that get lost after the tool runs.

Bug: Not finding symbols from Mono

Root cause: The workspace properties to open it with RuntimeFlavor=Mono were not being used, so when the tool needed to check for a Mono System.Private.CoreLib, it was using the coreclr one instead of the mono one.

Lines containing <param> items do not get new lines properly added

As reported by @tannergooding.
Example: System.Runtime.Intrinsics summaries.

Bug: When there's a `<seealso cref="..."/>` in remarks triple slash, it does not get ported correctly

Example: https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/src/libraries/System.ObjectModel/src/System/Collections/ObjectModel/ObservableCollection.cs#L94

The triple slash look like this:

        /// <summary>
        /// Occurs when the collection changes, either by adding or removing an item.
        /// </summary>
        /// <remarks>
        /// see <seealso cref="INotifyCollectionChanged"/>
        /// </remarks>
        [field: NonSerialized]
        public virtual event NotifyCollectionChangedEventHandler? CollectionChanged;

Ignore for a moment the fact that this reference should've been an <altmember cref="..."/>.

Instead of getting it ported exactly this way:

        <remarks>
          <format type="text/markdown"><![CDATA[

## Remarks

see <xref:System.Collections.Specialized.INotifyCollectionChanged>

          ]]></format>
        </remarks>

It gets unexpectedly ported as:

        <remarks>
          <format type="text/markdown"><![CDATA[

## Remarks

see seealsoContents

          ]]></format>
        </remarks>

Don't add remarks sections to enum fields

Remarks for enum fields are invalid (there's nowhere to display them in docs and they now cause a build warning). Please update the tool to not add empty remarks nodes for enum fields.

Remove undesired endlines

Any endline not preceded by a . or a : should be removed.

Some markdown-formatted remarks are ported into triple slash comments with invalid structure

As seen in MemoryManager<T> from System.Memory in dotnet/runtime, some <remarks> sections are getting produced with invalid structure, and duplicated content.

From `MemoryManager`1.xml`

  <Docs>
    <typeparam name="T">The type of items in the memory buffer managed by this memory manager.</typeparam>
    <summary>An abstract base class that is used to replace the implementation of <see cref="T:System.Memory`1" />.</summary>
    <remarks>
      <format type="text/markdown"><![CDATA[  

The `MemoryManager<T>` class is used to extend the knowledge of types that <xref:System.Memory%601> is able to represent. For example, you can derive from `MemoryManager<T>` to allow <xref:System.Memory%601> to be backed by a <xref:System.Runtime.InteropServices.SafeHandle>.

> [!NOTE]
> The `MemoryManager<T>` class is intended for advanced scenarios. Most developers do not need to use it.

       ]]></format>
    </remarks>
  </Docs>

Before porting the docs into triple slash comments, this was the content of MemoryManager.cs:

    /// <summary>
    /// Manager of <see cref="System.Memory{T}"/> that provides the implementation.
    /// </summary>
    public abstract class MemoryManager<T> : IMemoryOwner<T>, IPinnable

After porting, this is the result:

    /// <summary>An abstract base class that is used to replace the implementation of <see cref="System.Memory{T}" />.</summary>
    /// <typeparam name="T">The type of items in the memory buffer managed by this memory manager.</typeparam>
    /// <remarks>The `MemoryManager<T>` class is used to extend the knowledge of types that <see cref="System.Memory{T}" /> is able to represent. For example, you can derive from `MemoryManager<T>` to allow <see cref="System.Memory{T}" /> to be backed by a <see cref="System.Runtime.InteropServices.SafeHandle" />.
    /// <format type="text/markdown"><![CDATA[
    /// > [!NOTE]
    /// > The `MemoryManager<T>` class is intended for advanced scenarios. Most developers do not need to use it.
    /// ]]></format></remarks>
    public abstract class MemoryManager<T> : IMemoryOwner<T>, IPinnable

The <remarks> section has the content duplicated, once inside the <format> and once before it, which is an invalid structure.

How to deal with cross assembly crefs

I backported docs to System.IO as an experiment. It made FileSystemName (which is in corelib) have <see cref="System.IO.FileSystemWatcher" /> (which is in a different assembly). This is fine for the docs build because its context is apparently "everything". However the compiler complains with CS1574 and I have to remove the cref.

When making code the source of truth, how can we preserve cref's across assemblies?

@carlossanlop

Type docs applied to all partial classes

https://github.com/dotnet/winforms has multiple types that span across multiple files. The tool is currently applying type xmldocs to all partial definitions.

Instead we should track whether a type has been annotated, and not annotate other partials. When deduping we should perhaps look for the file with the shortest name, e.g. from the following list pick Application.cs:

src/System.Windows.Forms/src/System/Windows/Forms/Application.ComponentManager.cs
src/System.Windows.Forms/src/System/Windows/Forms/Application.cs
src/System.Windows.Forms/src/System/Windows/Forms/Application.MarshallingControl.cs
src/System.Windows.Forms/src/System/Windows/Forms/Application.ModalApplicationContext.cs
src/System.Windows.Forms/src/System/Windows/Forms/Application.ParkingWindow.cs
src/System.Windows.Forms/src/System/Windows/Forms/Application.ThreadContext.cs
src/System.Windows.Forms/src/System/Windows/Forms/Application.ThreadWindows.cs

Tool is slow

Root cause: some projects are reopened from scratch. I should cache the workspaces, projects and compilations in dictionaries.

DocsPortingTool fails to convert generic targets of `<see cref` correctly

The following line: https://github.com/dotnet/dotnet-api-docs/blob/96b853d5e314fc00f8fa2acad402ddb3a00b76a7/xml/System.IO.Pipelines/PipeOptions.xml#L160

<summary>Gets the <see cref="T:System.Buffers.MemoryPool`1" /> object used for buffer management.</summary>

When ported did not change the syntax of the generic type. CSC then complains:

error CS1584: XML comment has syntactically incorrect cref attribute 'System.Buffers.MemoryPool`1' 

To fix it I had to manually change the generic parameter syntax to {T}. Hope I got it right?

The tool should be able to do this for me.

Command:

-Direction ToTripleSlash -CsProj c:\src\dotnet\runtime\src\libraries\System.IO.Pipelines\src\System.IO.Pipelines.csproj -docs c:\src\dotnet\dotnet-api-docs\xml -IncludedAssemblies System.IO.Pipelines -IncludedNamespaces System.IO

Using dotnet/runtime add68df26af8c0dab2afb13da1f0ca033753fd0b

There were a mess of these issues in this project. I ended up find->replacing `1 with {T} to deal with them. I checked and the resultant XML was correct. Not sure if that's a great strategy, ideally the actual type parameter name could be used.

Bottleneck reading XML element value

Up to 40-50% of the time can be spent in this method
https://github.com/carlossanlop/DocsPortingTool/blob/main/Libraries/XmlHelper.cs#L134-L143

It is simply reading the value of an element, but it needs to do it literally, ie., any CDATA wrapping it, etc, so it can't just get Value. This forces creating an Xml text writer, etc - it's very slow.

I added in a comment there an alternative way of doing this based on the stackoverflow post below

https://stackoverflow.com/questions/3793/best-way-to-get-innerxml-of-an-xelement

I didn't try every scenario but it seems to produce the same result albeit with sometimes different line wraps (possibly an improvement). It's about twice as fast so if it's good enough it could be swapped for what is there now.

Consider inserting exceptions before remarks in triple slash

As suggested by @eiriktsarpalis . Remarks are the least important items.

Don't indent remarks

When remarks get indented by 4 spaces, they're rendered as code blocks. For example, this PR introduced indentation that caused the remarks to render as code blocks. A customer fixed that in dotnet/dotnet-api-docs#7104. The /// for the remarks is here - perhaps the <para> tags are causing the indentation.

Add option to force update all comments

-forceupdate Port all triple slash comments, whether they are empty or not.

Types can also have parameters (for example: Delegates)

This file wraps a delegate, which is also considered a Type:
dotnet-api-docs\xml\System.Runtime.InteropServices\DllImportResolver.xml

It has parameters, which the tool does not recognize yet.

File 'Accessibility-version.dll' not found or not a PE file

An attempt to backport to /// fails with the following error:

Diagnostic messages found in workspace.OpenProjectAsync - D:\Development\dotnet-winforms\src\System.Windows.Forms\src\System.Windows.Forms.csproj:
    Failure - Msbuild failed when processing the file 'D:\Development\dotnet-winforms\src\Accessibility\src\Accessibility.ilproj' with message:
        EXEC: (0, 0): File 'D:\Development\dotnet-winforms\artifacts\bin\Accessibility-version\Debug\net6.0\Accessibility-version.dll' not found or not a PE file
    Failure - Msbuild failed when processing the file 'D:\Development\dotnet-winforms\src\Accessibility\src\Accessibility.ilproj' with message:
        C:\Users\igveliko\.nuget\packages\microsoft.net.sdk.il\6.0.0-rc.2.21454.1\targets\Microsoft.NET.Sdk.IL.targets: (106, 5): The command ""C:\Users\igveliko\.nuget\packages\\runtime.win-x64.microsoft.netcore.ildasm\6.0.0-rc.2.21454.1\runtimes\win-x64\native/ildasm" "D:\Development\dotnet-winforms\artifacts\bin\Accessibility-version\Debug\net6.0\Accessibility-version.dll" /OUT="D:\Development\dotnet-winforms\artifacts\obj\Accessibility\Debug\net6.0\/Accessibility.ref.il"" exited with code 1.

Running the following command:

-CsProj D:\Development\dotnet-winforms\src\System.Windows.Forms\src\System.Windows.Forms.csproj -Docs D:\Development\dotnet-dotnet-api-docs\xml -Direction ToTripleSlash -IncludedAssemblies System.Windows.Forms.dll -IncludedNamespaces System.Windows.Forms

For the time being I've disabled the diagnostics as follows, but there should be a better way to ignore this error:

diff --git a/Libraries/ToTripleSlashPorter.cs b/Libraries/ToTripleSlashPorter.cs
index b4a3110..e02039f 100644
--- a/Libraries/ToTripleSlashPorter.cs
+++ b/Libraries/ToTripleSlashPorter.cs
@@ -474,7 +474,8 @@ namespace Libraries
                         Log.Error(msg);
                     }
 
+                    //Log.Finished();
+                    //throw new Exception("Diagnostic errors found.");
                 }
             }
         }

Multiparagraphs with <param>

As suggested here: dotnet/runtime#46716 (comment)

If a description has multiple paragraphs, insert each paragraph with <param></param>

Red message saying [Not saved] if user forgot to add -Save true

Bug: Port only `<paramref>` descriptions that are found inside the `<TypeParams>` section

Take this API:

https://github.com/dotnet/runtime/blob/fe1f92e4dc49a396049e8fe0097ed3adefd3dedd/src/libraries/System.Private.CoreLib/src/System/Runtime/Intrinsics/Vector64.cs#L91-L99

The comments and signature currently look like this in main:

        /// <summary>Reinterprets a <see cref="Vector64{T}" /> as a new <see cref="Vector64{U}" />.</summary>
        /// <typeparam name="TFrom">The type of the input vector.</typeparam>
        /// <typeparam name="TTo">The type of the vector <paramref name="vector" /> should be reinterpreted as.</typeparam>
        /// <param name="vector">The vector to reinterpret.</param>
        /// <returns><paramref name="vector" /> reinterpreted as a new <see cref="Vector64{U}" />.</returns>
        /// <exception cref="NotSupportedException">The type of <paramref name="vector" /> (<typeparamref name="TFrom" />) or the type of the target (<typeparamref name="TTo" />) is not supported.</exception>
        [Intrinsic]
        [MethodImpl(MethodImplOptions.AggressiveInlining)]
        public static Vector64<TTo> As<TFrom, TTo>(this Vector64<TFrom> vector)

This is how the intellisense file generates the documentation for the As method:

        <member name="M:System.Runtime.Intrinsics.Vector64.As``2(System.Runtime.Intrinsics.Vector64{``0})">
            <summary>Reinterprets a <see cref="T:System.Runtime.Intrinsics.Vector64`1" /> as a new <see cref="T:System.Runtime.Intrinsics.Vector64`1" />.</summary>
            <typeparam name="TFrom">The type of the input vector.</typeparam>
            <typeparam name="TTo">The type of the vector <paramref name="vector" /> should be reinterpreted as.</typeparam>
            <param name="vector">The vector to reinterpret.</param>
            <returns><paramref name="vector" /> reinterpreted as a new <see cref="T:System.Runtime.Intrinsics.Vector64`1" />.</returns>
            <exception cref="T:System.NotSupportedException">The type of <paramref name="vector" /> (<typeparamref name="TFrom" />) or the type of the target (<typeparamref name="TTo" />) is not supported.</exception>
        </member>

The typeparams were recently modified in .NET 7.0. Their names used to be T and U back in .NET 6.0 RTM, which is what we currently have in the docs xml:

https://github.com/dotnet/dotnet-api-docs/blob/26ce7bb6462d524b6376e81709abc342d73fac15/xml/System.Runtime.Intrinsics/Vector64.xml#L160-L173

      <TypeParameters>
        <TypeParameter Name="T">
          <Constraints>
            <ParameterAttribute>DefaultConstructorConstraint</ParameterAttribute>
            <ParameterAttribute>NotNullableValueTypeConstraint</ParameterAttribute>
            <BaseTypeName>System.ValueType</BaseTypeName>
          </Constraints>
        </TypeParameter>
        <TypeParameter Name="U">
          <Constraints>
            <ParameterAttribute>DefaultConstructorConstraint</ParameterAttribute>
            <ParameterAttribute>NotNullableValueTypeConstraint</ParameterAttribute>
            <BaseTypeName>System.ValueType</BaseTypeName>
          </Constraints>
        </TypeParameter>
      </TypeParameters>

Since dotnet-api-docs in main only has APIs that are shipped, the triple slash of TFrom and TTo should not be ported automatically, but rather skipped.

Additionally, we need to offer the option (just like we do with normal <params>) to get prompted to map the parameters to the correct one.

Docs Porting Tool improvement

There are several improvements we need to address to stabilize this tool for usage in all dotnet repos in an automated fashion.

P0

Before tackling anything else, we need to solve the following:

• Separate the two directions into two separate processes. This will help ensure we keep concerns separate and simplify maintenance and future improvements.
• Add classes that wrap the dotnet-api-docs xml types and members, but only return xml objects. This will simplify working with Roslyn APIs.
  • Use the code in this branch as insipration: #99 so that the Roslyn code consumes the new xml-returning classes.
• Understand and fix the issue causing the tool to not be able to load Roslyn projects every time we bump dependencies. Use the code in this branch as inspiration: #97

P1

Only tackle this after we fix the P0s:

• ToDocs: tests into actual unit tests, instead of feature tests. First PR addressing this: #110
• ToTripleSlash: move all remarks to their own xml files next to the source code, and point to the file in the triple slash xml <remarks> item.
  • Convert the dotnet-api-docs markdown to normal xml if possible.
• ToTripleSlash: Figure out how to detect examples in remarks efficiently, move them to their own xml files next to the source code, and point to the file in the triple slash <example> item.
• Detect multi-platform documents (*.Unix.cs, *.Windows.cs, etc.) and move their documentation to a single xml file next to the source code, then point to that file in all the triple slash xml items.
• ToDocs: Port the -or- lines in exceptions correctly, to ensure they have enough endlines so they render correctly in MS Docs.
• ToTripleSlash: backport -or- lines in exceptions with <br/> or other features to ensure they get wrapped and save space.
• ToTripleSlash: Add the ability to wrap lines to 120 characters (or a custom number), as requested by WinForms.

P2

• Save logs to file. There used to be a change that used System.Threading.Channels, but it was causing some messages to get lost, or not print well so it was backed-out. We can reuse its code but fix the issues: #75
• Move the code from this tool into a forked arcade repo and adapt the two executables to be consumable by arcade. We already moved this tool to its own repo under dotnet.

Some xrefs for generics produce invalid triple slash comments

The System.Buffers and System.Memory areas of dotnet/runtime have XML docs containing <xref> elements for generic types that have the "`" characters escaped as "%60", and this escaped format is being ported into triple slash comments, but the escaped format is invalid.

Example: System.Buffers.IPinnable ends up with the following:

    /// <summary>Provides a mechanism for pinning and unpinning objects to prevent the garbage collector from moving them.</summary>
    /// <remarks>The <xref:System.Buffers.MemoryManager%601> class implements the `IPinnable` interface.</remarks>
    public interface IPinnable

The expected result is:

    /// <summary>Provides a mechanism for pinning and unpinning objects to prevent the garbage collector from moving them.</summary>
    /// <remarks>The <see cref="System.Buffers.MemoryManager{T}" /> class implements the `IPinnable` interface.</remarks>
    public interface IPinnable

Exceptions are getting duplicated, triplicated

Some constructor DocIDs end with #ctor, the tool must convert them to their full method name

Otherwise Roslyn will flag it as a potential warning.

Lists should be rendered with <item>

As suggested here: dotnet/runtime#46716 (comment)

Lists should be rendered with <item>

Example: https://github.com/dotnet/roslyn/blob/b09625add4da1b872e8033d2775ea514786d86e1/src/Features/Core/Portable/DocumentationComments/AbstractDocumentationCommentFormattingService.cs#L43-L56

Ensure endline at the end of the xml file

Consider word wrapping triple slash comments

Requested by @adamsitnik

cref values including generic types on method parameters get formatted incorrectly

While porting from API docs into triple slash comments, a scenario was encountered with <see cref="" /> elements that reference methods accepting generic arguments where the resulting triple slash comments are incorrectly formatted.

Example from ArrayPool`1.xml:

        <param name="clearArray">Indicates whether the contents of the buffer should be cleared before reuse.
        If <paramref name="clearArray" /> is set to <see langword="true" />, and if the pool will store the buffer
        to enable subsequent reuse, the <see cref="M:System.Buffers.ArrayPool`1.Return(`0[],System.Boolean)" />
        method will clear the <paramref name="array" /> of its contents so that a subsequent caller using the
        <see cref="M:System.Buffers.ArrayPool`1.Rent(System.Int32)" /> method will not see the content of the
        previous caller. If <paramref name="clearArray" /> is set to <see langword="false" /> or if the pool will
        release the buffer, the array's contents are left unchanged.</param>

This results in:

        /// <param name="clearArray">Indicates whether the contents of the buffer should be cleared before reuse.
         If <paramref name="clearArray" /> is set to <see langword="true" />, and if the pool will store the buffer
         to enable subsequent reuse, the <see cref="System.Buffers.ArrayPool{T}.Return({T}[],bool)" />
         method will clear the <paramref name="array" /> of its contents so that a subsequent caller using the
         <see cref="System.Buffers.ArrayPool{T}.Rent(int)" /> method will not see the content of the
         previous caller. If <paramref name="clearArray" /> is set to <see langword="false" /> or if the pool will
         release the buffer, the array's contents are left unchanged.</param>

The <see cref="System.Buffers.ArrayPool{T}.Return({T}[],bool)" /> is invalid. It should be <see cref="System.Buffers.ArrayPool{T}.Return(T[], bool)" />.

DocsPortingTool fails to translate some `<xref` tags

The following line:
https://github.com/dotnet/dotnet-api-docs/blob/96b853d5e314fc00f8fa2acad402ddb3a00b76a7/xml/System.IO.Pipelines/PipeWriter.xml#L135

The canceled <xref:System.IO.Pipelines.PipeWriter.FlushAsync(System.Threading.CancellationToken)> or <xref:System.IO.Pipelines.PipeWriter.WriteAsync(System.ReadOnlyMemory{System.Byte},System.Threading.CancellationToken)> operation returns a <xref:System.IO.Pipelines.FlushResult> where <xref:System.IO.Pipelines.FlushResult.IsCanceled> is `true`.

When ported only translated the first of the two <xref tags to <see cref tags. The second was left as <xref

/// <remarks>The canceled <see cref="System.IO.Pipelines.PipeWriter.FlushAsync(System.Threading.CancellationToken)" /> or <xref:System.IO.Pipelines.PipeWriter.WriteAsync(System.ReadOnlyMemory{byte},System.Threading.CancellationToken)> operation returns a <see cref="System.IO.Pipelines.FlushResult" /> where <see cref="System.IO.Pipelines.FlushResult.IsCanceled" /> is <see langword="true" />.</remarks>

Command:

-Direction ToTripleSlash -CsProj c:\src\dotnet\runtime\src\libraries\System.IO.Pipelines\src\System.IO.Pipelines.csproj -docs c:\src\dotnet\dotnet-api-docs\xml -IncludedAssemblies System.IO.Pipelines -IncludedNamespaces System.IO

Using dotnet/runtime add68df26af8c0dab2afb13da1f0ca033753fd0b

Detect attribute decorators like Obsolete

They show up in their own section in the xml files. The tool is not porting those yet.

Remove space at end of xml elements

Low pri.

As suggested here: dotnet/runtime#46716 (comment)

Remove the space at the end of a self-closing xml element like <see cref="" />

MSBuild task to compare generated xmls

Docs for all platforms for all public APIs must be equal.

Save a file with its original encoding

Validate against duplicate or otherwise corrupt content

A lurking bug for sure, but one that would likely cause issues for us elsewhere too. The docs site would probably struggle if duplicates existed. We could benefit from a "validate" mode in this tool that would report on any such issues; we could then define the behavior for invalid content.

Originally posted by @jeffhandley in #77 (comment)

Error messages when prerequisite is not built

If runtime:

• Detect if libs is not built.
• Detect if Unsafe.ilproj is not built.

If other repo:

• Warn that the repo is built.

Output to files

Error messages should go into an .err file.
Information messages into a .log file.
Ported elements should go to a file called ported.csv.
Undoc APIs should all go into a file called undoc.csv.

Remove namespace prefix from docid if it can be resolved

As suggested here dotnet/runtime#46716 (comment)

Use only the type name + API, without the namespace prefix, if the namespace can be resolved in the current file.

Allow selecting each one of the types of elements that are desired to be ported

The user should be able to decide if they want to only port summaries, or only params, or only properties, or only returns, etc.
If nothing is specified, then by default, everything except exceptions get ported.

-IncludedAssemblies is assuming a namespace is an assembly

This also applies to -ExcludedAssemblies.

Need to split the detection of assembly and namespace.

The namespace is used by the folder names in dotnet-api-docs/xml/<folders>.

The assembly is used in the dotnet-api-docs xml files, in the <AssemblyInfo> section.

Some APIs, like those that live in the System namespace, are spread in several different assemblies, so sometimes the tool is unable to detect them properly when the user specifies the actual assembly in the -IncludedAssemblies argument.

Mismatched param names could probably list available options

When a method param name does not have the same name in ref/src as in Docs, an error like this one shows up:

Problem in member M:System.Text.Json.JsonSerializer.SerializeAsync(System.IO.Stream,System.Object,System.Type,System.Text.Json.JsonSerializerOptions,System.Threading.CancellationToken) in file D:\dotnet-api-docs\xml\System.Text.Json\JsonSerializer.xml!
The param from triple slash called 'type' probably exists in code, but the name was not found in Docs. What would you like to do?
    1 - Type the correct name as it shows up in Docs.
    2 - Add the newly detected param to the Docs file (not recommended).
        Note: Whatever your choice, make sure to double check the affected Docs file after the tool finishes executing.

Ideally, and if possible, we should show the available options to choose from, instead of making the user type the expected name manually.

Some users report the csproj files are not getting loaded

Some people are reporting that the tool is not working for them because they are seeing this error when the tool attempts to load the Project instance:

Diagnostic messages found in workspace.OpenProjectAsync - 
D:\runtime\src\libraries\System.Runtime.Intrinsics\src\System.Runtime.Intrinsics.csproj: 
Failure - Msbuild failed when processing the file 'D:\runtime\src\libraries\System.Runtime.Intrinsics\src\System.Runtime.Intrinsics.csproj'
with message: The SDK resolver type "WorkloadSdkResolver" failed to load.
Could not load file or assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
The system cannot find the file specified. 

Santi and I are investigating it.

Add "testhost" to the folders to skip

Inside artifacts\bin, this folder does not contain documentation.

Omit the trailing space in self-closing XML tags

Per https://github.com/dotnet/runtime/pull/48841/files#r585232027, I recommend we change the XML tag generation to omit the trailing space in self-closing tags. That would align with the behavior that VS provides when authoring XML doc comments, even though it's different from what we've done in dotnet-api-docs.