dotnet / runtimelab

This repo is for experimentation and exploring new ideas that may or may not make it into the main dotnet/runtime repo.

.NET Runtime Lab

This repo is for experimentation and exploring new ideas that may or may not make it into the main dotnet/runtime repo. Encouraging .NET Runtime Experiments describes reasons that motivated creating this repository.

Active Experimental Projects

Currently, this repo contains the following experimental projects:

• Utf8String - A new UTF-8 String data type in the runtime.
• FreeBSD - Port of .NET runtime to FreeBSD
• NativeAOT-LLVM - LLVM generation for Native AOT compilation (including Web Assembly)
• NativeAOT-Mint - Mono interpreter ported to Native AOT for dynamic execution support
• ManagedQuic - Fully managed implementation of QUIC protocol
• LLHTTP - a set of flexible, lower-level HTTP APIs.
• CompatibilityPackages - a set of packages which help satisfy binary dependencies of .NETFramework assemblies on .NET.
• Hot-Cold Splitting - Support the hot-cold splitting optimization in crossgen2.
• ManagedZLib - Fully managed implementation of DEFLATE algorithm and GZip/ZLib envelope formats.
• Async - Move support for async state machine generation from the C# compiler to the runtime.
• SwiftBindings - Swift bindings for .NET.

You can create your own experiment, learn more here!

Completed Projects

• s390x - Port of .NET runtime (Mono) to the s390x architecture. The changes were upstreamed to dotnet/runtime.
• DllImportGenerator - Roslyn Source Generator used for generating P/Invoke IL stubs.
• NativeAOT - .NET runtime optimized for ahead of time compilation.
• JsonCodeGen - Code generation for JSON.
• AssemblyBuilder.Save() - Prototyping the implementation of AssemblyBuilder.Save() functionality.
• Green Threads - Prototyping the implementation of green threads.
• NativeAOT for Android - Prototyping the Native AOT for Android apps and Java interop layer.

Filing issues

This repo should contain issues that are tied to the experiments hosted here.

For other issues, please use the following repos:

• For .NET Runtime issues, file in the dotnet/runtime repo
• For .NET SDK issues, file in the dotnet/sdk repo
• For ASP.NET issues, file in the dotnet/aspnetcore repo.

Reporting security issues and security bugs

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.

Also see info about related Microsoft .NET Core and ASP.NET Core Bug Bounty Program.

.NET Foundation

.NET Runtime is a .NET Foundation project.

This project has adopted the code of conduct defined by the Contributor Covenant to clarify expected behavior in our community. For more information, see the .NET Foundation Code of Conduct.

License

.NET (including the runtime repo) is licensed under the MIT license.

runtimelab's Issues

ArrayWithOffset marshalling

Interop/PInvoke/ArrayWithOffset/ArrayWithOffsetTest

JsonSerializableAttribute should have checks on its usages

• JsonCodeGen branch (source generation feature)

Expected Behavior
Expect when users use custom JsonSerializableAttribute with usage like this on a type [JsonSerializable(string, [...], Type, [...])] to not fail silently and to receive user comprehensible message through error handling.

Current Behavior
Silent behavior and compilation error for trying to handle null values.

System.Runtime needs explicit inclusion on creating compilation object

Overview:

When creating a compilation object for unit testing given string-based source code, we get the following error:

Collection: [(12,18): error CS0012: The type 'Attribute' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'., (12,18): error CS0012: The type 'Type' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'., (7,18): error CS0012: The type 'Attribute' is defined in an assembly that is not referenced. You must add a reference to assembly 'System.Runtime, Version=5.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.]

Current workaround can be seen here (by explicitly including system.runtime and loading its assembly):
https://github.com/dotnet/runtimelab/blob/JsonCodeGen/src/libraries/System.Text.Json/System.Text.Json.SourceGeneration.UnitTests/CompilationHelper.cs#L31

Repro:

Comment out the workaround line and run unit tests for System.Text.Json.SourceGeneration.UnitTests

MarshalAs(UnmanagedType.Error)

Interop/StructMarshalling/PInvoke/MarshalStructAsLayoutSeq
Interop/StructMarshalling/ReversePInvoke/MarshalExpStruct/ReversePInvokeManaged/ReversePInvokeTest
Interop/StructMarshalling/ReversePInvoke/MarshalSeqStruct/DelegatePInvoke/DelegatePInvokeTest
Interop/StructMarshalling/ReversePInvoke/MarshalSeqStruct/ReversePInvoke/ReversePInvokeTest

Fail predictably and with better error message for WinRT

Interop/WinRT/WinRT

Typewrapper MetadataLoadContext crashes when file path not present

• JsonCodeGen branch (source generation feature)

Expected Behavior
MetadataLoadContext should be able to be created even if one or more assemblies doesn't have a file path.

_compilation = compilation;
            Dictionary<AssemblyName, IAssemblySymbol> assemblies = compilation.References
                                        .OfType<PortableExecutableReference>()
                                        .ToDictionary(r => AssemblyName.GetAssemblyName(r.FilePath),
                                                      r => (IAssemblySymbol)compilation.GetAssemblyOrModuleSymbol(r)!);

should be

_compilation = compilation;
            Dictionary<AssemblyName, IAssemblySymbol> assemblies = compilation.References
                                        .OfType<PortableExecutableReference>()
                                        .ToDictionary(
                                                r => string.IsNullOrWhiteSpace(r.FilePath) ? 
                                                      new AssemblyName(r.Display) : AssemblyName.GetAssemblyName(r.FilePath),
                                                r => (IAssemblySymbol)compilation.GetAssemblyOrModuleSymbol(r)!);

Comment regarding the name should also be removed:

// REVIEW: We need to figure out full framework
// _assemblies[item.Key.FullName] = item.Value;

Actual Behavior
Currently, given a compilation object where one or more assemblies doesn't have a filepath, the creation of MetadataLoadContext fails and crashes.

Can be seen here:
ba611d8#diff-1ebbb80272a0b60e7ced6bdc12a633ddR198

Define pattern for single-library experiments

Today single library experiments that don't change the runtime/shared-framework don't have a great pattern to follow for build / test. We should decide how these should behave and create a pattern/template to follow.

CoreLib unification

These were ifdeffed out in #51.

• Untangle GCMemoryInfo where Common has definition for datastructures that share CoreCLR's VMs native layout. Mono avoided the problem by PNSEing the API.
• Shared portion of Enum has code that we likely don't want use.

StackTrace doesn't skip "infra" frames at the top of stack

We can see e.g. the StackTrace constructor and a call into Redhawk to get the stack trace.

JIT/Directed/forceinlining/NegativeCases

Invalid interop compatibility

Interop/PInvoke/Array/MarshalArrayAsField/AsLPArray/AsLPArrayTest

Test FAIL: TestLibrary.AssertTestException: Assert.Fail: Expected 'System.TypeLoadException' to be thrown, however 'System.Exception' was thrown. TakeIntArraySeqStructByVal
   at TestLibrary.Assert.HandleFail(String, String) + 0x5a
   at TestLibrary.Assert.Fail(String, Object[]) + 0x64
   at TestLibrary.Assert.Throws[T](Action, String, AssertThrowsOptions) + 0x1e1
   at Test.RunTest1(String) + 0xd4
   at Test.Main(String[]) + 0x29

Interop/PInvoke/Miscellaneous/HandleRef/HandleRefTest

InvalidMarshalPointer_Return
Test Failure: TestLibrary.AssertTestException: Assert.Fail: Expected 'System.Runtime.InteropServices.MarshalDirectiveException' to be thrown, however 'System.Exception' was thrown.
   at TestLibrary.Assert.HandleFail(String, String) + 0x5a
   at TestLibrary.Assert.Fail(String, Object[]) + 0x64
   at TestLibrary.Assert.Throws[T](Action, String, AssertThrowsOptions) + 0x1e1
   at HandleRefTest.Main(String[]) + 0x4e4

Tests that need to be triaged for native exception propagation

Maybe we should have a RuntimeFeature.SupportsNativeExceptions that would only be true on CoreCLR/Windows. The test can then key off that.

There might be other aspects of the test that are valuable to keep on Native AOT:

Interop/NativeLibrary/Callback/CallbackStressTest_TargetWindows
JIT/Directed/pinvoke/calli_excep

Enable nullability checking in DllImportGenerator project.

Today we have nullability partially enabled in the DllImportGenerator code and fully enabled in some of the project files in the experiment. We should update all of the projects to have the nullability annotations and warnings enabled.

UnmanagedFunctionPointer(CallingConvention.ThisCall)

Interop/PInvoke/Miscellaneous/ThisCall/ThisCallTest

More unit testing for TypeWrappers and TypeDiscovery for source generation feature

There are lack of unit testing for the use of typewrappers such as iteration of memebers checks and the use of attributeData from the wrappers around type. We should test these before continuing onto the code generation phase to verify code generation is applied to all necessary members of the type.

There are also lack of unit testing for type discovery for edge cases such as unexpected attribute usage, error handling and referenced types in general.

Since code generation depends on all of the above to be working, we should definitely include testing to assure we have this phase complete before moving on.

Create a benchmarking project using source generation feature

We should benchmark startup-time and throughput comparing current json serializers to the source generated serializers. This would be to measure impact and showcase the feature to the community.

Theoretically there should be improvements as shown in initial experiments mentioned in the comments in the design doc.

Support BestFitMapping in marshalling

Interop\PInvoke\BestFitMapping\Char\Assembly_False_False

Test Failure: TestLibrary.AssertTestException: Assert.AreNotEqual: Expected any value except:[??????]. Actual:[??????]. [Error] Location tcbsb6
   at TestLibrary.Assert.HandleFail(String, String) + 0x5a
   at TestLibrary.Assert.AreNotEqual[T](T, T, String) + 0x71
   at BFM_CharMarshaler.testCharBufferStringBuilder() + 0x18a
   at BFM_CharMarshaler.runTest() + 0x19
   at BFM_CharMarshaler.Main() + 0x86

Make compilation errors during source generation more verbose to the user

Currently within the feature, if the compilation fails in the generation phase, only a generic message is given to the user.

Let's investigate a way to inject messages to roslyn's output compilation warnings during source generations and standarize output messages to the user.

Unification tech debt

Tech debt that we need to pay to RI into dotnet/runtime:

• Build a System.Linq.Expressions that works on both full AOT and JIT-based runtimes (debt introduced in #102)
• Nullable annotations in the CoreLib (nullability warnings in the libraries partition disabled in src/libraries/Directory.Build.props in #70)
• Re-enable IL linker warnings disabled in 00210dc
• #140 undid dotnet/runtime#41966. We should find a way to do what we need while not regressing size of CoreCLR-based packages (unless we want to use the same package).
• System.IO.Compression.Native build options from #461
• RequiresDynamicCodeAttribute needs to go through API review and we need to annotate ref assemblies (validation for this is currently disabled)

Support CriticalHandle marshalling

Interop/PInvoke/CriticalHandles

Disambiguate CalliMarshallingMethodThunk based on modopts

Name mangling doesn't take modopts into account so we end up with two different signatures having the same simple name.

We should probably drop all modopt except for the one specifying calling convention.

Might need to add code to respect the calling convention too.

Test: baseservices/callconvs/TestCallingConventions

Reenable vxsort in GC

This was disabled due to missing symbols.

The VXSort feature enables intrinsic-optimized sorting within the garbage collector.

Sourcegenerator support for custom converters.

Current base metadata for known collections and values don't support user defined converters.

Initial todo explanation can be seen here.

Example of needed code:

//if (_options.HasCustomConverters)
  //{
  //JsonConverter<DateTime> converter = (JsonConverter<DateTime>)_options.GetConverter(typeof(DateTime));
  //_dateTime = new JsonValueInfo<DateTime>(converter, _options);
//}```

Tests that need to be triaged for their Ref.Emit use

These likely don't need to ref emit. They might just have a scenario where they ref emit that could be skipped based on a RuntimeFeature check:

• baseservices/TieredCompilation/TieredVtableMethodTests
• Interop/UnmanagedCallersOnly/UnmanagedCallersOnlyTest
• JIT/Regression/JitBlue/GitHub_25020/GitHub_25020
• Loader/classloader/DictionaryExpansion/DictionaryExpansion

VBByRefStr marshalling

Interop/StringMarshalling/VBByRefStr/VBByRefStrTest

AsAny marshalling for arrays and strings

Interop/PInvoke/AsAny/AsAnyTest

Implement ICustomMarshaler

Related tests:

Interop/ICustomMarshaler/ConflictingNames/SameNameDifferentAssembly
Interop/ICustomMarshaler/Primitives/ICustomMarshaler_TargetWindows
Interop/PInvoke/CustomMarshalers/CustomMarshalersTest

Move Utf8String experiment to dotnet/runtimelab

• Create Utf8String branch that is clone of dotnet/runtime
• (optional) Trim the the build in this branch so that it builds just ref pack, runtimepack and the Utf8String package, CoreCLR-only, no mono.
• Add lightweight pre-checkin CI that just builds say Windows and Linux, and runs tests on it
• Add post-commit CI that publishes ref pack, runtimepack and the Utf8String package into a new feed that we will setup for runtimelab packages.
• Add readme and notes for how to setup a new experiment (in dotnet/runtimelab:master)
• Delete the experimental code in dotnet/runtime

Tests that need to be triaged for their Assembly.Location use

Interop/NativeLibrary/API/NativeLibraryTests
Interop/NativeLibrary/AssemblyLoadContext/ResolveUnmanagedDllTests
Interop/PInvoke/Attributes/SuppressGCTransition/SuppressGCTransitionTest
Loader/AssemblyDependencyResolver/AssemblyDependencyResolverTests/AssemblyDependencyResolverTests
Loader/AssemblyLoadContext30Extensions/AssemblyLoadContext30Extensions
Loader/ContextualReflection/ContextualReflection

Full name instead of name for assemblies in metadataloadcontext wrapper

Currently, we do not construct an assembly full name and this satisfies our current problems. For production, we would want this to be resolved in the near future.

Code can be seen here:
https://github.com/dotnet/runtimelab/blob/JsonCodeGen/src/libraries/System.Text.Json/System.Text.Json.SourceGeneration/ReflectionUtils/MetadataLoadContext.cs#L27

Support DateTime marshalling

Interop/PInvoke/DateTime/DateTimeTest

Infrastructure cleanup

• Remove old ObjectWriter feed from NuGet.config

Current Source generation for collections relies on interface checks

Currently, we are checking if a typewrapper of List implements IList and if Dictionary implments IDictionary.
This may cause unwanted behavior when new collections are added.

TypeWrapper.IsAssignableFrom() currently has a bug not supporting this check causing the workaround mentioned above.

IsAssignableFrom() should be fixed in order to support the following code:

        public static bool IsIDictionary(this Type type)
        {
            return type.IsAssignableFrom(typeof(Dictionary<,>));
        }

Handling of recursive native layout

Interop/LayoutClass/LayoutClassTest

Testing that using this type in interop produces a TypeLoadException. The AOT compiler runs out of stack instead.

    [StructLayout(LayoutKind.Sequential)]
    public class RecursiveTestClass
    {
        public RecursiveTestStruct s;
    }

    public struct RecursiveTestStruct
    {
        public RecursiveTestClass c;
    }

Code generation for POCOs with Collections

Using previous structure for simple POCOs with value types we should support collections and create tests to make sure they work.

This feature should also be aligned with current json serializer behavior handling collections.

considered api branch:
master...steveharter:ApiAdds

Triage tests that depend on COM

Most of these already key off of Windows so that they don't run on Unix. The rest of the test might be valuable:

Interop/PInvoke/Delegate/DelegateTest (IDispatch)
PInvoke/Array/MarshalArrayAsParam/AsDefault/AsDefaultTest (VARIANT)
Interop/PInvoke/Array/MarshalArrayAsParam/AsLPArray/AsLPArrayTest (VARIANT)

TypeDiscovery failure when aliasing attribute

• JsonCodeGen branch (Source Generation project)

Minimal Repro
Any aliasing on JsonSerializableAttribute such as the following examples:

https://sharplab.io/#v2:EYLgxg9gTgpgtADwGwBYA0AXEBDAzhgHwAEAmARgFgAoIgBgAIBBAOwgwAsYoA5bAWxi4ADtjAx6AXnpEyAbmp16AZQgCOAS2YBzAKIAbXDB3MM62HoCek6WQB0Sruux71AL2zA9MRhgxR1wACuGDDyVNQA2g7+zm4eXgC6CgDM0iRM9ADeAL7UkTL2jrHunt6+/kEhSTSppPQAQlm54VQRLGycPPyCImKFMS4liSlp9ADCTXmtKmrsmroGRiZmMJbVRLXpACJNQA===

Expected Behavior
Semantic model comparison for attributes instead of name comparison using syntax tokens.

Actual Behavior
Currently for JsonCodeGen branch, type discovery relies on string comparison on attribute syntax name as shown below

// src/libraries/System.Text.Json/System.Text.Json.SourceGeneration/JsonSerializableSyntaxReceiver.cs
serializableAttributes = attributeList.Attributes.Where(node => (node is AttributeSyntax attr && attr.Name.ToString() == "JsonSerializable")).Cast<AttributeSyntax>();

Block generic COM interfaces

Looks like this test only needs us to declare generic COM interfaces as unsupported, which we can totally do. The rest of the test looks valuable.

Interop/PInvoke/Generics/GenericsTest

Source file deduplication

Tracking issue for cleaning up duplicate files between CoreCLR and NativeAOT

• nativeaot\Runtime\unix\no_sal2.h
• nativeaot\Runtime\unix\pshpack1.h
• nativeaot\Runtime\unix\pshpack4.h
• nativeaot\Runtime\unix\sal.h
• nativeaot\Runtime\unix\specstrings.h
• nativeaot\Runtime\unix\specstrings_strict.h
• nativeaot\Runtime\unix\unixasmmacros*.inc
• src\coreclr\src\nativeaot\libunwind and src\coreclr\src\pal\src\libunwind
• src\coreclr\src\nativeaot\System.Private.Reflection.Core\src and src\libraries\System.Reflection.MetadataLoadContext\src
• nativeaot\Runtime\sha1.cpp
• nativeaot\Runtime\slist.*
• nativeaot\Runtime\shash.*
• nativeaot\Runtime\yieldprocessornormalized.*
• nativeaot\Runtime\eventtrace.cpp
• nativeaot\Runtime\stressLog.cpp
• nativeaot\Runtime\inc\stressLog.h

BSTR marshalling

Interop/PInvoke/Array/MarshalArrayAsField/AsByValArray/AsByValArrayTest
Interop/PInvoke/SizeParamIndex/
Interop/StringMarshalling/AnsiBSTR/AnsiBStrTest
Interop/StringMarshalling/BSTR/BSTRTest

Implement support for default interface methods

• Support in the managed type system (shared with Crossgen2)
• Support in the compiler and runtime dispatch
• Support in reflection

Add current System.Text.Json.Serialization attributes to source generation feature

There are current System.Text.Json.Serialization attributes that are not compatible with source generation and we should investigate and document how we handle each one.

The ideal scenario would be to support most if not all the current attribute features.

Inital code generation for POCOs source generation feature

We should start structuring basic code generation structures using the new api overloads. This can be done by using a generated DLL or for System.Text.Json or the ref file from the considered api branch.

considered api branch:
master...steveharter:ApiAdds

This would show us the problems and aspects to consider when code generating for json serialization to come up with different approaches and flesh out how code generation would look like overall.

.Net will get better and better, and recommend a JSON library . [Swifter.Json]

Hello everyone! I recommend a JSON library；https://github.com/Dogwei/Swifter.Json
This is supposed to be the fastest-performing JSON on earth, and it has a full range of functions. I recommend it. You may not like such exaggeration, but it is certainly excellent compared to the performance of JSON libraries in different programming languages. It use Emit and Unsafe.

Field support in Json Source Generator

Currently, due to the extra effort needed and the limited time, only properties are supported.

Currently the type representation along with the code branches allow us to use fields however the code generation and how the code would look like would need to be defined.

Move native AOT experiment to dotnet/runtimelab

Directory mappings

Compiler

ILCompiler -> src\coreclr\src\tools\aot\...
ILCompiler.Build.Tasks -> src\coreclr\src\tools\aot\...
ILCompiler.Compiler -> src\coreclr\src\tools\aot\...
ILCompiler.MetadataTransform -> src\coreclr\src\tools\aot\...
ILCompiler.MetadataWriter -> src\coreclr\src\tools\aot\...
ILCompiler.RyuJit -> src\coreclr\src\tools\aot\...
ILCompiler.TypeSystem -> src\coreclr\src\tools\aot\...

Libraries

Runtime.Base - consider folding into src\coreclr\src\nativeaot\System.Private.CoreLib
System.Private.CoreLib -> src\coreclr\src\nativeaot\System.Private.CoreLib (consider unifying with src\coreclr\src\System.Private.CoreLib later)
System.Private.DeveloperExperience.Console -> src\coreclr\src\nativeaot\...
System.Private.Interop -> src\coreclr\src\nativeaot\... src\coreclr\src\nativeaot\System.Private.Interop (consider folding into System.Private.CoreLib)
System.Private.Reflection.Core -> src\coreclr\src\nativeaot\... (deduplicate with System.Reflection.MetadataLoadContext later)
System.Private.Reflection.Execution -> src\coreclr\src\nativeaot\...
System.Private.Reflection.Metadata -> src\coreclr\src\nativeaot\...
System.Private.StackTraceGenerator -> src\coreclr\src\nativeaot\...
System.Private.StackTraceMetadata -> src\coreclr\src\nativeaot\...
System.Private.TypeLoader -> src\coreclr\src\nativeaot...(consider factoring out USG support later)System.Private.DisabledReflection->src\coreclr\src\nativeaot... Test.CoreLib->src\coreclr\src\nativeaot...`

Unmanaged runtime

Native\Bootstrap -> src\coreclr\src\nativeaot\Bootstrap
Native\Common -> fold into src\libraries\Native\Unix\System.Native
Native\inc -> fold into src\coreclr\src\nativeaot\Runtime
Native\libunwind -> src\coreclr\src\nativeaot\libunwind (consider replacing with src\coreclr\src\pal\src\libunwind later)
Native\ObjWriter -> src\coreclr\src\tools\aot\ObjWriter
Native\Runtime -> src\coreclr\src\nativeaot\Runtime
Native\System.Private.CoreLib.Native -> fold into src\libraries\Native\Unix\System.Native

Infrastructure

BuildIntegration -> src\coreclr\src\nativeaot\BuildIntegration
Common -> distribute among src\libraries\common, src\coreclr\src\tools\common, src\coreclr\src\nativeaot\Common, etc.
Framework -> various

Existing directories

Native\Runtime\coreclr -> src\coreclr\src\inc,vm
System.Private.CoreLib\shared -> src\libraries\System.Private.CoreLib\src
Native\gc -> src\coreclr\src\gc
Native\jitinterface -> src\coreclr\src\tools\aot\jitinterface (rename crossgen2 to aot)
ILCompiler.DependencyAnalysisFramework -> src\coreclr\src\tools\aot\ILCompiler.DependencyAnalysisFramework
JitInterface -> src\coreclr\src\tools\Common\JitInterface

Phase 2 - more experimental components, potentially separate branch(es)

Native\System.Private.TypeLoader.Native - needed for USG only
System.Private.Interpreter
System.Private.Jit
ILCompiler.WebAssembly
ILCompiler.CppCodeGen

Support IDynamicInterfaceCastable

New feature in .NET 5.

Depends on #161.

Vector struct packing

Interop/StructPacking/StructPacking

Throw when LCID marshalling is needed

We might want to at least fix the bad failure mode.

Interop/PInvoke/Attributes/LCID/LCIDTest

Roundtripping delegates through a native function pointer

Interop/MarshalAPI/FunctionPointer/FunctionPtrTest

Support Decimal marshalling

Interop/PInvoke/Decimal/DecimalTest