jaredpar / complog Goto Github PK

In the scratch project, I tried to do this:

var filePath = @"full_path_to_complog";
VerifyAll(filePath);

But the generators fail to load. I had to do some changes for it to work properly:

First is:

- VerifyAll(filePath);
+ VerifyAll(filePath, new BasicAnalyzerHostOptions(BasicAnalyzerKind.OnDisk));

Second is in BasicAnalyzerHostOnDisk:

-        var name = AssemblyName.GetAssemblyName(fullPath);
-        return LoadFromAssemblyName(name);
+        return Assembly.LoadFrom(fullPath);

Note: Assembly.LoadFrom is already used under .NET Framework, but the scratch project is .NET 7 and assembly loading did only work with Assembly.LoadFrom for me.

cc @jaredpar

The extenison should be complog. The tool is complog, there for it should be the extension.

Need to consider expanding the set of target frameworks supported in Basic.CompilerLog.Util.

• Need net6.0: it is the current LTS.
• Consider net472: this will limit our options for how BasicAnalyzers are implemented. That is certainly a downside but it's no worse than running the compiler on net472. Having net472 would improve our ability to say profile compiler differences between .NET Framework and .NET Core so this could be a substantial win.
• Consider netstandard2.0: if net472 support is added then should consider netstandard2.0 as well. It won't be much work, if any, to support that and it would mean Basic.CompilerLog.Util could be used anywhere the compiler libraries are used.

Items that could be included for NuPkg

• Where a package was downloaded from
• What packages a project used (direct or indirect)

Another mechanism for creating compiler logs would be to use an MSBuild logger. The only requirements of creating a compiler log are:

1. Need to be on the same machine the build occurs on.
2. Need access to the csc and vbc command line.

That should all be possible from a logger. The advantage to a logger is that it will be significantly simpler to integrate into build pipelines than the global tool approach.

The complog create command faithfully packs up the details and artifacts necessary to re-run a compilation. That includes packing up the full set of /analyzer references, including source generators, and including them in the log. That means code that analyzers a compiler log ends up running analyzers and generators locally.

That in many ways is an explicit design goal. It's important for example such that a compiler team can faithfully debug bugs occuring on customer mahcines.

For other scenarios it's undesirable. The generator and analyzer represent foreign code and they should be running only on the machine that created the log. As such we should provide an option that does this and creates a compiler log that is free from analyzer and generators.

Note: there is one complication here. Diagnostic suppressors are analyzers that push warnings to non-diagnostic. That means removing all analyzers from a build can convert a successful build to a failing one. It could have /warnaserror and a suppressor which is suppressing a warning. As such this feature probably wants to remove all warning to error promotion.

Right now everything is a DLL reference, needs to support compilation references. Just unwinding the MVID

Running the replay command without arguments re-runs the compilation in memory. That doesn't output anything to disk yet the command is outputting a path

> complog replay
Outputting to ...

That message should only appear if --emit is used

When the compiler is hosted as an API today, the binaries are loaded from our referenced NuGet packages. This works in the vast majority of scenarios due to the back compat nature of the compiler. There are times though when it's advantageous to load a different compiler:

1. Want to use the exact compiler contained in the log.
2. Want to use a compiler from a build of dotnet/roslyn. Would be very useful for debugging.
3. Want to use a compiler from a specific .NET SDK.

This can be done when running on .NET Core through AssemblyLoadContext usage. Likely have to re-engineer the code a bit but this could be supported.

Couple sources of confusion:

• The name is emit at the top level but help says replay
• There is an --emit option ... weird.

Need to know at least:

• Operating system the log was created on as logs aren't portabble between operating systems
• Add a version number so we can be more tolerant of changes in the format

At the moment the package name is Basic.CompilerLog yet the tool name is complog. That was done basically because I didn't have a lot of experience with global tools. It's much more common for the tool name and package name to match. Going to change them to match.

That does mean existing users will need to uninstall / reinstall the package but it's a one time hit.

The create command should issue an error if it fails to pack a single project. This will help customers spot when they say use the -p syntax incorrectly and exclude all projects.

Need to merge CompilationReader and CompilerLogReader. The split here is arbitrary and weird. The original goal was to have an interface that could read assets equally well from a binlog or compilerlog. That is better done as an interface.

There are several forms of content that are not currently being embedded and round tripped:

• Resources
• Targets of /embed arguments
• Targets of /keyfile arguments
• Targets of /sourcelink arguments
• /publicsign because we don't support keyfile

These need to be fixed up.

Today the export command creates build files that effectively dotnet exec csc.dll for the generated .rsp file. This means it strongly favors .NET Core based compilers vs .NET Framework ones. For the marority of cases the difference is immaterial. There are a few cases though where the runtime matters:

1. Tracking down certain classes of analyzer bugs
2. Performance investigation

The export command should be changed to emit both files to support these scenarios.

Introduce a complog serve command that fires up a source browser instance on a compiler log and runs the server locally.

https://github.com/jaredpar/SourceBrowser

The new record reading code is mistakenly replacing normal compilations with satellite ones. The issue is that satellite compilations occur within the same project context as the original build. The map expects each of them to be different. Need to handle satellite compilations as separate items here.

The tool should store the original compiler in the complog file. That way export commands can produce a perfect .cmd file for reproducing the behavior on local machines. Can also include a command to extract the compiler onto the local machine for inspection by developers.

The compiler has sever issues with #line directives:

• When the source file is embedded it does not embed the #line target
• Does not do path map such that targets of #line embeds get handled in export

The BasicAnalyzerKind.None is interpretted as pretend there are no analyzers at all. There seems to be room for a couple of other models:

1. No generators but run analyzers
2. Fake generators (have a generator that fakes adding the source) and keep analyzers

At the moment the design is a bit mixed because BasicAnayzerKind includes both how the analyzers are hosted (in memory, on disk) and what is / isn't included (analyzers and generators). It may be better to split that up and then put caching on one or the other.

At the moment when opening a log from a different OS the tool issues a hard error. This should be a warning. Otherwise can't even do things like look at references.

Need to add helper methods to CompilationData that make it easier to call Emit. That has so many conditional arguments. Better to abstract that all away and make it an easier API to use.

This will force the code to expose all of the other content types it's embedding today but not actually surfacing like win32icon, ruleset, etc ...

The Read style APIs use base one indexers right now, should use zero based indexers. Cannot fathom why I did this originally.

Today it results in a silent failure. Need to get the same fallback / diagnostics as core compiler.

Components like razor will write out full paths into .editorconfig file during build.

[C:/projects/scratch/BlazorApp1/BlazorApp1/App.razor]
build_metadata.AdditionalFiles.TargetPath = QXBwLnJhem9y
build_metadata.AdditionalFiles.CssScope = 

This means though when rehydrating locally editorconfig does not take effect because the paths are meaningless. Need to consider remapping these to local paths when reading the files.

The full paths here are necessary in global .editorconfig files so this isn't a fix in the compiler. Have to do it in the rehydration.

This makes it harder to get a log from CI when I want to do something like this:

  - bash: |
      dotnet tool install --global Basic.CompilerLog
      complog create --projectName=Uno.UI $(build.artifactstagingdirectory)\build-$(GitVersion.FullSemVer)-$(XAML_FLAVOR_BUILD).binlog
    condition: always()

A workaround for this issue is:

  - task: CmdLine@2
    inputs:
      workingDirectory: $(build.artifactstagingdirectory)
      script: |
        dotnet tool install --global Basic.CompilerLog
        complog create --projectName=Uno.UI build-$(GitVersion.FullSemVer)-$(XAML_FLAVOR_BUILD).binlog
    condition: always()

• default suffix should be .complog not .compilerlog
• need an --out to customize the name

They are still looked for on the current machine.

Should include a README in the package even if it's just a simple pointer back to the complog repository.