Jerome Laban’s Blog

How to profile C# 9.0 Source Generators

2020-10-10T05:00:00+00:00

C# 9.0 source generators are progressing a lot lately, with the addition of generated symbols IDE navigation, as well as the stabilization of the generation APIs.

As generators are run a part of the compiler’s pipeline, when creating small generators, or processing small inputs, performance profiling can be done quite easily, and even easier using Chris Sienkiewicz’s kittitas tool, which packages the generation driver inside a single, easy to use tool.

Yet, when the build is more complex, or the solution takes an important environmental set of parameters (such as Directory.Build.props), some in-place debugging may be required.

Read on to find out how to do this.

In-Place Profiling of Source Generators using Visual Studio

While debugging the Uno source generators’ migration to C# 9.0, I found out that the generation was taking quite a while, particularly at the last stage of the many-projects solution that builds Uno itself.

As the generators are part of the build pipeline, one way to profile the generators is to actually profile the invocation of csc.exe itself. Note that you can debug your generators using the Debugger.Launch() technique (Nick has a great blog entry about this).

To profile your generators, first you’ll need to :

Install the Visual Studio Project System Tools. Those tools enable the generation of msbuild binlog files from within Visual Studio. Binlogs are even created for design time builds, a nice touch.
Install the amazing MSBuild Binary and Structured Log Viewer, an amazing tool by Kirill Osenkov. You may need to open it once for the .binlog extension to be registered.

Then, you’ll need to do the following:

In the Tools menu, Options, go to Project and Solutions, Build and Run, then select Diagnostics in the MSBuild project build log verbosity options.
Go to the View, Other Windows, Build Logging menu to open the Build Logging tool window
In that Build Logging window, click on the Play icon to start the logging
Build your project which uses the source generators
Once built, in the Build Logging window, select the line that contains your project’s build results, and right click Open Logs External.
In the top left text box, type csc.exe, you’ll find the actual command line visual studio used to build the code from your project, including the generators’ execution.
In tree view on the left, click on the line Property CommandLineArguments = ...
On the right, the full line is selected and right click to Copy Value
Paste the content in your favorite text editor

Now, let’s configure your generators project to be profiled:

Set your generators project as startup in the solution explorer
Open the generators project properties, and the Debug tab
Take the path to your csc.exe in the file created above, and set it to the Executable
Set the Working directory to the folder containing the project that was previously built
Save the file created with the compiler options, while removing the path to csc.exe, leaving only the command line parameters. This file can be anywhere, let’s say C:\Temp\myresponsefile.rsp
In the Application arguments, set the following:
```
  @C:\Temp\myresponsefile.rsp
```
This will instruct the compiler to take its command line parameters from the response file we created.
Go to the Debug, Performance Profiler menu, select CPU Usage
Click Start

And voila!

It’s not the simplest of steps, but it works pretty nicely.

Note that the response file is not strictly necessary, but if your project has lots of files (which is the case for the Uno.UI solution), the Application arguments field is characters limited and may not work properly.

In closing

In the context of debugging Uno’s generators, here’s what came out:

Indicating that the generators are spending a lot of time generating the names of ISymbol instances.

Looks like we’re going to have a good time improving the performance of Uno’s source generators!

Until next time, happy generation!

Using MSBuild Items and Properties in C# 9 Source Generators

2020-09-13T05:00:00+00:00

C# 9.0 Source Generation is progressing quite nicely lately (Thanks, Jared!), with the addition of the ability to interact with the MSBuild environment such as getting Properties and Items to control how the generation happens.

In this post, I’ll explain how to parse .resw files of a project to generate an enum that contains all the resources.

The full sample for this article is here in the Fonderie Generators project.

Reading msbuild items and properties

In the Roslyn generators cookbook, new entries have been added to include the APIs needed to get information from msbuild. In order to make the reading of those properties easier, here’s a small extensions class:

internal static class SourceGeneratorContextExtensions
{
    private const string SourceItemGroupMetadata = "build_metadata.AdditionalFiles.SourceItemGroup";

    public static string GetMSBuildProperty(
        this SourceGeneratorContext context,
        string name,
        string defaultValue = "")
    {
        context.AnalyzerConfigOptions.GlobalOptions.TryGetValue($"build_property.{name}", out var value);
        return value ?? defaultValue;
    }

    public static string[] GetMSBuildItems(this SourceGeneratorContext context, string name)
        => context
            .AdditionalFiles
            .Where(f => context.AnalyzerConfigOptions
                .GetOptions(f)
                .TryGetValue(SourceItemGroupMetadata, out var sourceItemGroup)
                && sourceItemGroup == name)
            .Select(f => f.Path)
            .ToArray();
}

Let’s dive into what those extensions do.

GetMSBuildProperty

The GetMSBuildProperty method is assuming that a defined property has a non-empty value, as per the msbuild semantics. Here’s how to get the default namespace for the current project:

var defineConstants = context.GetMSBuildProperty("RootNamespace");

Assuming that the associated msbuild property is added in the generator’s associated props file:

<ItemGroup>
    <CompilerVisibleProperty Include="RootNamespace" />
</ItemGroup>

GetMSBuildItems

For GetMSBuildItems, since the roslyn APIs does not provide a way to discriminate items per their MSBuild item name, we need to use some metadata that can be added to the AdditionalFiles items. In order to get the resw files from a WinUI project, we can do the following:

var priResources = context.GetMSBuildItems("PRIResource");

For this code to work, we need to change a little bit the way items are added to the roslyn context:

<Target Name="_InjectAdditionalFiles" BeforeTargets="GenerateMSBuildEditorConfigFileShouldRun">
    <ItemGroup>
        <AdditionalFiles Include="@(PRIResource)" SourceItemGroup="PRIResource" />
    </ItemGroup>
</Target>

The use of a target here is needed because of the way NuGet packages property or targets files are handled by msbuild. If the ItemGroup is included directly at the root of the project, its evaluation is performed too early in the build process. This sequence misses items being added by the project or dynamically by other targets.

At this point, there’s no clear injection point to execute this targe in Roslyn, but GenerateMSBuildEditorConfigFileShouldRun seems like an appropriate location for doing so at this point, right before the capture of the properties and items by the build.

Finally, to be able to discriminate items in the AdditionalFiles group, we use the SourceItemGroup metadata. For Roslyn to pick it up, we need to add the following:

<ItemGroup>
	<CompilerVisibleItemMetadata Include="AdditionalFiles" MetadataName="SourceItemGroup" />
</ItemGroup>

Generating code from the resw file

Now that we can read the items and properties, we can write a small generator that creates an enum with all the names found in a resw file:

[Generator]
public class ReswConstantsGenerator : ISourceGenerator
{
    public void Initialize(InitializationContext context)
    {
        // Debugger.Launch(); // Uncomment this line for debugging
        // No initialization required for this one
    }

    public void Execute(SourceGeneratorContext context)
    {
        var resources = context.GetMSBuildItems("PRIResource");

        if (resources.Any())
        {
            var sb = new IndentedStringBuilder();

            using (sb.BlockInvariant($"namespace {context.GetMSBuildProperty("RootNamespace")}"))
            {
                using (sb.BlockInvariant($"internal enum PriResources"))
                {
                    foreach (var item in resources)
                    {
                        XmlDocument doc = new XmlDocument();
                        doc.Load(item);

                        // Extract all localization keys from Win10 resource file
                        var nodes = doc.SelectNodes("//data")
                            .Cast<XmlElement>()
                            .Select(node => node.GetAttribute("name"))
                            .ToArray();

                        foreach (var node in nodes)
                        {
                            sb.AppendLineInvariant($"{node},");
                        }
                    }
                }
            }

            context.AddSource("PriResources", SourceText.From(sb.ToString(), Encoding.UTF8));
        }
    }
}

This will generate a file which contains an enum with all the resource names, in the default namespace of the current assembly.

Note that this generator does not validate the name’s format, and if there are reserved characters or keywords, those are needed to be rewritten for C# to accept it.

Wrapping up

This simple sample should most likely be improved.

For instance, it could be interesting to create a generator analyzing another generator to create a .targets file which contains the appropriate CompilerVisibleItemMetadata and CompilerVisibleProperty for that generator to work properly.

The extension also only supports getting the identity of an item, but getting additional metadata would be useful, like getting the Link attribute when dealing with linked files in projects.

You can find the sample of this article here, and as of the writing of this post, Visual Studio 16.8 Preview 2.1 does not yet show the generated code or highlights properly but builds with the generated code properly. This should be improving the next previews.

Until next time, happy generation!

How to get the SyntaxNode of an ISymbol using Roslyn

2020-07-19T05:00:00+00:00

In this post, I’ll describe how to determine if a property is an auto-property, using its ISymbol as the source, and not by using reflection into Roslyn which computes this information internally.

During the original development of the Uno CodeGen source generators, when building the Immutable generators (soon to be deprecated by the records feature in C# 9.0), we had to determine if a property’s backing field is generated or not.

At the time, not understanding fully the Roslyn sources, and with lack of time, we chose to go the easy and hacky route (with reflection on internal members). And like any hacky route, it always come back at your one way or another. In Roslyn 3.6, internals have changed breaking that reflection based code.

Still, we knew that it was going to break in the future so the error message was very explicit in saying that we were looking for something internal that could not be found.

How determine if a property is an auto-property

Given an IPropertySymbol, it’s possible to get the location of the definition:

if (symbol.Locations.FirstOrDefault() is Location location)

If a symbol has no location, it generally means it’s defined outside the current compilation (e.g. in a referenced assembly).

Then we can get the actual SyntaxNode of that location:

var node = location.SourceTree?.GetRoot()?.FindNode(location.SourceSpan);

The FindNode method uses the location’s source span to lookup the appropriate so it can be traversed and analyzed syntactically. Note that there may not be a SourceTree for the that location either.

Once we have the node, we can analyze it:

var isExplicitProperty = node
    .DescendantNodesAndSelf()
    .OfType<PropertyDeclarationSyntax>()
    .Any(prop =>
    {
        if(prop.AccessorList != null)
        {
            return prop.AccessorList.Accessors
                .Any(a => a.Body != null || a.ExpressionBody != null);
        }

        // readonly arrow property declaration
        return true;
    });

Here’s what happens:

The code take only the PropertyDeclarationSyntax nodes
Determine if there’s an accessor list (are there explicit get or set keywords)
Then if there are accessors, determine if they have a body (with curly braces) or are using the arrow syntax (lambda properties).
If there are no accessors, this means it’s a readonly property with an arrow syntax.

If this, you can know if a property is an auto-property or not.

This code is based on the actual roslyn source code for the internal IsAutoProperty property.

Wrapping up

This code is not future proof, though. Even if it’s not using internal APIs, the language will evolve and may add newer features that may break or ignore newer constructs that will appear in later versions of the language.

That will be the topic of a future post, if that happens :)

INotifyPropertyChanged with C# 9.0 Source Generators

2020-04-29T05:00:00+00:00

In a design meeting far, far away, source generators were designed to be part of C# 6.0, but sadly never came to be. At the time, wanting that source generation feature pretty badly, I went on implementing the specification which later became Uno.SourceGeneration, and it turns out it was the right decision to stick with a very similar API.

I could port the INotifyPropertyChanged (INPC) generator I built a while back that uses Uno.SourceGeneration package, convert it to use Roslyn’s shiny new C# 9.0 feature in a matter of minutes. Amazing!

The INPC Generator

As a quick refresher, the generator works by allowing the creation of a class this way:

partial class MyClass
{
  [GeneratedProperty]
  private string _myProperty;

  partial void OnMyPropertyChanged(string previousValue, string newValue)
  {
    Console.WriteLine($"OnMyPropertyChanged({previousValue}, {newValue}");
  }
}

You’ll notice here that the property is not visible at all, and it will be assumed that the generator will recognize the GeneratedPropertyAttribute and do the rest of the work for the author of the class.

The generator could then produce something like this:

partial class MyClass : INotifyPropertyChanged
{
  public string MyProperty
  {
    get => _myProperty;
    set
    {
      var previous = _myProperty;
      _myProperty = value;
      PropertyChanged?.Invoke(new PropertyChangedEventArgs(nameof(MyProperty)));
      OnMyPropertyChanged(previous, value);
    }
  }

  partial void OnMyPropertyChanged(string previousValue, string newValue);
}

This generator is generating the property on your behalf, and automatically implements the INotifyPropertyChanged as well as a partial method that you can optionally implement to get local notification on changes.

For a more detailed explanation of the generation process, head to this earlier article.

Upgrading the source generator to C# 9.0

First, we’ll need the INPC.Generator Sample Project to use the the C# 9.0 preview:

<PropertyGroup>
  <LangVersion>preview</LangVersion>
</PropertyGroup>

Then change the NuGet packages from Uno.SourceGeneration to Roslyn’s Analyzers:

<ItemGroup>
	<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="3.6.0-3.20207.2" PrivateAssets="all" />
	<PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.0.0-beta2.final" PrivateAssets="all" />
</ItemGroup>

Then we’ll need to adjust the source code to be use the new interfaces:

[Generator]
public class MySourceGenerator : ISourceGenerator
{
    public void Initialize(InitializationContext context)
    {
        // No initialization required for this one
    }

    public void Execute(SourceGeneratorContext context)
    {
        // Generation
    }
}

The execute method in Uno.SourceGeneration has the same signature, and the new SourceGeneratorContext type the only property that really matters Compilation.

The method to provide the generated source has changed a bit, and uses a SourceText instead of a string:

context.AddSource(sanitizedName, SourceText.From(builder.ToString(), Encoding.UTF8));

We’ll need to update the path to the binary, where SourceGenerator becomes Analyzer, in the INPC.Generator.props file:

<ItemGroup>
    <Analyzer Include="$(MSBuildThisFileDirectory)..\bin\$(Configuration)\netstandard2.0\INPC.Generator.dll"
            Condition="Exists('$(MSBuildThisFileDirectory)..\bin')" />
    <Analyzer Include="$(MSBuildThisFileDirectory)..\tools\INPC.Generator.dll"
            Condition="Exists('$(MSBuildThisFileDirectory)..\tools')" />
</ItemGroup>

And finally, we simply need to compile the main program with the C# 9.0 preview.

When running the sample, here’s what happens:

OnIntPropertyChanged(0,42)
OnIntPropertyChanged(,My 42)

The job is done!

In closing

I hope the feature will grow, and provide lots more like access to msbuild properties, or dependencies between generators, and it will finally fix the issues with dynamically generated <Compile /> item groups.

I hope you will give this feature a try, as it opens up a world of possibilities, as we’ve been finding out while developing Uno (Here’s the list of generators), as well as the Uno.CodeGen generators (which some may not be needed anymore with the C# 9.0 records feature!).

Happy source-generation!

C# interop with C/C++ and Rust in WebAssembly

2020-03-22T05:00:00+00:00

Having the ability to call code written in other languages is increasingly important, as there are many very useful libraries that are getting ported over to WebAssembly. In .NET, the common defined way for doing interop is P/Invoke and DllImport, and .NET for WebAssembly has support for it in the form of static linking of LLVM Bitcode object files.

In this article, I will walk through how to call some simple C/C++ and Rust code from C# in a WebAssembly app.

In the general .NET sense, P/Invoke was built to perform dynamic linking with Windows PE Dlls, but has been extended in mono to allow for static linking. This technique is also used by mono on iOS to call native code, and allows for a single executable package to contain the code to execute the application. This is what is used for the demos in this article.

Setting up the C# WebAssembly project

First let’s create create a .NET WebAssembly app, using the Uno WebAssembly Bootstrapper.

Note that macOS is not yet supported for the static linking scenario, and that on Windows 10 you’ll need to have WSL installed.

Create a .NET Standard 2.0 Library in Visual Studio for Windows or using dotnet new classlib under linux.

Replace the content of the project with the following:

  <Project Sdk="Microsoft.NET.Sdk.Web">

      <PropertyGroup>
          <OutputType>Exe</OutputType>
          <TargetFramework>netstandard2.0</TargetFramework>
          <MonoRuntimeDebuggerEnabled>false</MonoRuntimeDebuggerEnabled>
          <WasmShellMonoRuntimeExecutionMode>InterpreterAndAOT</WasmShellMonoRuntimeExecutionMode>
      </PropertyGroup>

      <ItemGroup>
          <PackageReference Include="Uno.Wasm.Bootstrap" Version="1.1.0-dev.426" />
          <DotNetCliToolReference Include="Uno.Wasm.Bootstrap.Cli" Version="1.1.0-dev.426" />
      </ItemGroup>

      <ItemGroup>
          <!-- This automatically includes any Bitcode file for static linking -->
          <Content Include="*.bc" />
      </ItemGroup>

  </Project>

Create a file named Program.cs

  using System;

  namespace MyApp
  {
      public class Program
      {
          static int Main(string[] args)
          {
              return 0;
          }
      }
  }

Build the app once. This will download the .NET WebAssembly SDK, and install emscripten for the app.

Build a C/C++ library

To build a C/C++ library, we’ll need to create a simple file that contains an extern "C" exported function, in order to have a signature and calling convention that can be used properly with .NET P/Invoke.

In the WebAssembly app, let’s create a folder named myclib and add a file named mylib.cpp:

#include <stdio.h>

extern "C" {
    int cpp_add(int a, int b) {
        return a + b;
    }
}

Save the file, and open a bash or WSL window.

We can go the path containing the file using the great wslpath tool:

cd `wslpath "C:\YourPathToYourProject\myclib"`

We’ll need to initialize emscripten:

source ../obj/emsdk-*/emsdk/emsdk_env.sh

Then build the library:

emcc mylib.cpp -r -o ../myclib.bc -s WASM=1

At this point, we’ve generated a Bitcode file that can be used by the .NET tool chain, and for which an extern "C" marked function can be called from C#.

Call the C++ function from C#

Now that we have our library built, we can update our C# program to make the C++ function callable:

using System.Runtime.InteropServices;

public class Program
{
    [DllImport("myclib")]
    private static extern int cpp_add(int a, int b);

    static int Main(string[] args)
    {
        Console.WriteLine($"cpp_add: {cpp_add(21,21)}");
        return 0;
    }
}

When building the app, and running it, this will appear in the browser’s console:

cpp_add: 42

Build a Rust static library

Following a similar path, to build a Rust static library we’ll need to create a simple file that contains a function marked with the #[export_name] attribute, so that it can be found via P/Invoke.

In the WebAssembly app, let’s create a folder named myrustlib and add a file named mylib.rs:

mod tests {
    #[export_name = "rust_add"]
    pub extern "C" fn rust_add(a: u32, b: u32) -> u32 {
        return a + b;
    }
}

Save the file, and open a bash window.

Let’s setup Rust, Rustup and add Cargo to your path:

curl https://sh.rustup.rs -sSf | sh
set PATH=$PATH:$HOME/.cargo/bin

Then setup WebAssembly support for Rust:

rustup install stable
rustup default stable
rustup target add wasm32-unknown-emscripten

We’ll need to initialize emscripten here as well, if not done previously:

source ../obj/emsdk-*/emsdk/emsdk_env.sh

Now we can go the path containing the file:

cd `wslpath "C:\YourPathToYourProject\myrustlib"`

Then build the library

rustc --target=wasm32-unknown-emscripten mylib.rs --crate-type=staticlib -o ../myrustlib.bc

In a similar way we’ve done this for C++, the Bitcode file is now available for the .NET tool chain to use. The staticlib parameter is important as it forces the rust compiler to create a standalone library, with all its internal support code included.

Calling the C++ function from C#

With the Rust library built, we can update our C# program to make the Rust function callable:

using System.Runtime.InteropServices;

public class Program
{
    [DllImport("myclib")]
    private static extern int cpp_add(int a, int b);

    [DllImport("myrustlib")]
    private static extern int rust_add(int a, int b);

    static int Main(string[] args)
    {
        Console.WriteLine($"cpp_add: {cpp_add(21,21)}");
        Console.WriteLine($"rust_add: {rust_add(21,22)}");
        return 0;
    }
}

When building and run the app again, the following will appear in the browser’s console:

cpp_add: 42
rust_add: 43

Current set of restrictions for P/Invoke

Under the covers, mono is generating a table of methods marked DllImport, and generates a set of callable methods from referenced external libraries.

The runtime needs to do so in order to determine what to call. Once a library and function has been found, the runtime has to determine the signature of the function in a pre-defined signatures list.

If the native function signature is not in the list, you may encounter the CANNOT HANDLE COOKIE XXXX assertion defined here. If you’re facing this error, you may want to adjust your native function signature so it finds itself in the pre-defined signatures list.

All of this is caused by the fact that, for security reasons, WebAssembly functions cannot be called with a set of parameters unknown at compile time; a technique the mono runtime has to use when calling functions through P/Invoke.

If you’re wondering how to determine the cookie of a function, as an example this function:

static extern int cpp_add(int a, int b);

has the cookie III, where the first character defines the return type.

Another example with this function:

static extern void OtherFunction(int a, double b, float c);

will have the VIDF cookie.

WebAssembly validations

Along with the restrictions of the P/Invoke list, making sure that the signature of the function tagged in the C# with DllImport matches the function defined in the other libraries. In case of a mismatch, browsers will raise an error such as RuntimeError: function signature mismatch.

Next up…

We’ll discuss how to use strings back and forth in both environments.

.NET Threading and WebAssembly

2020-02-29T05:00:00+00:00

Threading, in general operating systems sense, is not something that the web has been able to use until very recently. The addition of Threads support in WebAssembly, and the activation of the threading support in Chrome opens up a whole new world of possibilities, including the use of Reactive Extensions (Rx.NET) or the Task Parallel Library (TPL).

Let’s dive in, with some sample code.

A bit of history

The only technique that was available for actual local parallelization of work in the javascript land was to use WebWorkers. It’s technically not threading in the same sense known by out-of-browsers developers, as it does not provide the ability to share memory between WebWorkers. To synchronize work, messages are passed between workers and the main loop, which looks more like processes would exchange messages via IPC.

This changed recently with the ability for javascript to create shared array buffers, contiguous pieces of memory that can be both read and written from workers and the main loop. Those buffers can only be mapped to primitive types, for which access and manipulated can be synchronized via atomic operations. Atomics are also used to perform signalling between threads.

Those buffers had been disabled for a while because of CPU attacks Spectre and Meltdown, but are now enabled by default in Chrome and the new Edge.

Threads in Emscripten

WebAssembly Threads are supported in Emscripten via the pthreads library and are backed by WebWorkers.

When threads are created, new WebWorkers are created and provided with a set of information, such as stack size, thread ID, shared memory, etc… The same WebAssembly module as the main loop one is loaded in memory in the worker, and executes the entry point requested for the thread.

One interesting aspect of threading is that the creation of WebWorkers needs main loop to yield. This means that if the main loop does not yield control back to the environment, threads may never get the chance to start. That’s why Emscripten provides a way set of workers (none by default) to be created before executing any code.

At this point, it is important to note that if the atomics feature is not enabled in the browser (e.g. in Firefox or Safari), the emscripten built app will fail to start. This will most probably one of the reason that the Uno.Wasm.Bootstrap project will include multi-configuration generation based on browsers capabilities.

Threads in .NET for WebAssembly

.NET for WebAssembly now supports the ability to create threads, as the runtime (Mono) uses pthreads. All the existing internal .NET threading APIs have been enabled to make use of pthreads as they do for other platforms, and the System.Threading becomes available for use.

With this it becomes possible to use Monitor (with lock() statements), AutoResetEvent and ManualResetEvent and other synchronization primitives are working as intended between threads.

The ThreadPool is also available, along with System.Threading.Thread.ManagedThreadId, thread names and thread local storage (TLS).

Trying out WebAssembly Threading with Uno.Wasm.Bootstrap

The Uno.Wasm.Bootstrap package provides the configuration to enable threading in .NET for WebAssembly by using the latest 1.1-dev package, and changing the active runtime mode.

This can be done in the project file like this:

<MonoWasmRuntimeConfiguration>threads-release</MonoWasmRuntimeConfiguration>

After that, creating a thread becomes possible (view the full project here):

Which produces the following output:

[tid:1] Startup
[tid:1] Waiting for completion source
[tid:2] Thread begin
[tid:2] Waiting for event
[tid:1] Got task result, raising event
[tid:1] Main thread exiting
[tid:2] Got event, terminating thread

The execution is now interleaved properly, as expected when running this sample in common .NET environment.

This sample is build to work with no available upfront WebWorkers, which is why the Run method is invoked as Fire and Forget, so that the main loop can get the chance to start the workers.

The bootstrapper configuration does not yet provide a way to change the preexisting workers, but will soon get it.

Threading affinity

API thread affinity is a tricky subject. Most browser APIs can only be invoked from the main loop, such as DOM manipulation.

Emscripten provides a feature called “Proxying” which detects APIs need to be invoked on the main loop. The user code is then rewriten to create a blocking call in the worker until the method execution finishes on the other thread. This execution is done through WebWorkers message passing, but this is not something that we’ll be able to use in .NET, as IL does not use the proper APIs to executed proxied code.

Along with those limitations are the inability for the main loop to be blocked. Emscripten provides a way to seemingly block the main thread through spinlocks, though that’s generally not a good idea for the end user perceived performance.

Coming next…

I’ll continue the story on how enable threading in the CoreDispatcher.RunAsync() in the Uno Platform.

Roslyn Source Generation Reborn, the replace keyword and INotifyPropertyChanged

2019-12-08T05:00:00+00:00

A very long time ago, during the C# 6.0 time frame, a Source Generation proposal was added to the list of possible features, it was abandoned, but in a recent PR, the Roslyn team is taking a look again at the feature proposal, as there are lots of generation happen around Microsoft that could benefit from an integrated story.

The original feature of C# 6.0

One of its goals were to tackle the INotifyPropertyChanged (INPC) problem, where having to manually raise the PropertyChanged event was a very repetitive task. That feature could have enabled very interesting scenarios such as the inclusion of the replace keyword. It could allow for a source generator to replace the content of a method (a property setter for instance), with another method, while allowing it to call back the original method through an original keyword.

The objectives were probably a bit too broad at the time. The implications with the build pipeline as a whole, and particularly with the IDE were a bit too large to chew. I discussed that a little bit in article I wrote a few months ago.

At time though, I though it would still be interesting to write a source generator that would have less ambitious goals, but still would allow for source generation in a build pipeline, even if it would not be tightly integrated in Roslyn. That’s how the Uno.SourceGeneration project was born.

It’s still going strong as it is used throughout Uno, to solve a large variety of problems.

Source Generation reborn

In most recent use cases at Microsoft, the use of T4 templates seems to be the choice, and it has its set of issues such as build performance, the inability to work on partially valid or complete source trees, the inability to use information from the syntactic model or the duplication of the semantic model work.

The team is looking at a smaller set of features, removing the ability to use replace or original, but provide an end to end experience that includes the IDE, something that the Uno.SourceGeneration framework is not able to support completely.

This would solve one of the most glaring issue of source generation in Visual Studio, where arbitrary modified files (e.g. non XAML files) can’t re-rerun the generation without an explicit build, and where intellisense caching is getting in the way of using generated file symbols.

The INotifyPropertyChanged case

As Robin Sue mentions, the often requested scenario for generation is INPC, for which tagging a property automatically raising the PropertyChanged event, will not be addressed by the newly proposed version of the feature.

There’s still a way around this, albeit a bit more verbose or at first counter intuitive. Generating source in replace-less model requires to make use of partial classes and methods, and more generally delegate the creation of significant language items to the source generator.

For the case of INPC, it is possible to write a class this way:

partial class MyClass
{
  [GeneratedProperty]
  private string _myProperty;

  partial void OnMyPropertyChanged(string previousValue, string newValue)
  {
    Console.WriteLine($"OnMyPropertyChanged({previousValue}, {newValue}");
  }
}

Notice here that the property is not visible at all, and it will be assumed that the generator will recognize the GeneratedPropertyAttribute and do the rest of the work for the author of the class.

The generator could then produce something like this:

partial class MyClass : INotifyPropertyChanged
{
  public string MyProperty
  {
    get => _myProperty;
    set
    {
      var previous = _myProperty;
      _myProperty = value;
      PropertyChanged?.Invoke(new PropertyChangedEventArgs(nameof(MyProperty)));
      OnMyPropertyChanged(previous, value);
    }
  }

  partial void OnMyPropertyChanged(string previousValue, string newValue);
}

The complete property body is then generated with the appropriate boiler plate. It can also include a OnMyPropertyChanged method that can optionally be implemented as needed in the main class to react to the property changes. Same could also be done for the getter, if there’s work to be done in that context.

In the end, users of the class won’t notice that the properties were generated.

This approach is not without drawbacks, though. For instance, documentation is not generated here, and the generator may need to forward the documentation of the _myProperty field over to the generated property. The same applies to the attributes that may be needed on that property; it’s not possible to apply an attribute on a property defined in another partial type declaration.

Implementing the field backed INPC generator

Let’s try creating this generator using the Uno.SourceGeneration package. Its API is very similar to the current Roslyn proposal (aside from the optional msbuild ties). It could be easily ported over to a future roslyn-based implementation of the generators if it does not change too much.

First, we can assume that there will be an available property named INPC.GeneratedPropertyAttribute, which allows us to find its symbol:

public class INPCGenerator : SourceGenerator
{
    public override void Execute(SourceGeneratorContext context)
    {
        // Search for the GeneratedPropertyAttribute symbol
        var _generatedPropertyAttributeSymbol =
            context.Compilation.GetTypeByMetadataName("INPC.GeneratedPropertyAttribute");
    }
}

This will ease our ability to find fields in types that are tagged with this attribute.

Now let’s find those tagged fields:

// Search in all types defined in the current compilation (but not in the dependents)
var query = 
    from typeSymbol in context.Compilation.SourceModule.GlobalNamespace.GetNamespaceTypes()
    from property in typeSymbol.GetFields()

    // Find the attribute on the field
    let info = property.FindAttributeFlattened(_generatedPropertyAttributeSymbol)
    where info != null

    // Group properties by type
    group property by typeSymbol into g
    select g;

This will create a grouped IEnumerable, for which the Key is the owner of the marked fields, and the group items are the fields symbols.

Then we can generate the class and the properties plumbing:

foreach(var type in query)
{
  var builder = new IndentedStringBuilder();

  builder.AppendLineInvariant("using System;");
  builder.AppendLineInvariant("using System.ComponentModel;");

  using (builder.BlockInvariant($"namespace {type.Key.ContainingNamespace}"))
  {
    // Add the INotifyPropertyChanged interface to the existing type
    using(builder.BlockInvariant($"partial class {type.Key.Name} : INotifyPropertyChanged"))
    {
      // Implement the event itself
      builder.AppendLineInvariant($"public event PropertyChangedEventHandler PropertyChanged;");

      foreach(var fieldInfo in type)
      {
        // Uppercase name for camel case
        var propertyName = fieldInfo.Name.TrimStart('_');
        propertyName = propertyName[0].ToString().ToUpperInvariant() + propertyName.Substring(1);

        // Create the property body
        using (builder.BlockInvariant($"public {fieldInfo.Type} {propertyName}"))
        {
          builder.AppendLineInvariant($"get => {fieldInfo.Name};");

          using (builder.BlockInvariant($"set"))
          {
            builder.AppendLineInvariant($"var previous = {fieldInfo.Name};");
            builder.AppendLineInvariant($"{fieldInfo.Name} = value;");
            builder.AppendLineInvariant(
                $"PropertyChanged?.Invoke(this, new PropertyChangedEventArgs(nameof({propertyName})));");
            builder.AppendLineInvariant($"On{propertyName}Changed(previous, value);");
          }
        }

        builder.AppendLineInvariant(
            $"partial void On{propertyName}Changed({fieldInfo.Type} previous, {fieldInfo.Type} value);");
      }
    }
  }

  var sanitizedName = type.Key.ToDisplayString().Replace(".", "_");
  context.AddCompilationUnit(sanitizedName, builder.ToString());
}

And it’s done! It’s a big string generator that most probably produces valid C#, depending on the efficiency of the type names sanitization (to remove non-valid characters).

This generator makes use of the Uno.Roslyn package, which provides a set of helpers to browser the types and their members more easily, such as GetNamespaceTypes(), FindAttributeFlattened() or GetFields(). It also provides the IndentedStringBuilder class, which allows for a nicely formatted generated files (because developers always want to read generated code 😊).

Finally, here’s how to create a class that uses it :

namespace INPC {
    public class GeneratedPropertyAttribute : Attribute { }
}

public partial class MyClass
{
    [GeneratedProperty]
    private string _stringProperty;

    [GeneratedProperty]
    private int _intProperty;

    partial void OnIntPropertyChanged(int previous, int value)
        => Console.WriteLine($"OnIntPropertyChanged({previous},{value})");
    partial void OnStringPropertyChanged(string previous, string value) 
        => Console.WriteLine($"OnIntPropertyChanged({previous},{value})");
}

And how to use it:

var c = new MyClass();
c.IntProperty = 42;
c.StringProperty = "My 42";

// Output:
//
// OnIntPropertyChanged(0,42)
// OnIntPropertyChanged(,My 42)

Wrapping up

This generator does not handle all possible cases for properties generation, such as nested types or fancy identifier names. That leaves lots to experiment with if you choose source generation to enable this scenario for your project.

You can find a sample solution for this generator on GitHub.

That’s it for now! Until next time, happy source-gen’ing!

MVP 2020-2021

2019-12-01T05:00:00+00:00

What a year!

I’ve been doing full Open Source work for the past year and a half, giving talks about .NET, Xamarin, Mono and WebAssembly, contributing to open source projects, and Microsoft noticed!

I’ve been awarded MVP by Microsoft for 2020-2021 for the Developer Technologies, 8 years after being a “Visual C#” MVP from 2009 to 2011.

I’d like to personally thank the amazing MVPs that supported me through this process, Sébastien Lachance, Jim Wilcox, Geoffrey Huntley, as well as Microsoft employees (Thanks Clint, Karel, Rochelle, Jesus, Ryan, Betsy).

I’m glad to be back as an MVP, and off to a new year with this great community!

NuGet, Visual Studio 2019 Solution Filters and Large Cross-targeted Solutions

2019-06-29T05:00:00+00:00

The combination of the Visual Studio solution filtering and NuGet support for it allows for faster trimmed-down solution loading. This enables developers of large cross-targeted solutions to have a viable environment by temporarily focusing on a single target framework, and still have Visual Studio cooperate.

Cross-targeted projects support in Visual Studio

Since the introduction of the new msbuild project file structure (also known as Common Project System or CPS), it’s been increasingly easy to create libraries that target multiple platforms at once. Oren’s msbuild.sdk.extras definitely helped a lot to make this seamless.

It’s been a delight to use at first, as it replaces linked files and a myriad of framework specific projects, makes the creation of multi-targeted NuGet packages, supports conditional project and package references, and many other features.

But not all is rosy. The impact of this new project format on Visual Studio, and on NuGet is very significant. Each new target framework specified in the <TargetFrameworks> property is actually creating a new independent project in memory, even if large parts of the project definition are the same. For instance, using conditional globbing to include files per target framework can give a hard time to Visual Studio.

The CPU goes up significantly, memory as well (along with longer GC pauses), where it’s very easy to reach the 32 bits process memory limit (and the old 64 bits debate).

Intellisense is also impacted, as for each file present in a target framework, each popup showing types information has to iterate through all of them. NuGet is impacted in the same way.

How did we get here ?

Through quite a natural flow. Xamarin.iOS, Xamarin.Android (per SDK version), Xamarin.Mac, UWP, .NET Core, WebAssembly, .NET Standard, etc… are valid targets for many libraries that provide abstractions over platforms. The Uno Platform is a prime target for that, as each platform gets its own implementation of the UWP API.

This means that to get a valid solution, validate builds for all targets, and provided in-solution sample applications to get a proper debugging experience, it’s required to include all those target frameworks. And there can be many of those, as many upcoming (.NET Standard 2.0, netcoreapp 3.0, uap versions, etc…)

Temporarily reducing the target framework count

In general, when developing a cross-targeted library, it’s very rare to work on multiple targets at once. This means that reducing the TargetFrameworks count during a development session can make sense.

Problem is that when reducing the number of target frameworks from many to xamarinios10, for instance, will make NuGet fail to restore the solution. This happens because some projects (sample apps generally) won’t be able to find valid target frameworks from their dependencies, as they’ve been removed (e.g. an android app references a library that should support monoandroid90, but only supports xamarinios10).

A temporary solution is to make multiple solutions, but this gets out of hands very quickly when new projects get added to the main solution, and trimmed-down solutions don’t.

Combining solution filtering and single target frameworks

Visual Studio added the Solution Filtering feature recently, which gives the ability to create a partially loaded solution for an improved IDE experience. There are two nice touches for this feature, with unloaded projects being hidden by default when a solution is opened from an .slnf file, and also the file format being json, not the exotic .sln format.

From the outside, this seems like just a nice solution to load less projects. It does not allow to reduce the number of loaded target frameworks per project.

But, hidden there, in the NuGet changelog, is the the fact that the NuGet restore operation is now able to take the solution filtering into account, and ignore unloaded projects!

This means that it’s now possible to create a Directory.Build.Props file with this content:

<Project ToolsVersion="15.0">
  <PropertyGroup>
    <!--<TargetFrameworkOverride>netstandard2.0</TargetFrameworkOverride>-->
  </PropertyGroup>
</Project>

And for each project that can be adjusted, the following:

<Project Sdk="MSBuild.Sdk.Extras" ToolsVersion="15.0">
  <PropertyGroup>
    <TargetFrameworks>xamarinmac20;xamarinios10;MonoAndroid90;net461;netstandard2.0</TargetFrameworks>
  </PropertyGroup>

  <PropertyGroup Condition="'$(TargetFrameworkOverride)'!=''">
    <TargetFrameworks>$(TargetFrameworkOverride)</TargetFrameworks>
  </PropertyGroup>
</Project>

When debugging or developing for a specific target framework, un-commenting the TargetFrameworkOverride will essentially change the active target framework, and have Visual Studio only consider that one.

After that, to avoid having NuGet fail for projects that are not valid, create a solution filter file, and make sure to use it when appropriate.

A few things to note:

Changing that property must not be done when the solution is loaded, otherwise project and NuGet caching issues may arise, exposing instabilities in Visual Studio’s project system.
The .slnf format is not supported by CLI msbuild, which makes the use of the TargetFrameworkOverride fail the build, as all the projects of the solution will be tentatively restored, incorrectly.
The TargetFrameworkOverride property should be declared in a file that is excluded from source control, so that it does not unwillingly break a build.

Final thoughts

It still feels like a hack, because cross-targeting is not yet a first class citizen in Visual Studio.

Yet, it definitely does work properly and gives a lot of processing power and time back to the developer, even it is at the expense of being able to view all target frameworks at once.

Using the Span in Xamarin Cross-Targeted projects

2019-03-31T05:00:00+00:00

System.Memory exists to provide the Span<T>, Memory<T> and similar types, for which C# 7.x provided support, and it can enable significant performance improvements where appropriate. While .NET Standard 2.0 does not have support for it in its available API surface, there the System.Memory NuGet package that can be added to enable it.

As part of the code sharing effort, Mono and Xamarin added support for System.Memory in the BCL as part of the releases bundled with VS 2019.

While this is a very important step to make Mono and .NET Core very similar in behavior, this brings issues because the Xamarin.iOS, Xamarin.Mac and MonoAndroid target frameworks are not versioned like .NET Core or .NET Standard. The Visual Studio version used to build a project has an impact on the code’s behavior or compatibility that cannot be adjusted as easily as changing a Target Framework version.

VS2019 issues

In VS2019, the following code :

var s = new Span<int>();

with a project file like this:

<Project Sdk="MSBuild.Sdk.Extras/1.6.46">
  <PropertyGroup>
    <TargetFrameworks>xamarinios10;monoandroid90</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="System.Memory" Version="4.5.2"/>
  </ItemGroup>
</Project>

does not build in Xamarin.iOS10, giving this error:

error CS0433: The type 'Span<T>' exists in both 'System.Memory, Version=4.0.1.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' and 'mscorlib, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'

One easy fix would be to remove the System.Memory package because mscorlib already provides it, but it would prevent this library from building with VS15.9 and below.

We could also only add System.Memory as a normal assembly reference, which would work on both VS versions, but it would break the NuGet package creation and make the consumption of the project’s resulting NuGet package difficult.

Adjusting references during the build

In recent MSBuild, there is a new MSBuildVersion property which tells the currently running version. While this is far from perfect, we can assume that when a given MSBuild version is installed at the same time as Xamarin for an installation of Visual Studio, the MSBuild version is implicitly tied to the installed Xamarin version.

To keep the PackageReference visible for the NuGet packing, but remove the assembly reference added by the package, only in VS16.x, we can inject an MSBuild target:

<Project Sdk="MSBuild.Sdk.Extras/1.6.46">
  <PropertyGroup>
    <TargetFrameworks>xamarinios10</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="System.Memory" Version="4.5.2"/>
  </ItemGroup>
  
  <Target 
    Name="VS16_RemoveSystemMemory" 
    BeforeTargets="FindReferenceAssembliesForReferences" 
    Condition="'$(MSBuildVersion)' &gt;= '16.0'">
    <ItemGroup>
      <_ReferencePathToRemove 
        Include="@(ReferencePath)" 
        Condition="'%(ReferencePath.NuGetPackageId)'=='System.Memory'" />
      <ReferencePath Remove="@(_ReferencePathToRemove)" />
    </ItemGroup>
    <Message Text="Removing System.Memory for VS 2019 compatibility" Importance="high"/>
  </Target>
</Project>

The '$(MSBuildVersion)' >= '16.0' condition ensure that the target only runs when VS 16.0 is used to build the project. Then we remove the ReferencePath item that is generated by the loading of the <PackageReference Include="System.Memory" />.

The target is injected automatically before the FindReferenceAssembliesForReferences that can be found by looking at the generated binary log, and searching where the ReferencePath values are manipulated to be provided to the CSC task.

Interestingly enough, this manipulation does not need to be made in Xamarin.iOS head projects, where the reference seems to be automatically adjusted to be removed by the existing targets.