edrodven / dotnet-avro

An Avro implementation for .NET

Chr.Avro

Chr.Avro is an Avro implementation for .NET. It’s designed to serve as a flexible alternative to the Apache implementation and integrate seamlessly with Confluent’s Kafka and Schema Registry clients.

• Chr.Avro: schema models, type resolution, and schema builder
• Chr.Avro.Binary: binary serialization implementation
• Chr.Avro.Cli: command line interface
• Chr.Avro.Codegen: experimental C# code generation
• Chr.Avro.Confluent: serializers/deserializers and glue
• Chr.Avro.Json: JSON schema representation

For more information, check out the documentation.

Quick start

To use the command line interface: Install Chr.Avro.Cli as a global tool:

$ dotnet tool install Chr.Avro.Cli --global
You can invoke the tool using the following command: dotnet-avro
Tool 'chr.avro.cli' (version '9.4.0') was successfully installed.
$ dotnet avro help
Chr.Avro 9.4.0
...

To use the Kafka producer/consumer builders in your project: Add Chr.Avro.Confluent as a project dependency. After that, check out this guide or read on for some other examples.

Examples

The CLI can be used to generate Avro schemas for .NET types (both built-in and from compiled assemblies):

$ dotnet avro create -t System.Int32
"int"
$ dotnet avro create -t System.Decimal
{"type":"bytes","logicalType":"decimal","precision":29,"scale":14}
$ dotnet avro create -a out/example.dll -t ExampleRecord
{"name":"ExampleRecord","type":"record","fields":[{"name":"Number","type":"long"}]}

It can also verify that a .NET type can be mapped to a Schema Registry schema (useful for both development and CI):

$ dotnet avro registry-test -a out/example.dll -t ExampleRecord -r http://registry:8081 -i 242
A deserializer cannot be created for ExampleRecord: ExampleRecord does not have a field or property that matches the correlation_id field on example_record.

Extensions to the Confluent.Kafka ProducerBuilder and ConsumerBuilder configure Kafka clients to produce and consume Avro-encoded CLR objects:

using Chr.Avro.Confluent;
using Confluent.Kafka;
using Confluent.SchemaRegistry;
using System;
using System.Collections.Generic;

namespace Example
{
    class ExampleRecord
    {
        public Guid CorrelationId { get; set; }
        public DateTime Timestamp { get; set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var consumerConfig = new ConsumerConfig()
            {
                BootstrapServers = "broker1:9092,broker2:9092",
                GroupId = "example_consumer_group"
            };

            var registryConfig = new SchemaRegistryConfig()
            {
                SchemaRegistryUrl = "http://registry:8081"
            };

            var builder = new ConsumerBuilder<string, ExampleRecord>(consumerConfig);

            using (var registry = new CachedSchemaRegistryClient(registryConfig))
            {
                builder.SetAvroKeyDeserializer(registry);
                builder.SetAvroValueDeserializer(registry);

                using (var consumer = builder.Build())
                {
                    var result = consumer.Consume(CancellationToken.None);
                    Console.WriteLine($"Consumed message! {result.Key}: {result.Value.Timestamp}");
                }
            }
        }
    }
}

Under the hood, SchemaBuilder is responsible for generating schemas from CLR types:

using Chr.Avro.Abstract;
using Chr.Avro.Representation;
using System;

namespace Example
{
    enum Fear
    {
        Bears,
        Children,
        Haskell,
    }

    struct FullName
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
    }

    class Person
    {
        public Guid Id { get; set; }
        public Fear GreatestFear { get; set; }
        public FullName Name { get; set; }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var builder = new SchemaBuilder();
            var writer = new JsonSchemaWriter();

            Console.WriteLine(writer.Write(builder.BuildSchema<double>));
            // "double"

            Console.WriteLine(writer.Write(builder.BuildSchema<DateTime>));
            // "string"

            Console.WriteLine(writer.Write(builder.BuildSchema<Fear>));
            // {"name":"Fear","type":"enum","symbols":["Bears","Children","Haskell"]}

            Console.WriteLine(writer.Write(builder.BuildSchema<Person>));
            // {"name":"Person","type":"record"...}
        }
    }
}

For more complex examples, see the examples directory:

• Default field values
• Discriminated union types

Contributing

Check out the contribution guidelines prior to opening an issue or creating a pull request. More information about the benchmark applications and documentation site can be found in their respective directories.

Cake handles all tasks related to building and publishing the Chr.Avro libraries and CLI. This repository doesn’t include bootstrapper scripts; installing and running Cake as a global tool is recommended:

$ dotnet tool install Cake.Tool --global
You can invoke the tool using the following command: dotnet-cake
Tool 'cake.tool' (version '0.35.0') was successfully installed.

$ dotnet cake build.cake --target=Pack
...

The following targets are supported:

Name	Description
Analyze	runs mdoc on the library projects and writes the results to docs/api
Benchmark	runs the benchmark applications and writes the results to docs/benchmarks
Build	builds all projects
Clean	removes all build, documentation, and release artifacts
Pack	creates NuGet packages for the library projects
Publish	pushes packages to NuGet
Test	runs the test projects