Microsoft Avro serializer is mangling provided schemas

After a fair amount of searching and poking through poorly-documented source code, I found a solution. Even though the official Avro library from Apache does require a generic type parameter for all of its readers and writers, if you specify the type as GenericRecord, it will let you work with the GenericRecord as a runtime-defined structure. (This is not an arbitrary dynamic type, as the values you assign to its fields still have to match the provided schema.)

Meanwhile, this library's type system has much wider support for Avro's type set than the abandoned Microsoft one does. It correctly works with Avro's logical types and converts between them and CLR types the way you would expect it should work, with one notable exception: serializing a DateTime will convert it from the system's local time to UTC. (There's probably a way to work around this but I haven't found it yet.)

Just leaving this here in case anyone runs into similar problems in the future.

I tried to reproduce the behavior and put together a small program, that doesn't use generics.

Indeed, from what I can see the DateTime sub type is still omitted from the schema, which is really confusing and frustrating, because the receiver needs to know up-front, possible by using the field name ?!, which long fields are of the type DateTime and which not.?! By default it uses Ticks to parse DateTimes. I looked a bit over the Avro library on github and I saw it uses runtimeType for DateTime and DateTimeOffset.

It probably doesn't help you too much, but maybe it helps someone out there with similar problems.

[DataContract]
public struct TestMsg
{
    [DataMember]
    public int Id;
    [DataMember]
    public double Amount;
    [DataMember]
    public DateTime OrderSubmitted;
    public TestMsg(int id, double amount, DateTime orderSubmitted)
    {
        Id = id;
        Amount = amount;
        OrderSubmitted = orderSubmitted;
    }
}


internal class Program
{
    static void Main(string[] args)
    {
        string line = Environment.NewLine;

        string fileName = "Messages.avro";
        string filePath = null;
        if (Environment.NewLine.Contains("\r"))
        {
            filePath = new DirectoryInfo(".") + @"\" + fileName;
        }
        else
        {
            filePath = new DirectoryInfo(".") + @"/" + fileName;
        }

        ArrayList al = new ArrayList
        {
            new TestMsg(1, 189.12, DateTime.Now),
            new TestMsg(2, 345.94, new DateTime(2000, 1, 10, 15, 20, 23, 103))
        };

        var schema = @"
        {
            ""type"" : ""record"",
            ""name"" : ""TestAvro.TestMsg"",
            ""fields"" : [
                {
                    ""name"" : ""Id"",
                    ""type"": ""int""
                },
                {
                    ""name"" : ""Amount"",
                    ""type"":  ""double""
                },
                {
                    ""name"" : ""OrderSubmitted"",
                    ""type"": ""long"",
                    ""runtimeType"": ""DateTime""
                }
            ]
        }";
            
        using (var dataStream = new FileStream(filePath, FileMode.Create))
        {
            var serializer = AvroSerializer.CreateGeneric(schema);
            using (var avroWriter = AvroContainer.CreateGenericWriter(schema, dataStream, Codec.Null))
            {
                
                using (var seqWriter = new SequentialWriter<object>(avroWriter, al.Count))
                {

                    var avroRecords = new List<AvroRecord>();
                    

                    foreach (var item in al)
                    {
                        dynamic record = new AvroRecord(serializer.WriterSchema);
                        record.Id = ((TestMsg)item).Id;
                        record.Amount = ((TestMsg)item).Amount;
                        record.OrderSubmitted = ((TestMsg)item).OrderSubmitted.Ticks;
                        
                        seqWriter.Write(record);
                    }
                }
            }
            dataStream.Dispose();
        }

        Console.WriteLine("Now reading file.");

        using (var dataStream = new FileStream(filePath, FileMode.Open))
        {
            using (var avroReader = AvroContainer.CreateGenericReader(schema, dataStream, true, new CodecFactory()))
            {
                using (var seqReader = new SequentialReader<object>(avroReader))
                {

                    foreach (var dynamicItem in seqReader.Objects)
                    {
                        dynamic obj = (dynamic)dynamicItem;
                        
                        Console.WriteLine($" {obj.Id} - {obj.Amount} - {new DateTime(obj.OrderSubmitted).ToString()}");
                    }
                }
            }
            dataStream.Dispose();
        }

        Console.ReadLine();

    }
}