TL;DR
I am trying to (de-)serialize BSON data from/to Julia/C#, some libraries know how to write and read this in each language, but I can't figure how to translate things between the two implementations.
It feels like a Json.net JSON Converter should be used but I do not see how to configure it without rewriting a converter for every single type.
I do not expect a complete answer, but helpful pointers will be gratefully appreciated !
Addition : Here is a somewhat close topic Json.NET Custom JsonConverter with data types.
Context
Project
I am currently working on a communication between a computation kernel using julia and a .net program that serves as a graphical interface. To be able to share quite complex datasets between the two programs, I looked for a data format that
- is supported in both languages,
- is flexible enough to handle heavy data (such as ND arrays of complex number)
After trying MsgPack (that unfortunately does not work for ND arrays yet, ouch !), I am now testing BSON as a serialization format.
My goal is to be able to handle multiple custom types, without rewriting everything every time.
Available libraries
In julia, there is a BSON.jl package which writes and read pretty much everything, so let's assume it can handle anything I throw at him.
In C#, which is the main object of this question, there is the seemingly obvious choice of JSON.net BSON serializer.
Problem
Despite the available code, the julia serializer writes information that of course Json.net cannot serialize to or deserialize from natively.
So after reading quite a lot of posts here and there, it appears a custom JsonConverter<T> is required to achieve my goal.
A lot of posts seems to be already available to do this but :
- I've never done something like this in C# (or any other language for that matter)
- I am not sure where to start from to avoid unnecessary code repetition
Data sample
To server as an example, here is a JSON conversion of a BSON file generated with the julia package. The conversion was performed with https://json-bson-converter.appspot.com/
{
"dict": {
"tag": "struct",
"type": {
"tag": "datatype",
"params": [
{
"tag": "datatype",
"params": [],
"name": [
"Core",
"Any"
]
},
{
"tag": "datatype",
"params": [],
"name": [
"Core",
"Any"
]
}
],
"name": [
"Main",
"Base",
"Dict"
]
},
"data": [
[
"string",
"arr",
"vec",
"cplxvec",
"scal"
],
[
"blablabla",
{
"tag": "array",
"type": {
"tag": "datatype",
"params": [],
"name": [
"Core",
"Int8"
]
},
"size": [
2,
3
],
"data": "683oIbTI"
},
{
"tag": "array",
"type": {
"tag": "datatype",
"params": [],
"name": [
"Core",
"Int8"
]
},
"size": [
4
],
"data": "KxgCgA=="
},
{
"tag": "array",
"type": {
"tag": "datatype",
"params": [
{
"tag": "datatype",
"params": [],
"name": [
"Core",
"Float64"
]
}
],
"name": [
"Main",
"Base",
"Complex"
]
},
"size": [
3
],
"data": "bODExAuj5D9SoKEy0NDqP4jU3buU8+k/tFNpqcmh7z+miGD4H87oPyKwK7BUEuo/"
},
33.6
]
]
}
}
As far as I can analyze the contents of this file, I see :
- A scalar type is converted with a somewhat native format
- Other types are converted to dictionary-like structures with the following properties
- a
tagfield composed of a singlestring - a
typefield which has a specific format depending on the contents oftag - a
datafield encoded in a binary format - an optional
sizefield (once again based on the contents oftag
- a
- A
typefield is characterized by the following properties- a
tagas before - a
paramsfield that may be- empty (if the element type is a primitive type)
- a list of types (otherwise)
- a
namefield that lists type names from julia
- a
Custom converter
I know that to make a custom Converter, an starting point is the following, based on e.g. here :
public class JuliaObjectSerializer : JsonConverter
{
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
throw new NotImplementedException();
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
throw new NotImplementedException();
}
public override bool CanConvert(Type objectType)
{
throw new NotImplementedException();
}
}
However, I really do not know what to define there to be generic and versatile enough to read all the information from my source file. My first thought was to define some structures to receive the information, such as
public static class JuliaComm
{
public struct JuliaObject
{
public string tag { get; set; }
public JuliaType type { get; set; }
public long size { get; set; }
public sbyte data { get; set; }
}
public struct JuliaType
{
public string tag { get; set; }
public List<JuliaType> param { get; set; }
public List<string> name { get; set; }
public JuliaType(string tag, List<JuliaType> param, List<string> name)
{
this.tag = tag;
this.param = param;
this.name = name;
}
}
}
internal sealed class JuliaObjectConverter : JsonConverter
{
public override bool CanConvert(Type objectType)
{
var type = objectType.GetElementType();
var isarr = objectType.IsArray;
return isarr && (type == typeof(Array) || type == typeof(double) || type == typeof(Complex) || type == typeof(long));
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
throw new NotImplementedException();
if (reader.TokenType == JsonToken.StartObject)
{
JObject item = JObject.Load(reader);
if (item["tag"] != null)
{
var tag = item["tag"].ToString();
if (tag == "struct")
{
var type = item["type"].ToObject<JuliaComm.JuliaType>();
var data = item["data"]; // to something else ?
}
else if (tag == "array")
{
var type = item["type"].ToObject<JuliaComm.JuliaType>();
var size = item["size"].ToObject<int[]>();
var arr = Array.CreateInstance(type, size);
if (type["param"] == [])
{
// fill array with byte to type converter ?;
}
}
}
}
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
if (value.GetType().IsClass)
{
writer.WriteStartObject();
writer.WritePropertyName(value.ToString()); // should be the name of the variable
writer.WritePropertyName("tag");
writer.WriteValue("struct");
writer.WritePropertyName("type");
// serializer.Serialize(writer,); // should be a List<JuliaType> converion of all types in class
writer.WriteValue("data");
// serializer.Serialize(writer,); // should be a something ! not clear yet
writer.WriteEndObject();
}
else if (value.GetType().IsArray)
{
}
}
}
internal sealed class JuliaTypeConverter : JsonConverter
{
public override bool CanConvert(Type objectType)
{
return (objectType == typeof(JuliaComm.JuliaType));
}
public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
{
JObject jsonObject = JObject.Load(reader);
var tag = "datatype";
var type = jsonObject["type"].ToObject<JuliaComm.JuliaType>();
var param = jsonObject["params"].ToObject<List<JuliaComm.JuliaType>>();
var name = jsonObject["name"].ToObject<List<string>>();
return new JuliaComm.JuliaType(tag, param, name);
}
public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
{
throw new NotImplementedException();
var jtype = value as JuliaComm.JuliaType;
writer.WriteStartObject();
writer.WritePropertyName("tag");
serializer.Serialize(writer,"datatype");
writer.WritePropertyName("params");
serializer.Serialize(writer, jtype.param);
writer.WritePropertyName("name");
serializer.Serialize(writer, jtype.name);
writer.WriteEndObject();
}
}
but it might be totally inappropriate (I get static errors).
I know all the primitive type conversions between julia and C#, so using the information in the file to infer the C# types won't be an issue.
Sorry for the long post, I just wanted to be as clear as possible, though I surely forgot something.