Elasticsearch.NET (NEST) fails to deserialize my POCO JSON in version 7.x

PureKrome_30674 0
Viewed 4282

Elasticsearch 7.x for .NET has a documented breaking change with the internal Utf8 JSON serializer.

Instead of me using the another option -> Newtonsoft's Json.NET serializer, I'm trying to figure out which part of my JSON is breaking the serializer.

  • Client nuget: 7.3.1
  • Server: 7.2.0

Firstly the error feels like it's helpful, but I don't think it really is:

2019-10-18 17:15:33 ERR Fatal error.
Elasticsearch.Net.UnexpectedElasticsearchClientException: expected:'String Begin Token', actual:'2', at offset:540 ---> Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'String Begin Token', actual:'2', at offset:540
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentCore(Byte[]& resultBytes, Int32& resultOffset, Int32& resultLength)
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadString()
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
   at Nest.SourceFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase`4.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Nest.ReadAsFormatter`2.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)

... etc ...

So at character 540, something is bad.

So I think I have the json which came back. I think because I asked a collegue to use Fiddler and give me the payload result. I can't paste the full payload anywhere because it contains sensitive stuff but this is the part of code at line 540

enter image description here

There's no '2' there.

So what about the Json? Here's a nicely formatted screenshot of the json, with the custom results hidden:

enter image description here

and finally, the poco section, for the section I think is erroring (but I'm not sure):

        [Keyword(Index = false)]
        public string Street { get; set; }

        [Keyword]
        public string Suburb { get; set; }

        [Keyword]
        public string State { get; set; }

        [Keyword]
        public string StateCode { get; set; }

        [Keyword]
        public string Country { get; set; }

        [Keyword]
        public string Postcode { get; set; }

        [GeoPoint]
        public GeoLocation LatLong { get; set; }

So:

  • I'm not sure how to grab the exact json which is parsed by NEST. Is it the json I grabbed, which can be confirmed by the snippet I provided?
  • Is there a way I can manually create a unit test + use the internal Utf8 serializer with NEST to load my json if I save this JSON payload to a file in order to debug this further?
elasticsearch serialization json
1 Answers

Issue is with enums. Out ES DB has some small subset of documents with a number/int value while our POCO has a string value. The string value is the string representation of our enums, while the int value was the previous int representation of the enums.

public class Foo
{
    public string statusType { get; set }
}

To figure out what the error is/was, I needed to do the following:

  • Clone ES repo and put a breakpoint in ES's JsonSerializer Deserialize<T> method because SourceLink is not enabled on the ES nuget packages (as of the time of this posting).
  • Update the method to convert the byte array to a string, so I can see what the JSON payload is, before it tries to get deserialized into our POCO.
  • Step through the code, to finally hit the breakpoint and see what the json payload is.

So the error message is in respect to the json segment of my document. Not the entire json ES payload. In the above examplei have 5 'hits' (document segments) so the Deserialize was getting hit 1x per document segment. When it hit the segment with the int values (which should have been string value) it then crashed. So I can now see what the error at char 540 is and viola....

  "statusType": 2,

which is bad. is should be:

  "statusType": "available",

So - problem identified.

PureKrome_306740
Related