Using Spark 3.3.0 with Java I have a DataFrame with a string column foo containing JSON. I use the Spark function get_json_object() to extract an object within the JSON.
df = df.withColumn("foobar", get_json_object(col("foo"), "$.foo.bar"));
Unfortunately some of the column values were corrupted, containing plain, unquoted text such as This value is corrupted. Yet get_json_object() did not complain in any way that I could see; instead it simply ignored the data. I had to manually examine the output to find that some rows were missing the foobar data altogether.
The documentation for get_json_object() says that the input is "A STRING expression containing well formed JSON." Thus shouldn't get_json_object() indicate an error if the input is not well-formed JSON? How can I configure Spark so that get_json_object() provides some feedback that it encountered bad data, instead of simply ignoring it?