I have a JSON string column in a Spark DF and am trying to use the schema_of_json function to extract the schema into a different column:
df.select(F.schema_of_json(F.col("kvp_full")).alias('kvp_schema')).show()
This is throwing the following error:
org.apache.spark.sql.AnalysisException: cannot resolve 'schemaofjson(`kvp_full`)' due to data type mismatch: The input json should be a string literal and not null; however, got `kvp_full`.;;
I can verify that the column is of type string, and has no Null values. Browsing around online, there seems to be a hacky work around of taking the first row of the JSON string column, but unfortunately each of my JSON strings has it's own unique schema.