I a dataframe lke this
data_frame = spark.CreateDataFrame(
tmp = ["[{\"Julia is awesome δ; or \": 1, \"ok2\": 2}, {\"ok\":1 , \"ok3\": 3}]"])
so it looks like this
+--------------------------------------------------------------------+
|tmp |
+--------------------------------------------------------------------+
|[{"Julia is awesome δ; or ": 1, "ok2": 2}, {"ok":1 , "ok3": 3}]|
|[{"ok": 1, "ok2": 2}, {"ok":1 , "ok3": 3}, {"ok2":3}] |
+--------------------------------------------------------------------+
As you can see there are unicode?
I want to read it as JSON array and then do some further processing
from pyspark.sql import functions as F
data_frame.withColumn("tmp1", F.from_json(F.col("tmp"), "array<string>").\
withColumn("tmp2", F.col("tmp1").getItem(0))
But I get
{"Julia is awesome δ; \uD83D\uDC14 or \uD83E\uDD5A":1,...
as return so the unicodes are gone. So I tried to use
F.decode() like so
data_frame.withColumn("tmp1", F.from_json(F.col("tmp"), "array<string>").\
withColumn("tmp2", F.decode(F.col("tmp1").getItem(0), "utf-8")
``
but it gives the same output.
The issue is that
data_frame.withColumn("tmp1", F.from_json(F.col("tmp"), "array").select("tmp1").collect()
already show the unicdes are messed up but there is no "use_unicode=True" optin in `from_json`.
So how do I handle unicode correctly with `from_json`?
The documentation here is not very clear: http://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.functions.from_json