PySpark: How to make `from_json` work with unicode?

Viewed 374

I a dataframe lke this

data_frame = spark.CreateDataFrame(
  tmp = ["[{\"Julia is awesome δ;  or \": 1, \"ok2\": 2}, {\"ok\":1 , \"ok3\": 3}]"])

so it looks like this

+--------------------------------------------------------------------+
|tmp                                                                 |
+--------------------------------------------------------------------+
|[{"Julia is awesome δ;  or ": 1, "ok2": 2}, {"ok":1 , "ok3": 3}]|
|[{"ok": 1, "ok2": 2}, {"ok":1 , "ok3": 3}, {"ok2":3}]               |
+--------------------------------------------------------------------+

As you can see there are unicode?

I want to read it as JSON array and then do some further processing

from pyspark.sql import functions as F

data_frame.withColumn("tmp1", F.from_json(F.col("tmp"), "array<string>").\
  withColumn("tmp2", F.col("tmp1").getItem(0))

But I get

{"Julia is awesome δ; \uD83D\uDC14 or \uD83E\uDD5A":1,...

as return so the unicodes are gone. So I tried to use

F.decode() like so

data_frame.withColumn("tmp1", F.from_json(F.col("tmp"), "array<string>").\
  withColumn("tmp2", F.decode(F.col("tmp1").getItem(0), "utf-8")
``

but it gives the same output.

The issue is that 

data_frame.withColumn("tmp1", F.from_json(F.col("tmp"), "array").select("tmp1").collect()


already show the unicdes are messed up but there is no "use_unicode=True" optin in `from_json`.

So how do I handle unicode correctly with `from_json`?

The documentation here is not very clear: http://spark.apache.org/docs/3.0.1/api/python/pyspark.sql.html#pyspark.sql.functions.from_json
0 Answers
Related