How to read a json from BytesIO with PySpark?

user_15509484 0
Viewed 138

I want read a json from IO variable using PySpark.

My code using pandas:

io = BytesIO()
ftp.retrbinary('RETR '+ file_name, io.write)
io.seek(0)
# With pandas
df = pd.read_json(io)

What I tried using PySpark, but don't work:

  io = BytesIO()
  ftp.retrbinary('RETR '+ file_name, io.write)
  io.seek(0)
  df = spark.read\
      .format("binaryfile")\ # I already tried with format "json"
      .option('inferSchema', 'true')\
      .option('header', 'true')\
      .json(io)

Obs: Is not possible save in a file and read that after

pyspark pandas apache-spark
1 Answers

Transform io variable to json format and create a dataframe:

Bellow code:

io = BytesIO()
ftp.retrbinary('RETR '+ file_name, io.write)
io.seek(0)
data_json = io.json()
df = spark.createDataFrame(io, jsonSchema)
user_155094840
Related