SPARK read.json throwing java.io.IOException: Too many bytes before newline

user_2305993 0
Viewed 4155

I am getting following error on reading a large 6gb single line json file:

Job aborted due to stage failure: Task 5 in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 (TID 5, localhost): java.io.IOException: Too many bytes before newline: 2147483648

spark does not read json files with new lines hence the entire 6 gb json file is on a single line:

jf = sqlContext.read.json("jlrn2.json")

configuration:

spark.driver.memory 20g
spark-dataframe bigdata pyspark apache-spark json
2 Answers

I stumbled upon this while reading a huge JSON file in PySpark and getting the same error. So ,if anyone else is also wondering how to save a JSON file in the format that PySpark can read properly, here is a quick example using pandas:

import pandas as pd
from collections import dict

# create some dict you want to dump
list_of_things_to_dump = [1, 2, 3, 4, 5]
dump_dict = defaultdict(list)
for number in list_of_things_to_dump:
    dump_dict["my_number"].append(number)

# save data like this using pandas, will work of the bat with PySpark
output_df = pd.DataFrame.from_dict(dump_dict)
with open('my_fancy_json.json', 'w') as f:
    f.write(output_df.to_json(orient='records', lines=True))

After that loading JSON in PySpark is as easy as:

df = spark.read.json("hdfs:///user/best_user/my_fancy_json.json", schema=schema)
user_49840660
Related