Can I tell spark.read.json that my files are gzipped?

user_909227 0
Viewed 14750

I have an s3 bucket with nearly 100k gzipped JSON files.

These files are called [timestamp].json instead of the more sensible [timestamp].json.gz.

I have other processes that use them so renaming is not an option and copying them is even less ideal.

I am using spark.read.json([pattern]) to read these files. If I rename the filename to contain the .gz this works fine, but whilst the extension is just .json they cannot be read.

Is there any way I can tell spark that these files are gzipped?

pyspark apache-spark
1 Answers

SparkSession can read compressed json file directly, just like this:

val json=spark.read.json("/user/the_file_path/the_json_file.log.gz") json.printSchema()

user_44314900
Related