I am writing a pyspark application that reads json files (from a relative path) stored in S3.
rdd= spark.read.json("s3://data/19-11-*/Test-*.json", multiLine=True)
And I am getting an exception because folder 19-11-15 contains a file name that contains ":".
java.net.URISyntaxException: Relative path in absolute URI: draft_311819_2019-11-05T17:09:22.812Z.json'
Traceback (most recent call last):
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 274, in json
return self._df(self._jreader.json(self._spark._sc._jvm.PythonUtils.toSeq(path)))
File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
answer, self.gateway_client, self.target_id, self.name)
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)
I understand that spark / hadoop do not accept file names containing ":" but I was expecting Spark to read only files starting with "Test" which is obviously not the case of file name "draft_311819_2019-11-05T17:09:22.812Z.json".
Files starting with draft are generated by another application and I cannot change their names to remove the ":".
When I target the specific folder of 19-11-15, the application works just fine : no error is raised.
rdd= spark.read.json("s3://data/19-11-15/Test-*.json", multiLine=True)
Would you have an idea how to overcome this issue?
Thank you.