Pyspark read json : java.net.URISyntaxException: Relative path in absolute URI

Viewed 801

I am writing a pyspark application that reads json files (from a relative path) stored in S3.

rdd= spark.read.json("s3://data/19-11-*/Test-*.json", multiLine=True)

And I am getting an exception because folder 19-11-15 contains a file name that contains ":".

java.net.URISyntaxException: Relative path in absolute URI: draft_311819_2019-11-05T17:09:22.812Z.json'
Traceback (most recent call last):
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 274, in json
    return self._df(self._jreader.json(self._spark._sc._jvm.PythonUtils.toSeq(path)))
  File "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py", line 1257, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
    raise IllegalArgumentException(s.split(': ', 1)[1], stackTrace)

I understand that spark / hadoop do not accept file names containing ":" but I was expecting Spark to read only files starting with "Test" which is obviously not the case of file name "draft_311819_2019-11-05T17:09:22.812Z.json".

Files starting with draft are generated by another application and I cannot change their names to remove the ":".

When I target the specific folder of 19-11-15, the application works just fine : no error is raised.

rdd= spark.read.json("s3://data/19-11-15/Test-*.json", multiLine=True)

Would you have an idea how to overcome this issue?

Thank you.

0 Answers
Related