pyspark in AWS Glue skip bad file

user_8520779 0
Viewed 622

I am using pyspark in AWS Glue to read ETL 100K S3 files, however, I don't have permissions to read tens of files.

I used following code:

datasource0 = glueContext.create_dynamic_frame_from_options("s3",
    {'paths': ["s3://mykkkk-test"],
    'recurse':True,
    'groupFiles': 'inPartition',
    'groupSize': '10485760'}, 
    format="json",
    transformation_ctx = "datasource0")
## @type: toDF
## @args: []
## @return: df
## @inputs: [frame = datasource0]
df = datasource0.toDF()

It says

An error occurred while calling o70.toDF. java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
...
Caused by: java.io.FileNotFoundException: No such file or directory
s3://mykkkk-test/1111/2222/3333.json

I don't have permission to read 3333.json then the entire job stopped.

Is there a way to catch the exception and skip files, and let the script continue handle other files?

aws-glue pyspark
1 Answers

No you cannot... This is becuase spark assumes that i can access all the data files in the folder you have mentioned as the source. Your best option would be to identify before hand the list of files you have access to, move them to a different folder and then read the data from there.

Or try and get the list of files you have access to and then ready each file individually in a loop

user_129503400
Related