We use AWS Glue Jobs for some of our data processing. We use pyspark to process the data, but from time to time we see this error on some step of a job:
An error occurred while calling o82.parquet. Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: ABC111; S3 Extended Request ID: ABC111abc111)
This error seems to be intermittent and sometimes just rerunning the same job with same parameters seems to run fine, but it is definitely not a very descriptive error and we'd like to avoid it as our number of automated jobs grows.
In the Cloudwatch logs the latest logs I see:
WARN [Executor task launch worker for task 1318] client.YarnClient (YarnClient.java:makeRestApiRequest(66)) - The GET request failed for the URL http://0.0.0.0:8088/ws/v1/cluster/apps/application_1583197528647_0001
om.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.conn.HttpHostConnectException: Connect to 0.0.0.0:8088 [/0.0.0.0] failed: Connection refused (Connection refused)
at com.amazon.ws.emr.hadoop.fs.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:158)
...
Caused by: java.net.ConnectException: Connection refused (Connection refused)
...
ERROR [SIGTERM handler] executor.CoarseGrainedExecutorBackend (SignalUtils.scala:apply$mcZ$sp(43)) - RECEIVED SIGNAL TERM
The overview of the job: reading json files using Glue Data Catalog, writes aggregated data into s3 in parquet format (I see a new partition here, but pretty sure it fails at this step as I don't see any messages that I put into the code after this); reads the data from the last step, reads csv mapping file from s3, joins 2 datasets, doing some additional calculations using pyspark, finally writes output to s3 in csv format.