Sometimes when I want to run an ETL Job in AWS Glue, it is triggered immediately. But quite often it happens to me that it takes few minutes before ETL Job is doing anything - I see nothing in logs, only "pending execution". Is there any way I can influence it in configuration? Or is it totally up to AWS when it will start the job?
1 Answers
I know this is an old thread, but I recently experienced the same symptoms. You'll have to confirm if the cause is/was the same. The job I originally created was to move data from my data lake in to a redshift db. When the job was created it had the redshift connection as a required resource, but it seems that glue was not able to acquire that resource prior to starting the job so the job would fail before it started every time... no logs.
I removed the connection as a requirement for the job by editing the job from the console. (Action->Edit job-> Required Connections->"X")
After I removed the requirement, the job ran just fine. It took me a long time to figure that one out. Hopefully it helps anyone else to runs into the same scenario.
Related