I want to set a variable inside EMR job and then pass it to next state of step function. I have searched a lot but could not find any way to do this.
The complete use case is to read files from s3, if there is no file in the directory I will get an exception and in except block I will set a value so that on the next state of AWS Step function choice state can take decision, either to move to next Job or to terminate the Cluster and exit.
try:
spark.read.csv("path goes here")
except:
## set parameter here
One solution of this is to use Lambda and check either files exist in the directory and set a value and return, I will receive that value in the choice sate. But I want to do it directly without adding Lambda.
I have also tried generating custom exception and catching them in Catch Block of state function but that is also not working.