I wrote a python script that will do a bulk unload of all tables within a schema to s3, which scales to petabytes of data. While my script was running perfectly okay, my python script got interrupted due to a network disconnection.
Now, I'm in the midst of an unload job, unsure of how I can resume from the last point of failure. While I am debating about rerunning everything from the start, I am obviously thinking of using a Jenkins slave to run my job, however, I don't want to lose the hours of unload that had already completed. Is there a way to resume from where it stopped? My thinking is that it is hard to figure out the point to resume when the files are coming in as a ZIP.
What are the best practices I could use to avoid this in the future?
Any strategies to be able to pick up from where it left off?
Part of my code snippet that does unload:
#function to unload the data from tables into s3
def unloadData(passed_tables,schema,cur):
#loop through each table from the specified schema
for table in passed_tables:
#extract the table name string and store it in a variable
i=(table[0])
try:
#unload query to migrate the table to s3
unload='''unload('select * from {0}.{1}') to 's3://<bucket>/{2}/{3}/'
iam_role 'arn:aws:iam::*****:role/***';'''.format(schema,i,schema,i)
cur.execute(unload)
print("Unload in progress! Check S3 bucket in a while to confirm.")
except Exception, e:
print("Failed to unload data! Try again or check the query.")
print(sys.stderr, "Exception: %s" % str(e))
sys.exit(1)