How do i trigger a cloud-run after a dataproc job finishes

Viewed 95

I am running a dataproc pyspark job to split files in a gcs bucket into multiple shards .I then use cloud run to load the files and send them with an api call to a server.for all the files to be sent through cloud run it takes about 45 min.now i need to somhow trigger cloud run preferably by a post request sending the location of the file shards.One option is to send an http post request to cloud run from the pyspark job but it would take an extra 45 min for dataproc clusters to keep running waiting for a response from cloud run which is alot of cost.I've thought about pub/sub but it needs to recieve an ack within a 600 sec range an i cant afford that with the cloud run job i am running.Is there a way to somehow trigger the cloudrun job without waiting for a response ?

0 Answers
Related