I have some big Spark workloads that can be run with a flexible schedule. I would like to take advantage of the low cost of the Spot instances of the cloud providers. Can I run the Spark jobs in the Iguazio platform with the Spot instances?
1 Answers
Yes, you can run Spark jobs in Iguazio using spot instances. To run the job on the Spot instance, first, you will need to config a new NodeGroup with Spot instances (e.g. added-spot-ng). Then in your job, specify the node selection like the code snippet below:
fn.with_node_selection(node_selector={'app.iguazio.com/node-group': 'added-spot-ng'})
This way, the function (Spark job) will be scheduled to the Spot instance node-group.
Related