Suppose I read data from RDS and write it into S3 using EMR cluster (Spark), should I use Task nodes only?
Example:
* 1 Master node
* 4 Task nodes
In my case I don't use HDFS to store data, so that using Core node isn't necessary, if I understand it right. Or should I have at least one Core node in any way? Any ideas?
To my knowledge, You should have at least one core node.
I also had a similar use case a very long time back, where I had used Spark-SQL to read data from S3 and insert it into RDS (opposite of your use case, but that does not matter in any way).
Since the nature of this job was not heavy, I had used only the Master node and Core node. I did not use any Task node, since I did not find the need of using it for a small job.
I think it is a little misunderstanding that Only when HDFS is used, we should look towards usage of the Core node. The way I see it is at the end of the day, even core node is an instance where I can run an application.
So even the Core node can perform the job of task/worker node and I have seen multiple examples where the core node is of high instance type(say r5.24xlarge) and even on this instance your executors would be running.
In my above example, all the tasks were performed on the core node itself since I did not have any task node.
In my experience, I have seen a lot of EMR which has only Master node and Core node. Have not seen anything with just the Master node and task node.
One key point which I want to share is Please use at least one on-demand instance in the core node. You can have a fleet of instances in the core node(composed of both on-demand and spot), but having at least one on-demand node is highly advisable.
More readings can be found here:
So the moral of the story is:
should I have at least one Core node in any way?
Yes, You should in my opinion.