I would like to understand how the receiver works in Spark Streaming. As per my understanding there will be a receiver tasks running in executors that collect data and saves as RDD's. Receivers start reading when the start() is called. Need clarification on the following.
- How many receivers does the Spark Streaming job starts? Multiple or One?
- Is the receiver is implemented as push based or pull based?
- In any case does the receiver can become a bottleneck?
- To achieve the degree of parallelism the data should be partitioned across the worker nodes. So for the streaming data the how the data is distributed across the nodes.
- If a new RDD's are formed on an new node based on batch time interval, how does SparkContext serialize the transform functions to the node after the Job is submitted?
- Can the amount of receivers launch be controlled by a parameter?
Would like to know the anatomy of Spark Streaming and receiver.