I am trying to read 1TB of parquet data from s3 into spark dataframes and have assigned 80 executors with 30 gb and 5 cores to process and ETL the data.
However i am seeing the data is not distributed equally among the executors to make use of the cores while reading the data. My understanding is that the input is divided into chunks and then distributed equally among the executors for processing . I am not using any shuffles or joins of any kind and also the explain plan does not have any hash partitioning or aggregations of any kind . Please suggest if this is expected and how we can better redistribute the data to make use of all the cores.
