Getting the above mentioned WARN message, and performance issue, while using the following SQL Spark transformation
val sqlstr = """SELECT * FROM (
SELECT AC.*, ROW_NUMBER OVER (PARTITION BY ACCOUNT_TYPE, CATEGORY_CODE, PROCESS_CAT ORDER BY GEO_LOC DESC) AS RowNum FROM Account AC)
WHERE RowNum = 1"""
val acct_sel_df = sqlContext.sql(sqlstr)
Running Spark 2.1, Scala and spark sql.
The above statement will result in the data frame acct_sel_df. This
is transformed from the Account dataframe (which was registered as
temp view using Account.createOrReplaceTempView("Account").
Is there a way to specify partition in the source or resulting data frame? Will that avoid moving data to a single partition?
The number of partitions could be found in advance by querying the source data frame. Without this there is a serious performance hit.