I'm using sparklyr library in order to calculate some metrics on big data. I've decide to test my code on a small portion of data (parquet-table format ~ 1,3m rows, 40MB), but after executing some basic filtering SPARK needs approximately 2-3 minutes to calculate sum.
Here is an example of my code
#Connecting
df <- spark_read_parquet(spark_conn, name = "df", path = "hdfs://****.db/table", header = TRUE, memory = FALSE, repartition = 1, overwrite = FALSE)
#Basic filtering
df <- df %>%
mutate (ZD = datediff(Maturity_Date - Treaty_Date))
df <- df %>%
filter(Treaty_Amount <= 30000, ZD <= 30)
df <- df %>%
dplyr::mutate(MD_Received_From_Borrower_2018_1 = rowSums(.[139:150], na.rm = TRUE))
df <- df %>%
dplyr::mutate(PD_Received_From_Borrower_2018_1 = rowSums(.[175:186], na.rm = TRUE))
df <- df %>%
mutate (Total_Amount_Received_From_Borrower_2018_1= (MD_Received_From_Borrower_2018_1 + PD_Received_From_Borrower_2018_1))
#More filtering for a project purposes
df <- df %>%
mutate(Preditog1_2018_01 = ifelse((Debt_Duration_As_Of_31.01.2018 >= 361 & Debt_Duration_As_Of_31.01.2019 == 0 & Total_Amount_Received_From_Borrower_2018_1 == 0), 361, 0))
df <- df %>%
mutate(Preditog2_2018_01 = ifelse((Date_Of_Debt_Sale <= "2019-01-31" & !is.na(Date_Of_Debt_Sale)), 361, 0))
df <- df %>%
mutate(Preditog3_2018_01 = ifelse((Date_Of_Debt_Cancellation <= "2019-01-31" & !is.na(Date_Of_Debt_Cancellation )), 361, 0))
df <- df %>%
mutate(Preditog_2018_01 = ifelse((Preditog1_2018_01 == 361 | Preditog2_2018_01 == 361 | Preditog3_2018_01 == 361), 361, 0))
#df <- df %>%
mutate(Itog_2018_01 = ifelse(Preditog_2018_01 == 361, 361, Debt_Duration_As_Of_31.01.2019))
#Calculations (Where the process is extremely slow especially when calculating sum)
##Extra filtering
df <- df %>%
mutate(f_0_v_360_2018_01_d = ifelse((Debt_Duration_As_Of_31.01.2018 == 0 & Itog_2018_01 > 360), Main_Debt_As_Of_31.01.2018, 0))
##Calculating sum
df %>%
summarise(res1 = sum(f_0_v_360_2018_01_d)
Maybe I'm doing smth wrong? How is it possible to optimize calculation speed?
Thank you in advance for any help!