I have R code in the Databricks that loads "huge" csv file (approx. 550 MB) and after that it does make some transformations, but let's focus only on the loading part.
My code looks like that:
############################################
## INSTALL LIBRARIES
############################################
library(tidyr)
library(ggplot2)
library(dplyr)
library(lubridate)
library(data.table)
library(hexbin)
library(RColorBrewer)
library(Boruta)
library(stringr)
library(SparkR)
sparkR.session()
############################################
## DEFINE CURRENT DATE
############################################
currentDate <- format(Sys.time(), "%Y/%m/%d")
############################################
## FILES PATHS
############################################
pathStart <- Sys.getenv("PADLE_DataLake")
assignmentsPath <- dbutils.fs.ls(sprintf('%s/RAW/XXX/ALL_ASSIGNMENT_REPORT_EXTRACT/%s', pathStart, currentDate))[1][[1]]$path
############################################
## READ FILES
############################################
WD_PROD_Assignments = collect(read.df(assignmentsPath, source = "csv", header = TRUE, encoding = 'UTF-8', multiLine = TRUE))
I am using dbutils because sometime our process in the background creates two files instead of one, I just want to load one of them, data is laying on the ADLS, that's why I don't use fread.
With "fresh" cluster it takes approx. 1.8 minutes to run, on the second run it takes about 8-10 minutes, why is that?
Cluster details:
- 72 GB
- 16 Cores
- DBR 9.1 LTS
- Spark 3.1.2
- Scala 2.12
- Driver: Standard_F8
- Worker: Standard_DS12_v2
- Min workers: 2, Max workers: 5