I'm having an issue with spark executing jobs, that is, between a filter and the spark_write_csv function , the elapsed time is around 14 minutes. I have already reinstalled java, spark and hadoop, all with no effect. Besides, recently I had to reinstall my current OS(windows) on a ssd, it seems to me that after that this "error" appeared. My code is structured as follows.
conf <- spark_config()
conf$spark.serializer <- "org.apache.spark.serializer.KryoSerializer"
conf$`sparklyr.shell.driver-memory` <- "20G"
conf$spark.memory.fraction <- 0.6
conf$spark.dynamicAllocation.enabled <-'false'
sc <- spark_connect(master = "local",config = conf)
df <- spark_read_csv(sc = sc,name='mark1',
path=path_to_file,
columns = columns_name,
infer_schema = FALSE,
delimiter = ";",header = TRUE,
overwrite = TRUE,memory=T,repartition=5)
df %>% dplyr::filter(ids %in% id_list) %>%
dplyr::select(columns_to_select) %>%
spark_write_csv(x=.,path=partions_save_folder,mode='overwrite')
The id_list is a list containing 1.8M ids, and the file I'm looking into is small, 10GB.
My spark version is 3.1.1 and hadoop 3.2
And my enviroment is:
R version 4.0.5 (2021-03-31)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale:
[1] LC_COLLATE=Portuguese_Brazil.1252 LC_CTYPE=Portuguese_Brazil.1252 LC_MONETARY=Portuguese_Brazil.1252 LC_NUMERIC=C
[5] LC_TIME=Portuguese_Brazil.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] forcats_0.5.1 stringr_1.4.0 dplyr_1.0.6 purrr_0.3.4 readr_1.4.0 tidyr_1.1.3 tibble_3.1.1 ggplot2_3.3.3 tidyverse_1.3.1 sparklyr_1.6.2
loaded via a namespace (and not attached):
[1] Rcpp_1.0.6 lubridate_1.7.10 forge_0.2.0 prettyunits_1.1.1 ps_1.6.0 assertthat_0.2.1 rprojroot_2.0.2 digest_0.6.27 utf8_1.2.1
[10] R6_2.5.0 cellranger_1.1.0 backports_1.2.1 reprex_2.0.0 httr_1.4.2 pillar_1.6.0 rlang_0.4.11 readxl_1.3.1 rstudioapi_0.13
[19] callr_3.7.0 desc_1.3.0 config_0.3.1 devtools_2.4.1 r2d3_0.2.5 htmlwidgets_1.5.3 munsell_0.5.0 broom_0.7.6 compiler_4.0.5
[28] modelr_0.1.8 pkgconfig_2.0.3 base64enc_0.1-3 pkgbuild_1.2.0 htmltools_0.5.1.1 tidyselect_1.1.1 fansi_0.4.2 crayon_1.4.1 dbplyr_2.1.1
[37] withr_2.4.2 grid_4.0.5 jsonlite_1.7.2 gtable_0.3.0 lifecycle_1.0.0 DBI_1.1.1 magrittr_2.0.1 scales_1.1.1 writexl_1.4.0
[46] cli_2.5.0 stringi_1.6.1 cachem_1.0.4 remotes_2.3.0 fs_1.5.0 testthat_3.0.2 xml2_1.3.2 ellipsis_0.3.2 generics_0.1.0
[55] vctrs_0.3.8 tools_4.0.5 glue_1.4.2 hms_1.0.0 pkgload_1.2.1 processx_3.5.2 parallel_4.0.5 fastmap_1.1.0 yaml_2.2.1