I am first time trying to use sparklyr and was copying a rds read dataset in r to spark connect. But it kept on going for 30-45 minutes and nothing happened. Is this an expected behavior and I should wait longer or there is something wrong ?
Number of rows in nasdaq_data_tbl data: 24,197,442
Update: Ran again for almost 3 hours & still it kept on running.
nasdaq_data_tbl <- read_rds("Nasdaq_stocks_compiled_data.rds")
sc <- spark_connect(
+ master = "local",
+ version = "3.3.0",
+ config = conf
+ )
>
> sc
output
$master
[1] "local[12]"
$method
[1] "shell"
$app_name
[1] "sparklyr"
$config
$config$sparklyr.cores.local
[1] 12
$config$`sparklyr.shell.driver-memory`
[1] "10G"
$config$spark.memory.fraction
[1] 0.9
$config$sparklyr.shell.name
[1] "sparklyr"
$state
<environment: 0x000002167ad704e0>
$extensions
$extensions$jars
character(0)
$extensions$packages
character(0)
$extensions$initializers
list()
$extensions$catalog_jars
character(0)
$extensions$repositories
character(0)
$extensions$dbplyr_sql_variant
$extensions$dbplyr_sql_variant$scalar
list()
$extensions$dbplyr_sql_variant$aggregate
list()
$extensions$dbplyr_sql_variant$window
list()
$spark_home
[1] "C:\\Users\\vinee\\AppData\\Local\\spark\\spark-3.3.0-bin-hadoop2"
$backend
A connection with
description "->localhost:8881"
class "sockconn"
mode "wb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
$monitoring
A connection with
description "->localhost:8881"
class "sockconn"
mode "wb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
$gateway
A connection with
description "->localhost:8880"
class "sockconn"
mode "rb"
text "binary"
opened "opened"
can read "yes"
can write "yes"
$output_file
[1] "C:\\Users\\vinee\\AppData\\Local\\Temp\\RtmpArX6Gd\\file108c72b437f5_spark.log"
$sessionId
[1] 27127
$home_version
[1] "3.3.0"
attr(,"class")
[1] "spark_connection" "spark_shell_connection" "DBIConnection"
nasdaq_data <- copy_to(sc,
nasdaq_data_tbl,
name = "nasdaq_data",
overwrite = TRUE)
Log:
22/09/08 15:35:14.809 nioEventLoopGroup-2-2 INFO HiveConf: Found configuration file file:/C:/Users/vinee/AppData/Local/spark/spark-3.3.0-bin-hadoop2/conf/hive-site.xml
22/09/08 15:35:14.989 nioEventLoopGroup-2-2 INFO SparkContext: Running Spark version 3.3.0
22/09/08 15:35:15.105 nioEventLoopGroup-2-2 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/09/08 15:35:15.194 nioEventLoopGroup-2-2 WARN SparkConf: Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN).
22/09/08 15:35:15.228 nioEventLoopGroup-2-2 INFO ResourceUtils: ==============================================================
22/09/08 15:35:15.229 nioEventLoopGroup-2-2 INFO ResourceUtils: No custom resources configured for spark.driver.
22/09/08 15:35:15.229 nioEventLoopGroup-2-2 INFO ResourceUtils: ==============================================================
22/09/08 15:35:15.230 nioEventLoopGroup-2-2 INFO SparkContext: Submitted application: sparklyr
22/09/08 15:35:15.263 nioEventLoopGroup-2-2 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
22/09/08 15:35:15.276 nioEventLoopGroup-2-2 INFO ResourceProfile: Limiting resource is cpu
22/09/08 15:35:15.277 nioEventLoopGroup-2-2 INFO ResourceProfileManager: Added ResourceProfile id: 0
22/09/08 15:35:15.336 nioEventLoopGroup-2-2 INFO SecurityManager: Changing view acls to: vinee
22/09/08 15:35:15.336 nioEventLoopGroup-2-2 INFO SecurityManager: Changing modify acls to: vinee
22/09/08 15:35:15.337 nioEventLoopGroup-2-2 INFO SecurityManager: Changing view acls groups to:
22/09/08 15:35:15.337 nioEventLoopGroup-2-2 INFO SecurityManager: Changing modify acls groups to:
22/09/08 15:35:15.337 nioEventLoopGroup-2-2 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(vinee); groups with view permissions: Set(); users with modify permissions: Set(vinee); groups with modify permissions: Set()
22/09/08 15:35:15.499 nioEventLoopGroup-2-2 INFO Utils: Successfully started service 'sparkDriver' on port 57576.
22/09/08 15:35:15.523 nioEventLoopGroup-2-2 INFO SparkEnv: Registering MapOutputTracker
22/09/08 15:35:15.565 nioEventLoopGroup-2-2 INFO SparkEnv: Registering BlockManagerMaster
22/09/08 15:35:15.588 nioEventLoopGroup-2-2 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/09/08 15:35:15.588 nioEventLoopGroup-2-2 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/09/08 15:35:15.593 nioEventLoopGroup-2-2 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
22/09/08 15:35:15.619 nioEventLoopGroup-2-2 INFO DiskBlockManager: Created local directory at C:\Users\vinee\AppData\Local\spark\spark-3.3.0-bin-hadoop2\tmp\local\blockmgr-22141f7a-7b26-481c-bf41-1396cb4d47c3
22/09/08 15:35:15.637 nioEventLoopGroup-2-2 INFO MemoryStore: MemoryStore started with capacity 8.7 GiB
22/09/08 15:35:15.659 nioEventLoopGroup-2-2 INFO SparkEnv: Registering OutputCommitCoordinator
22/09/08 15:35:15.662 nioEventLoopGroup-2-2 WARN Utils: The configured local directories are not expected to be URIs; however, got suspicious values [C:/Users/vinee/AppData/Local/spark/spark-3.3.0-bin-hadoop2/tmp/local]. Please check your configured local directories.
22/09/08 15:35:15.885 nioEventLoopGroup-2-2 INFO Utils: Successfully started service 'SparkUI' on port 4040.
22/09/08 15:35:15.952 nioEventLoopGroup-2-2 INFO SparkContext: Added JAR file:/C:/Users/vinee/AppData/Local/R/win-library/4.2/sparklyr/java/sparklyr-master-2.12.jar at spark://LAPTOP-2FLMJ29F:57576/jars/sparklyr-master-2.12.jar with timestamp 1662631514979
22/09/08 15:35:16.035 nioEventLoopGroup-2-2 INFO Executor: Starting executor ID driver on host LAPTOP-2FLMJ29F
22/09/08 15:35:16.044 nioEventLoopGroup-2-2 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
22/09/08 15:35:16.059 nioEventLoopGroup-2-2 INFO Executor: Fetching spark://LAPTOP-2FLMJ29F:57576/jars/sparklyr-master-2.12.jar with timestamp 1662631514979
22/09/08 15:35:16.104 nioEventLoopGroup-2-2 INFO TransportClientFactory: Successfully created connection to LAPTOP-2FLMJ29F/192.168.68.102:57576 after 18 ms (0 ms spent in bootstraps)
22/09/08 15:35:16.109 nioEventLoopGroup-2-2 INFO Utils: Fetching spark://LAPTOP-2FLMJ29F:57576/jars/sparklyr-master-2.12.jar to C:\Users\vinee\AppData\Local\spark\spark-3.3.0-bin-hadoop2\tmp\local\spark-922522cb-5b48-4929-acb6-924306e64b01\userFiles-b1ee1537-a761-4a88-a0c1-7d27717fe4b5\fetchFileTemp8040704204737501265.tmp
22/09/08 15:35:16.204 nioEventLoopGroup-2-2 INFO Executor: Adding file:/C:/Users/vinee/AppData/Local/spark/spark-3.3.0-bin-hadoop2/tmp/local/spark-922522cb-5b48-4929-acb6-924306e64b01/userFiles-b1ee1537-a761-4a88-a0c1-7d27717fe4b5/sparklyr-master-2.12.jar to class loader
22/09/08 15:35:16.218 nioEventLoopGroup-2-2 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57632.
22/09/08 15:35:16.219 nioEventLoopGroup-2-2 INFO NettyBlockTransferService: Server created on LAPTOP-2FLMJ29F:57632
22/09/08 15:35:16.220 nioEventLoopGroup-2-2 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/09/08 15:35:16.228 nioEventLoopGroup-2-2 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, LAPTOP-2FLMJ29F, 57632, None)
22/09/08 15:35:16.232 dispatcher-BlockManagerMaster INFO BlockManagerMasterEndpoint: Registering block manager LAPTOP-2FLMJ29F:57632 with 8.7 GiB RAM, BlockManagerId(driver, LAPTOP-2FLMJ29F, 57632, None)
22/09/08 15:35:16.236 nioEventLoopGroup-2-2 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, LAPTOP-2FLMJ29F, 57632, None)
22/09/08 15:35:16.237 nioEventLoopGroup-2-2 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, LAPTOP-2FLMJ29F, 57632, None)
22/09/08 15:35:16.613 nioEventLoopGroup-2-2 INFO SharedState: Setting hive.metastore.warehouse.dir ('C:/Users/vinee/AppData/Local/spark/spark-3.3.0-bin-hadoop2/tmp/hive') to the value of spark.sql.warehouse.dir.
22/09/08 15:35:16.619 nioEventLoopGroup-2-2 INFO SharedState: Warehouse path is 'file:/C:/Users/vinee/AppData/Local/spark/spark-3.3.0-bin-hadoop2/tmp/hive'.
22/09/08 15:35:19.929 nioEventLoopGroup-2-2 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.9 using Spark classes.
22/09/08 15:35:20.151 nioEventLoopGroup-2-2 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.9) is file:/C:/Users/vinee/AppData/Local/spark/spark-3.3.0-bin-hadoop2/tmp/hive
22/09/08 15:35:20.325 nioEventLoopGroup-2-2 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
22/09/08 15:35:20.326 nioEventLoopGroup-2-2 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
22/09/08 15:35:20.326 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
22/09/08 15:35:20.362 nioEventLoopGroup-2-2 INFO ObjectStore: ObjectStore, initialize called
22/09/08 15:35:20.482 nioEventLoopGroup-2-2 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
22/09/08 15:35:20.483 nioEventLoopGroup-2-2 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
22/09/08 15:35:21.501 nioEventLoopGroup-2-2 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
22/09/08 15:35:22.916 nioEventLoopGroup-2-2 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
22/09/08 15:35:22.918 nioEventLoopGroup-2-2 INFO ObjectStore: Initialized ObjectStore
22/09/08 15:35:22.978 nioEventLoopGroup-2-2 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
22/09/08 15:35:22.978 nioEventLoopGroup-2-2 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore UNKNOWN@192.168.68.102
22/09/08 15:35:22.987 nioEventLoopGroup-2-2 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
22/09/08 15:35:23.134 nioEventLoopGroup-2-2 INFO HiveMetaStore: Added admin role in metastore
22/09/08 15:35:23.136 nioEventLoopGroup-2-2 INFO HiveMetaStore: Added public role in metastore
22/09/08 15:35:23.182 nioEventLoopGroup-2-2 INFO HiveMetaStore: No user is added in admin role, since config is empty
22/09/08 15:35:23.271 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:23.273 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:23.289 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: global_temp
22/09/08 15:35:23.289 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: global_temp
22/09/08 15:35:23.290 nioEventLoopGroup-2-2 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
22/09/08 15:35:23.292 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:23.293 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:23.295 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:23.295 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:23.297 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_tables: db=default pat=*
22/09/08 15:35:23.297 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_tables: db=default pat=*
22/09/08 15:35:23.620 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:23.621 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:23.622 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:23.622 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:23.624 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_tables: db=default pat=*
22/09/08 15:35:23.624 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_tables: db=default pat=*
22/09/08 15:35:46.099 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:46.099 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:46.101 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_database: default
22/09/08 15:35:46.101 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_database: default
22/09/08 15:35:46.103 nioEventLoopGroup-2-2 INFO HiveMetaStore: 0: get_tables: db=default pat=*
22/09/08 15:35:46.103 nioEventLoopGroup-2-2 INFO audit: ugi=vinee ip=unknown-ip-addr cmd=get_tables: db=default pat=*
22/09/08 15:35:46.612 nioEventLoopGroup-2-2 INFO CodeGenerator: Code generated in 162.0456 ms
22/09/08 15:35:46.742 nioEventLoopGroup-2-2 INFO SparkContext: Starting job: collect at utils.scala:26
22/09/08 15:35:46.754 nioEventLoopGroup-2-2 INFO DAGScheduler: Job 0 finished: collect at utils.scala:26, took 0.004489 s
22/09/08 16:05:09.541 Thread-0 INFO SparkContext: Invoking stop() from shutdown hook
22/09/08 16:05:09.614 Thread-0 INFO SparkUI: Stopped Spark web UI at http://LAPTOP-2FLMJ29F:4040
22/09/08 16:05:09.655 dispatcher-event-loop-2 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/09/08 16:05:09.683 Thread-0 INFO MemoryStore: MemoryStore cleared
22/09/08 16:05:09.684 Thread-0 INFO BlockManager: BlockManager stopped
22/09/08 16:05:09.709 Thread-0 INFO BlockManagerMaster: BlockManagerMaster stopped
22/09/08 16:05:09.718 dispatcher-event-loop-5 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/09/08 16:05:09.734 Thread-0 INFO SparkContext: Successfully stopped SparkContext
22/09/08 16:05:09.734 Thread-0 INFO ShutdownHookManager: Shutdown hook called
22/09/08 16:05:09.736 Thread-0 INFO ShutdownHookManager: Deleting directory C:\Users\vinee\AppData\Local\Temp\spark-cf52ec32-d18a-4f92-b524-c6b168671971
22/09/08 16:05:09.738 Thread-0 INFO ShutdownHookManager: Deleting directory C:\Users\vinee\AppData\Local\spark\spark-3.3.0-bin-hadoop2\tmp\local\spark-922522cb-5b48-4929-acb6-924306e64b01