I create and persist a df1 on which then I am doing the below:
df1.persist (From the Storage Tab in spark UI it says it is 3Gb)
df2=df1.groupby(col1).pivot(col2) (This is a df with 4.827 columns and 40107 rows)
df2.collect
df3=df1.groupby(col2).pivot(col1) (This is a df with 40.107 columns and 4.827 rows)
-----it hangs here for almost 2 hours-----
df4 = (..Imputer or na.fill on df3..)
df5 = (..VectorAssembler on df4..)
(..PCA on df5..)
df1.unpersist
I have a cluster with 16 nodes(each node has 1 worker with 1 executor with 4 cores and 24Gb Ram) and a master(with 15Gb of Ram). Also spark.shuffle.partitions is 192. It hangs for 2 hours and nothing is happening. Nothing is active in Spark UI. Why does it hang for so long? Is it the DagScheduler? How can I check it? Please let me know if you need any more information.
----Edited 1----
After waiting for almost two hours it proceeds and then eventually fails. Below is the stages and executor tabs from Spark UI:
Also, in the stderr file in the worker nodes it says:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000003fe900000, 6434586624, 0) failed; error='Cannot allocate memory' (errno=12)
Moreover, it seems there is a file produced named "hs_err_pid11877" in the folder next to stderr and stdout which says:
There is insufficient memory for the Java Runtime Environment to continue. Native memory allocation (mmap) failed to map 6434586624 bytes for committing reserved memory. Possible reasons: The system is out of physical RAM or swap space The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap Possible solutions: Reduce memory load on the system Increase physical memory or swap space Check if swap backing store is full Decrease Java heap size (-Xmx/-Xms) Decrease number of Java threads Decrease Java thread stack sizes (-Xss) Set larger code cache with -XX:ReservedCodeCacheSize= JVM is running with Zero Based Compressed Oops mode in which the Java heap is placed in the first 32GB address space. The Java Heap base address is the maximum limit for the native heap growth. Please use -XX:HeapBaseMinAddress to set the Java Heap base and to place the Java Heap above 32GB virtual address. This output file may be truncated or incomplete. Out of Memory Error (os_linux.cpp:2792), pid=11877, tid=0x00007f237c1f8700 JRE version: OpenJDK Runtime Environment (8.0_265-b01) (build 1.8.0_265-8u265-b01-0ubuntu2~18.04-b01) Java VM: OpenJDK 64-Bit Server VM (25.265-b01 mixed mode linux-amd64 compressed oops) Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
...and other information about the task it fails, GC information, etc..
----Edited 2----
Here is the Tasks Section of the the last pivot(stage with id 16 from stages picture).. just before the hanging. It seems that all 192 partitions have a pretty equal amount of data, from 15 to 20MB.
