Safe to assume pyspark's subtract on a rdd is slow / problematic?

user_2929998 0
Viewed 249

While running a pyspark job and as the input grows, I notice that I keep getting memory errors like the following...

ERROR cluster.YarnScheduler: Lost executor 12 on compute-2-10.local:
Container killed by YARN for exceeding memory limits. 1.5 GB of 1.5 GB
physical memory used. Consider boosting spark.yarn.executor.memoryOverhead.

While it will successfully complete on a smaller input even when the error is repeated through execution, it eventually dies as the input size increases.

I have ~20,000,000 million rows that I need to filter out ~661,000 rows from that. I can't think of any other way of dealing with this besides using subtract given the format of the key.

pyspark
2 Answers

1.5GB is very low for executor memory: typically try to give at least twice that amount. This problem should likely be resolved just by not starving the executor of resources.

user_10565630

I'd use at least 4gb of memory. You can set it like this: pyspark --num-executors 5 --driver-memory 2g --executor-memory 4g

user_82623420
Related