PySpark, avoid StackOverflowException after call dataframe withColomn method a lot of times

user_7904752 0
Viewed 153

AWS Glue Spark2.4 Python3 Glue Version2.0

Ifound StackOverflowException after call dataframe withColomn method a lot of times.

like...

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "database_name", 
table_name = "table_name", transformation_ctx = "datasource0")
df = datasource0.toDF()

df = df.withColumn('item_name', F.regexp_replace(F.col('item_name'), '^foo$', 'bar'))
df = df.withColumn('item_name', F.regexp_replace(F.col('item_name'), '^foo$', 'bar'))
df = df.withColumn('item_name', F.regexp_replace(F.col('item_name'), '^foo$', 'bar'))
df = df.withColumn('item_name', F.regexp_replace(F.col('item_name'), '^foo$', 'bar'))
... # and call hundreds times

Document says

Blockquote This method introduces a projection internally. Therefore, calling it multiple times, for instance, via loops in order to add multiple columns can generate big plans which can cause performance issues and even StackOverflowException. To avoid this, use select() with the multiple columns at once.

So I understand that I need to use select() with the multiple columns at once.

But I don't know how write in code.

aws-glue pyspark python
1 Answers

As per this yes withColumn will cause memory related issues and this can be prevented by using select as shown below:

df.select(F.regexp_replace(F.col('item_name')))

If you want to apply the same for multiple columns then you can as shown below:

df.select(F.regexp_replace(F.col('item_name')),F.regexp_replace(F.col('item_name')),.........upto n number of columns)
user_43269220
Related