Pyspark Column name alias when applying Aggregate using a Dictionary

user_2557844 0
Viewed 694

I am applying an aggregate function on a data frame in pyspark. I am using a dictionary to pass the column name and aggregate function

df.groupBy(column_name).agg({"column_name":"sum"})

I now want to apply an alias to this column that has been generated using the aggregate method. Is there a way to do it?

The reason I am using the dictionary method is that aggregates will be applied dynamically depending on input parameters.

So basically it will be like

def aggregate(df, column_to_group_by, columns_to_aggregate):
     df.groupBy(column_to_group_by).agg(columns_to_aggregate)

Where columns_to_aggregate will look like

{
   "salary":"sum"
}

I now want to apply alias to the newly created column, because If I try to save the result to disk as praquet I get the error

Column name "sum(salary)" contains invalid character(s). Please use alias to rename it.

Any help on how to apply alias dynamically will be great

Thanks !

pyspark apache-spark python
2 Answers
from pyspark.sql.functions import sum
df.groupBy("state") \
  .agg(sum("salary").alias("sum_salary"))

Please read the article

user_171150120

I can see that this question is from 4 months ago. Here is the link to a possible solution where you rename the columns after aggregation by replacing some characters: https://stackoverflow.com/a/70101696

The provided solution:

df.groupBy('group')
  .agg({'money':'sum',
        'moreMoney':'sum',
        'evenMoreMoney':'sum'
        })
    .select(*(col(i).alias(i.replace("(",'_').replace(')','')) for i in df.columns))

It will create columns: sum_money, sume_moreMoney etc. And of course you can choose to rename/replace differently.

user_195875290
Related