Equivalent of pandas apply in pyspark?

user_9629771 0
Viewed 3731

I really want to be able to run complex functions over a whole column of a spark dataframe, as i would do in Pandas with the apply function.

For example, in Pandas I have an apply function that takes a messy domain like sub-subdomain.subdomain.facebook.co.nz/somequerystring and just outputs facebook.com.

How would I do that in Spark?

I have looked at UDF's but I am not clear how I would run it on a single column.

Let's say I have a simple function like below where I extract different bits of a date from the column of the pandas DF:

def format_date(row):
    year = int(row['Contract_Renewal'][7:])
    month = int(row['Contract_Renewal'][4:6])
    day = int(row['Contract_Renewal'][:3])
    date = datetime.date(year, month, day)
    return date-now

In Pandas I would call it like:

df['days_until'] = df.apply(format_date, axis=1)

Can I achieve the same in Pyspark?

pyspark apache-spark
1 Answers

In this scenario, you may be able to use some combination of regexp_extract (http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=substring#pyspark.sql.functions.regexp_extract), regexp_replace (http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=substring#pyspark.sql.functions.regexp_replace), and split (http://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=substring#pyspark.sql.functions.split) to reformat the date the strings.

It's not as clean as defining your own function and using apply like Pandas, but it should be more performant than defining a Pandas/Spark UDF.

Good luck!

user_130736770
Related