A conceptual question to achieve high performance in Spark Job.
In ETL job, I am creating a data frame in Extract phase from a datasource (s3 or database or raw file),
i.e. df = extract(configs)
Then In transform phase, i.e. df_transform = transform(df, configs), I am using multiple python methods to transform my dataframe.
i.e. cleaning(df), setup_different_conditional_edits(df), windowing_function_transformations(df), other_statestical_tranformations(df)
All the transform functions will return df after applying transformations. (i.e. pandas style)
i.e.
def transform(df, config):
df = cleaning(df)
df = setup_different_conditional_edits(df)
df = windowing_function_transformations(df)
df = other_statestical_tranformations(df)
return dfNow the data frame I am creating in extract(configs) is quite huge (~ 10-100 GB). Assuming that I have enough memory and compute resources, is it a good practice to pass huge data frame between two python functions ?
or using UDF is a best bet ?
Also If I am using python class, Is it a good practice to declare spark dataframe as class instance variable ?
i.e. (conceptual)
class Extract(object):
def __init__(configs):
""" declaring configs as instance variable """
self.configs = configs
def run(self):
""" All Extraction Methods """
return df
class Transform(object):
def __init__(df):
""" passing df as instance variable """
self.df = df
def run(self):
""" All Transformation Methods """
self.df = cleaning()
self.df = setup_different_conditional_edits()
self.df = windowing_function_transformations()
self.df = other_statestical_tranformations()
return self.df
class Load(object):
def __init__(df):
""" passing df as instance variable """
self.df = df
def run():
""" Logic for loading transformed df """
return True