How to process the sub dataframe in parallel in pandas

user_6407393 0
Viewed 54
df = pd.read_csv("imeis_events_march.csv")
imeis = df['imei'].unique().tolist()
for imei in imeis:
    dfj = df.loc[df['imei']==imei]
    # calculate feature
    props = calculate_ratio(dfj)

I want to use the calculate_ratio function to process each sub dataframe in an efficient way. Is there any good approach to implement this in parallel ? The desiring result would be

imei:props

for each imei.

pyspark pandas
1 Answers

You can create a PySpark dataframe, group it by imei and then use applyInPandas to process each group as a separate Pandas dataframe in parallel:

pdf = pd.read_csv("imeis_events_march.csv")
df = spark.createDataFrame(pdf)

def calculate_ratio(pandas_df):
    pandas_df['ratio'] = 1.0 #implement the real logic here
    return pandas_df

schema_with_ratio = "imei string, ...other columns..., ratio float"

df.groupBy("imei").applyInPandas(calculate_ratio, schema_with_ratio).show()

calculate_ratio will be called by Spark with a Pandas dataframe once for each imei in the input data.

A downside of this approach is that each group of imeis has to fit completey into the memory of one of the Spark executors or an OutOfMemory error will occur.

user_21298010
Related