I am using Pyspark UDF where I am calling a function that either returns a Pandas Dataframe or just None.
But the error says that
TypeError: Return type of the user-defined function should be pandas.DataFrame, but is <class 'NoneType'>
Here is my code looks like
@pandas_udf(schema,functionType=PandasUDFType.GROUPED_MAP)
def my_func(df):
if condition:
return pd.DataFrame(....)
I am calling the function this way:
df.groupby('id').apply(my_func)
How can I handle the cases where the condition fails or also in case of any exceptions occur?