I am trying to get a concrete understanding on how the pandas UDF grouped map is working. While looking at the code here[1], i see that first the arrow object is converted into pandas series and then pd.concat is applied to create the full data frame.
What is confusing for me is , since arrow has support for Tabular format[2] and an API exist for converting table format to pandas in pyarrow[3] , why is that not being used.
I am pretty sure i am missing out on something very basic, any pointers would be useful ?
[2] https://arrow.apache.org/docs/cpp/tables.html
[3] https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas