according to the docs pandas UDF's work the following way:
The Python function should take a pandas Series as an input and return a pandas Series of the same length, and you should specify these in the Python type hints. Spark runs a pandas UDF by splitting columns into batches, calling the function for each batch as a subset of the data, then concatenating the results.
Is it possible to run the pandas UDF only on a single batch? I saw the option spark.sql.execution.arrow.maxRecordsPerBatch, but since its default to 10'000 and my dataset that is much smaller still runs in batches.