I have a python function that I converted to Pandas_UDF function and it worked fine up until last week but getting the below error from the last few days. We tried a simple python function with Pandas UDF and it is not throwing this error. I am not sure what exactly in my code is causing this. Has there been any change to spark environment. I am using Azure Databricks if it helps.
Search only turned up the this link but it is old.
Appreciate any pointers on how to fix this issue.
Thanks, Yudi
SparkException: Job aborted due to stage failure: Task 0 in stage 23.0 failed 4 times, most recent failure: Lost task 0.3 in stage 23.0 (TID 252, 172.17.69.7, executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last): File "/databricks/spark/python/pyspark/serializers.py", line 180, in _read_with_length return self.loads(obj) File "/databricks/spark/python/pyspark/serializers.py", line 669, in loads return pickle.loads(obj, encoding=encoding) File "/databricks/spark/python/pyspark/cloudpickle.py", line 875, in subimport import(name) ImportError: No module named '_pandasujson'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/databricks/spark/python/pyspark/worker.py", line 394, in main func, profiler, deserializer, serializer = read_udfs(pickleSer, infile, eval_type) File "/databricks/spark/python/pyspark/worker.py", line 234, in read_udfs arg_offsets, udf = read_single_udf(pickleSer, infile, eval_type, runner_conf) File "/databricks/spark/python/pyspark/worker.py", line 160, in read_single_udf f, return_type = read_command(pickleSer, infile) File "/databricks/spark/python/pyspark/worker.py", line 69, in read_command command = serializer._read_with_length(file) File "/databricks/spark/python/pyspark/serializers.py", line 183, in _read_with_length raise SerializationError("Caused by " + traceback.format_exc()) pyspark.serializers.SerializationError: Caused by Traceback (most recent call last): File "/databricks/spark/python/pyspark/serializers.py", line 180, in _read_with_length return self.loads(obj) File "/databricks/spark/python/pyspark/serializers.py", line 669, in loads return pickle.loads(obj, encoding=encoding) File "/databricks/spark/python/pyspark/cloudpickle.py", line 875, in subimport import(name) ImportError: No module named '_pandasujson'