I try to run following code but facing such error.
from pyspark.sql.functions import udf
def retrieve_night_flag(row):
x=row
if x >= night_time['start_time'] and x < night_time['end_time']:
return 'NIGHT'
elif x >= day_time['start_time'] and x < day_time['end_time']:
return 'DAY'
else:
return 'EVE'
UDF_NAME = udf(lambda row: retrieve_night_flag(row),StringType())
df_source3 =df_source2 .withColumn('night_flag', UDF_NAME(col('hour_stamp')))
error we found:
Py4JJavaError: An error occurred while calling o765.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 38.0 failed 1 times, most recent failure: Lost task 0.0 in stage 38.0 (TID 35) (INNOLX4356.itte.com executor driver): org.apache.spark.SparkException:
Error from python worker:
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 174, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/lib/python3.5/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "<frozen importlib._bootstrap>", line 969, in _find_and_load
File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 664, in _load_unlocked