Caused by: java.lang.NullPointerException - pyspark data processing

Viewed 341

I have created the spark session like this,

spark = SparkSession.builder \
    .appName("Spark NLP")\
    .master("local[*]")\
    .config("spark.driver.memory","256G")\
    .config("spark.driver.maxResultSize", "4096M") \
    .getOrCreate()

I have my UDF function below to take my string (document of size around 1500 characters) and parse to take out required entities from my data.

def parseStudentDetails(s):

    student_name_pattern = re.compile(r'(?:Student:)[^\r\n]*')
    student_dob_pattern = re.compile(r'(?:D.O.B:|DOB:)[^\r\n]*')
    student_age_pattern = re.compile(r'(?:AGE:)[^\r\n]*')
    date_of_exam_pattern = re.compile(r'(?:Date:|Date of exam:)[^\r\n]*')

    _student_name = re.search(student_name_pattern,s)
    _student_dob = re.search(student_dob_pattern,s)
    _student_age = re.search(student_age_pattern,s)
    _date_examined = re.search(date_of_exam_pattern,s)

    student_name =  None if not _student_name else (_student_name.group().split(':')[1].strip() if ':' in _student_name.group() else None)
    student_dob =  None if not _student_dob else (str(datetime.strptime(_student_dob.group().split(':')[1].strip(),'%d/%m/%Y').date()) if ':' in _student_dob.group() else None)
    student_age =  None if not _student_age else (_student_age.group().split(':')[1].strip() if ':' in _student_age.group() else None)
    date_examined =  None if not _date_examined else (str(datetime.strptime(_date_examined.group().split(':')[1].strip(),'%d/%m/%Y').date()) if ':' in _date_examined.group() else None)
    results = Row(student_name,student_dob,student_age,date_examined)

    return results

parseStudentDetailsSchema = StructType([
        StructField('student_name', StringType(), False),
        StructField('student_dob', StringType(), False),
        StructField('student_age', StringType(), False),
        StructField('date_examined', StringType(), False)
])

parseStudentDetailsUdf = udf(parseStudentDetails, parseStudentDetailsSchema)

I am trying to process around 25k documents through the above function,

dataWithStudentDetails = datav1.withColumn('student_details',parseStudentDetailsUdf('text_report')) \
    .withColumn('Referral', F.when(col("text_report").startswith("To:"), True).otherwise(False))

I keep getting the below error, not sure what is wrong here, the challenge is that this error occurs sometime, not some other time, so it is hard to understand why this is happening, error message is not descriptive enough, please help.

Caused by: java.lang.NullPointerException

I would debug by adding print statements in python, but not sure what is causing this and how to debug this in spark. Also, if there is an efficient way to rewrite this code.

Thanks.

0 Answers
Related