How to tag corrupted data in dataframe after an error has been raised

Viewed 190

I have a large dataframe containing, amongst other things, a (Norwegian) social security number. It is possible to get the date of birth out of this number via a special algorithm. However, every now and then an illegal social security number creeps into the database corrupting the calculation.

What I would like to do is to tag every line having an illegal social security number, along with a log message showing the error raised.

Consider the following, constructed, example

import pandas as pd
from datetime import date

sample_data = pd.DataFrame({'id' : [1, 2, 3], \
                            'sec_num' : [19790116, 19480631, 19861220]})


# The actual algorithm transforming the sec number is more complicated
# this is just for illustration purposes
def int2date(argdate: int):

    try:
        year = int(argdate / 10000)
        month = int((argdate % 10000) / 100)
        day = int(argdate % 100)
        return date(year, month, day)
    except ValueError:
        raise ValueError("Value:{0} not a legal date.".format(argdate))

I want to create the following output

   id   sec_num date_of_birth  is_in_error                    error_msg
0   1  19790116    1979-01-16        False  
1   2  19480631          None         True 19480631 is not a legal date         
2   3  19861220    1986-12-20        False 

I have tried

try:
    sample_data['date_of_birth'] = [int2date(sec_num) for \
                   sec_num in sample_data['sec_num']]
    sample_data['is_in_error'] = False
    sample_data['error_msg'] = ''
 except ValueError as e:
    sample_data['is_in_error'] = True
    sample_data['error_msg'] = str(e)

but this produces this

   id   sec_num  is_in_error                         error_msg
0   1  19790116         True  Value:19480631 not a legal date.
1   2  19480631         True  Value:19480631 not a legal date.
2   3  19861220         True  Value:19480631 not a legal date.

I guess the problem is that I assign the date_of_birth-column in one operation and the errors in another. I am not sure how to catch and create the is_in_error and error_msg columns simulationously.

2 Answers
Related