Checking a column in dataframe is in Epoch time is giving different results

user_9157627 0
Viewed 72

I wrote the below test code to check it is working properly and I get a "True" Value in the DF, but when i sure the same calculation on the just one value, it gives me "False". The correct value should be "False" as the date_time column is not in epoch/UNIX time

import pandas as pd
from datetime import datetime, timedelta
temp_list = []
i = 1
while i < 10:
    d = {
        'ticker': 'TEST',
        'date_time': datetime.now() + timedelta(days=i),
        'price': 100 + i,
        'volume': i
    }
    temp_list.append(d)
    i += 1
test_df = pd.DataFrame(data=temp_list)
test_df['isepoch'] = pd.notnull(pd.to_numeric(
    test_df['date_time'], errors='coerce'))
print(test_df)
print(pd.notnull(pd.to_numeric(test_df['date_time'][0], errors='coerce')))

Output of the two print statements:

  ticker                  date_time  price  volume  isepoch
0   TEST 2021-09-16 10:33:43.285935    101       1     True
1   TEST 2021-09-17 10:33:43.285935    102       2     True
2   TEST 2021-09-18 10:33:43.285935    103       3     True
3   TEST 2021-09-19 10:33:43.285935    104       4     True
4   TEST 2021-09-20 10:33:43.285935    105       5     True
5   TEST 2021-09-21 10:33:43.285935    106       6     True
6   TEST 2021-09-22 10:33:43.285935    107       7     True
7   TEST 2021-09-23 10:33:43.285935    108       8     True
8   TEST 2021-09-24 10:33:43.285935    109       9     True
False

I adapted the solution from Pandas: Check if value is epoch time using python, to create the 'isepoch' column in the dataframe.

python-3.9 pandas dataframe epoch python
1 Answers

If you heck pandas.to_numeric, scalar, list, tuple, 1-d array, or Series can be converted as an argument for pandas.to_numeric.

Above your code, the part as following :

test_df['isepoch'] = pd.notnull(pd.to_numeric(test_df['date_time'], errors='coerce'))

returns true as pandas.to_numeric accepts series value.

However, the test_df['date_time'][0] is neither, scalar, list, tuple, 1-d array, or Series as following :

In [1]: print(type(test_df['date_time'][0]) == list)
   ...: print(type(test_df['date_time'][0]) == tuple)
   ...: print(type(test_df['date_time'][0]) == np.array)
   ...: print(type(test_df['date_time'][0]) == pd.Series)
   ...: print(np.isscalar(test_df['date_time'][0]))
False
False
False
False
False

Therefore, if you change test_df['date_time'][0] as argument type which can be acceptable in pandas.to_numeric, it will return True value. Just like as following :

In [2]: pd.notnull(pd.to_numeric(np.array(test_df['date_time'][0]), errors='coerce'))
Out[2]: True

If you want to know more about pandas.to_numeric source code, you can visit here.

user_117025910
Related