Why can't I set a series type to equal another series type with Python pandas

user_8728073 0
Viewed 796

I'm fairly new to python so forgive me if this seems like a simple question.

I have a dataframe. My goal is to take the values of a dataframe and convert it into another type and replace that column. Here is the codes:

strtotime = {}
for x in range(0,len(results['CreationDate'])):
    strtotime[x] = datetime.strptime(results['CreationDate'][x], '%Y-%m-%dT%H:%M:%S.%f')
results['CreationDate'] = pd.to_datetime(pd.Series(strtotime))

I stored the values as a dictionary, converted it to a series using pd.Series, at which point I'm fairly certain I can just replace one series with another:

i.e results['CreationDate'] = pd.to_datetime(pd.Series(strtotime))

but what I get in return for results is a column of NaT instead of these neat datetimes 2015-01-01 10:59:37.403.

I then used results['CreationDate'] = list(pd.to_datetime(pd.Series(strtotime)))

which worked perfectly as I wanted it to be. So my question is why is this the case? Does it even have anything to do with object types?

pandas python-3.x datetime python
1 Answers

When you assign a Series to a DataFrame column, pandas matches the new values according to the index. Your original DataFrame presumably has some meaningful index, but your new Series it just has the default index of 0, 1, 2, 3... because those are the keys in your dictionary. Here is a simple example:

>>> d = pandas.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]}, index=[10, 11, 12])
>>> d
    A  B
10  1  4
11  2  5
12  3  6
>>> d["C"] = pandas.Series([8, 88, 888])
>>> d
    A  B   C
10  1  4 NaN
11  2  5 NaN
12  3  6 NaN
>>> d["C"] = pandas.Series([8, 88, 888], index=[10, 11, 12])
>>> d
    A  B    C
10  1  4    8
11  2  5   88
12  3  6  888

Notice that assigning a series with the wrong index resulted in NaN, but creating the new Series with the same index results in the values being put in as expected.

In your case, you are creating your new Series by applying a function to each element of the original column. Don't iterate to do that. Use the .map method. In this case, there is a builtin pandas function to convert a string to a datetime:

results['CreationDate'] = results['CreationDate'].map(pandas.to_datetime)

.map gives a new Series with the same index as the old. (If your dates don't parse correctly, you can apply a lambda that supplies a format argument to to_datetime.)

(As piRsquared noted in a comment, to_datetime actually accepts a Series argument, so you can just do results['CreationDate'] = pandas.to_datetime(results['CreationDate']).)

user_14274160
Related