Convert column to string, retaining NaN (as None or blank)

Viewed 6138

I would like to format a bunch of numbers in a list. The easiest way to do this is to convert it first to a bunch of strings. Here's an example of how I'm doing this:

df[col_name].astype('str').tolist()

However, the issue with this is I get values such as:

['12.19', '13.99', '1.00', 'nan', '9.00']

Is there a way I can return the 'nan' values as either None or an empty string, for example:

['12.19', '13.99', '1.00', None, '9.00']

Or:

['12.19', '13.99', '1.00', '', '9.00']

How would I do these two?

7 Answers

You can try like this.

1st way:

>>> df[col_name].apply(lambda v: str(v) if str(v) != 'nan' else None).tolist()
['12.19', '13.99', '1.00', None, '9.00']
>>>
>>> df[col_name].apply(lambda v: str(v) if str(v) != 'nan' else '').tolist()
['12.19', '13.99', '1.00', '', '9.00']
>>>

2nd way:

>>> df[col_name].apply(lambda v: str(v) if not pd.isnull(v) else None).tolist()
['12.19', '13.99', '1.00', None, '9.00']
>>>
>>> df[col_name].apply(lambda v: str(v) if not pd.isnull(v) else '').tolist()
['12.19', '13.99', '1.00', '', '9.00']
>>>

Here is the detailed explanation.

>>> import pandas as pd
>>> import numpy as np
>>>
>>> df = pd.DataFrame({
... "fullname": ['P Y', 'P T', 'T Y', 'N A', 'P Z'],
... "age": [36, 80, 25, 8, 34],
... "salary": ['12.19', '13.99', '1.00', np.nan, '9.00']
... })
>>>
>>> df
  fullname  age salary
0      P Y   36  12.19
1      P T   80  13.99
2      T Y   25   1.00
3      N A    8    NaN
4      P Z   34   9.00
>>>
>>> # PROBLEM
...
>>> col_name = "salary"
>>> df[col_name].astype("str").tolist()
['12.19', '13.99', '1.00', 'nan', '9.00']
>>>
>>> # SOLUTION
...
>>> df[col_name].apply(lambda v: str(v) if str(v) != 'nan' else None)
0    12.19
1    13.99
2     1.00
3     None
4     9.00
Name: salary, dtype: object
>>>
>>> df[col_name].apply(lambda v: str(v) if str(v) != 'nan' else '')
0    12.19
1    13.99
2     1.00
3
4     9.00
Name: salary, dtype: object
>>>
>>> df[col_name].apply(lambda v: str(v) if str(v) != 'nan' else None).tolist()
['12.19', '13.99', '1.00', None, '9.00']
>>>
>>> df[col_name].apply(lambda v: str(v) if str(v) != 'nan' else '').tolist()
['12.19', '13.99', '1.00', '', '9.00']
>>>
>>> df[col_name].apply(lambda v: str(v) if not pd.isnull(v) else None).tolist()
['12.19', '13.99', '1.00', None, '9.00']
>>>
>>> df[col_name].apply(lambda v: str(v) if not pd.isnull(v) else '').tolist()
['12.19', '13.99', '1.00', '', '9.00']
>>>

Use df.astype(str, skipna=True), it will skip all NA types.

Example:

import pandas as pd
df=pd.Series([12.19, 13.99, 1.00, None, 9.00])
print(df.astype(str, skipna=True).to_list())
pd.isna(df.astype(str, skipna=True))

Output:

['12.19', '13.99', '1.0', nan, '9.0']
0    False
1    False
2    False
3     True
4    False
dtype: bool

If you really need it to be None instead of np.nan, then add df=df.where(pd.notnull(df), None).

Example:

df=pd.Series([12.19, 13.99, 1.00, None, 9.00])
df=df.astype(str, skipna=True)
df=df.where(pd.notnull(df), None)
print(df.to_list())

Output:

['12.19', '13.99', '1.0', None, '9.0']

Note: skipna parameter vanished from .astype() in pandas 1.0 release, and the issue is currently open as of 2/6/2020.

astype(str) / astype_unicode: np.nan converted to "nan" (checknull, skipna)

Series.astype(str, skipna=True) vanished in the 1.0 release

try use fillna()

df[col_name].fillna('').astype('str').tolist()

This is a unique requirement, and I believe is best answered with a list comprehension:

df[col_name]
0    12.19
1    13.99
2     1.00
3      NaN
4     9.00
dtype: float64

[str(v_) if pd.notna(v_) else None for v_ in df[col_name]]
# ['12.19', '13.99', '1.0', None, '9.0'] 

If you would rather the values were filled in as blanks, that's equally simple:

[str(v_) if pd.notna(v_) else '' for v_ in df[col_name]]
# ['12.19', '13.99', '1.0', '', '9.0'] 

You can either do this:

df[col_name].fillna('').astype('str').tolist()

OR

l = df[col_name].astype('str').tolist()

Replace empty elements from the above created list with None:

list(map(lambda x: float(x) if x else None, l))

You can try removing the nan values after you create the list.

list = ["nan","1.27"]
for x in range(len(list)):
    if list[x] == "nan":
        list[x] = None # Or list[x] = ""

I don't have any knowledge of pandas so this might not be the best solution.

.isalpha() will work:

l = ['12.19', '13.99', '1.00', 'nan', '9.00']
print([None if i.isalpha() else i for i in l])

['12.19', '13.99', '1.00', None, '9.00']

Related