Pandas - can drop() function be used to remove missing values by applying it over the resulto of isnull() function?

user_13228427 0
Viewed 223

I am currently reading Introduction to Data Science book and all its code is written in python2 but I am trying to get it working on python3 So when I try to run:

edu.drop(edu['Value'].isnull(), axis=0)

it gives the following error:

KeyError                                  Traceback (most recent call last)
<ipython-input-60-445dd51fcf75> in <module>
----> 1 eduDrop = edu.drop(edu['Value'].isnull(), axis=0)
      2 eduDrop.head()

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in drop(self, labels, axis, index, columns, level, inplace, errors)
   3995             level=level,
   3996             inplace=inplace,
-> 3997             errors=errors,
   3998         )
   3999 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py in drop(self, labels, axis, index, columns, level, inplace, errors)
   3934         for axis, labels in axes.items():
   3935             if labels is not None:
-> 3936                 obj = obj._drop_axis(labels, axis, level=level, errors=errors)
   3937 
   3938         if inplace:

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\generic.py in _drop_axis(self, labels, axis, level, errors)
   3968                 new_axis = axis.drop(labels, level=level, errors=errors)
   3969             else:
-> 3970                 new_axis = axis.drop(labels, errors=errors)
   3971             result = self.reindex(**{axis_name: new_axis})
   3972 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\indexes\base.py in drop(self, labels, errors)
   5016         if mask.any():
   5017             if errors != "ignore":
-> 5018                 raise KeyError(f"{labels[mask]} not found in axis")
   5019             indexer = indexer[~mask]
   5020         return self.delete(indexer)

KeyError: '[ True  True False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n  True  True False False False False False False False False False False\n  True  True False False False False False False False False False False\n  True False False False False False False False False False False False\n  True False False False False False False False False False False False\n  True False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False  True  True  True  True  True  True\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n  True False False False False False False False  True  True  True  True\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False  True False  True False False False\n  True False False False False False False False False False False False\n False False False False False False False False False False False False\n False False False False False False False False False False False False\n False] not found in axis'

So, what is the problem here: does the current version of drop() does not support this or what?

pandas python-2.7 python-3.x python
2 Answers

The Issue

This is not a Python 2.7 / Python 3 issue. It's a wrong usage of Pandas. The drop method in the latest few versions of pandas expects either a list of column names (like ['Value']) or indices. You give it a series of True / False. I'm not aware of a version where it was different.

For future questions, make sure to add the versions of the relevant software.

Guessing: What do you want to achieve?

When you write

eduDrop = edu.drop(edu['Value'].isnull(), axis=0)

I guess that you want to drop all rows in which the column Value is null, right?

In future, please add this "what I want to achieve" to your question. It makes answers more likely to be useful.

Solution for the guess

I would use dropna like this:

df.dropna(subset=['Value'])
user_5627690

You can also use the following construct:

df[df.Value.notnull()]

which will select all the rows for which Value is not null.

In either case (using dropna or the above selection pattern), you will end up with a silent copy of parts of your original DataFrame. If you try to modify the result, you will receive a warning such as:

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

so you might want to suppress the warnings or do an explicit copy()of the result so your intention will be clear.

user_21164860
Related