Is it a must to convert pandas dataframe to numpy arrays for machine learning algorithms in scikit-learn

user_1769197 0
Viewed 1373

Is it a must to convert pandas dataframe to numpy arrays for machine learning algorithms in scikit-learn ?

I know the to_numpy() function does the conversion. This would mean I have to manually create a dummy matrix for categorical columns in pandas dataframe too.

What happens if I just use pandas dataframe as input in scikit-learn ? And If I convert pandas dataframe to numpy arrays, then does it means my column names is no longer preserved in the machine learning algorithm ? When it comes to model diagnostics, extra steps need to be taken to reconcile the column names with numpy arrays?

scikit-learn pandas numpy python
1 Answers

Supplying an array of floats is a safe bet, but it's not a must. Whatever you supply will be attempted to convert to numpy array internally. If it's not an array-like (see below) an exception will be raised.

If you take RandomForestRegressor as an example, you'll find out in sklearn they have a notion of an array-like. See for example docstring for RandomForestRegressor.fit():

X{array-like, sparse matrix} of shape (n_samples, n_features) The training input samples. Internally, its dtype will be converted to dtype=np.float32. If a sparse matrix is provided, it will be converted into a sparse csc_matrix.

You can gain a further insight of what is an array-like by reading glossary:

array-like
The most common data format for input to Scikit-learn estimators and functions, array-like is any type object for which numpy.asarray will produce an array of appropriate shape (usually 1 or 2-dimensional) of appropriate dtype (usually numeric).

This includes:

  • a numpy array

  • a list of numbers

  • a list of length-k lists of numbers for some fixed length k

  • a pandas.DataFrame with all columns numeric

  • a numeric pandas.Series

It excludes:

  • a sparse matrix

  • an iterator

  • a generator

If you go through the source, you'll found out that the data you supply to your methods will flow through self._validate_data that will make conversion for you.

You can always check beforehand if your data is acceptable by sklearn.utils.check_array, but it doesn't make too much practical sense because it will be done for you anyways when you supply your data to a method.

user_43170580
Related