ValueError: codes need to be array-like integers when extracting X and Y from dataframes

Question

I'm trying to learn more about dataframes by using a reproducible examples of arrays. What i'm doing is trying to extract from my reproducible example the values X and y from my dataframe with my classes enumerated where first 5 rows are features from class A and last 5 rows are features from class B.

My actual dataframe is this:

[[ 31,  25,  17,  62,  26,  23, 193, 143,  37,  29, 220, 216, 175, 195, 207, 198, 190, 222,
  178, 214]
 [ 31,  26,  19,  59,  25,  23, 193, 140,  37,  29, 220, 216, 174, 195, 207, 198, 190, 220,
  178, 214]
 [ 31,  23,  17,  67,  23,  22, 195, 147,  38,  31, 222, 215, 182, 195, 213, 198, 185, 221,
  178, 207]
 [ 31,  23,  19,  67,  23,  23, 194, 144,  37,  31, 222, 218, 179, 198, 216, 198, 186, 221,
  179, 207]
 [ 31,  28,  17,  65,  23,  22, 193, 142,  36,  31, 222, 217, 177, 195, 216, 196, 182, 220,
  174, 207]
 [ 16,  24,  33,  43,  43,  58, 163,  76,  57, 105, 205, 200, 193, 188, 186, 193, 182, 227,
  193, 227]
 [  9,  13,  22,  36,  13,  49, 163,  39,  33, 105, 204, 200, 193, 191, 188, 193, 183, 224,
  194, 227]
 [ 23,  17,  10,  28,  21,  40, 166,  46,  28, 102, 208, 206, 196, 198, 195, 202, 190, 225,
  196, 229]
 [ 25,  19,  11,  30,  23,  39, 166,  46,  26,  99, 208, 206, 199, 196, 198, 201, 189, 227,
  198, 231]
 [ 25,  20,  12,  31,  25,  40, 169,  48,  27, 101, 211, 206, 198, 198, 196, 202, 190, 226,
  198, 229]]

pd.DataFrame(my_array)

Dataframe image

Therefore, what i want to do, is my grab my own dataframe, extract the X and y. Where my features are pixel intensities from 0 to 255 and each column is the pixel index from 1 to 20

My problem is that i fail to understand how to pass the values accordingly. This is how i tried:

indices = ['class_A, class_B']
target_names = ['class_A, class_B']
pixel_index = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
X = pd.DataFrame(my_array, columns= pixel_index)
y = pd.Categorical.from_codes(indices,target_names)

Where indices is supposed to be the classes for each pixel_index, im not sure if its being done in the correct way

The error that is given:

ValueError: codes need to be array-like integers

What i want to achieve is that when i do this:

df = X.join(pd.Series(y, name='class'))

I get a new dataframe with classes included to my previous dataframe.

When i change the indices to integers and correct the strings parameters i get this:

17   18   19   20    class  
0  190  222  178  214  class_A  
1  190  220  178  214  class_B  
2  185  221  178  207      NaN  
3  186  221  179  207      NaN  
4  182  220  174  207      NaN  
5  182  227  193  227      NaN  
6  183  224  194  227      NaN  
7  190  225  196  229      NaN  
8  189  227  198  231      NaN  
9  190  226  198  229      NaN

However, each 5 rows is stacked by a different class, for instance the first 5 rows should be class A and last 5 rows should be class B.

what i am trying to achieve when i do df.head for instance would be this:

   17   18   19   20    class  
    0  190  222  178  214  class_A  
    1  190  220  178  214  class_A  
    2  185  221  178  207  class_A   
    3  186  221  179  207  class_A   
    4  182  220  174  207  class_A   
    5  182  227  193  227  class_B 
    6  183  224  194  227  class_B  
    7  190  225  196  229  class_B  
    8  189  227  198  231  class_B  
    9  190  226  198  229  class_B

user_7326981 · Answer

Orignal Answer:

First you should refer to this documentation.

According to the documentation class_A can be called 0 and class_B can be called 1 and therefore indices = [0,1] which will resolve the Value Error

Further target_names are not correctly defined instead of

target_names = ['class_A, class_B'] it should be

target_names = ['class_A', 'class_B']

I believe the 2nd part of your question where you want to join each row with a class, the length of indices should be equal to number of observations i.e rows.

E.g. code with 2 rows.

my_array = np.array([[ 31,  25,  17,  62,  26,  23, 193, 143,  37,  29, 220, 216, 175, 195, 207, 198, 190, 222,
                     178, 214], [ 31,  26,  19,  59,  25,  23, 193, 140,  37,  29, 220, 216, 174, 195, 207, 198, 190, 220,
                        178, 214]])

indices = [0,1]
target_names = ['class_A', 'class_B']
pixel_index = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
X = pd.DataFrame(my_array, columns= pixel_index)
y = pd.Categorical.from_codes(indices,target_names)

print(X.join(pd.Series(y,name='class')))

Edit:

As asked in question that there are 10 rows in dataset first 5 represent class A while the other remaining 5 represent class B.

Assuming there are only 10 rows you can change indices in code above by

indices = [0,0,0,0,0,1,1,1,1,1]

This would give desired results as stated in question.

Let's assume the rows would change by a multiple of 5. E.g. if the rows increased from 10 to 15. Then you would require indices and class to change dynamically.

I've created dummy data below and added code that would be dynamic.

import string
import numpy as np
import pandas as pd                         

#random array generated with 15 rows (experiment by changing the rows)
my_array = np.random.rand(15,20)

#creating indices based on numbers of rows assuming the indice changes after every 5 rows
indices = sorted(list(range(0,int(my_array.shape[0]/5)))*5)

#creating a dict to assign each indice a class 
class_dict = dict(zip(range(0,int((my_array.shape[0]/5))), string.ascii_uppercase ))
target_names = ["Class_" + c for c in class_dict.values()]

pixel_index = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]
X = pd.DataFrame(my_array, columns= pixel_index)
y = pd.Categorical.from_codes(indices,target_names)
print(X.join(pd.Series(y,name='class')))

ValueError: codes need to be array-like integers when extracting X and Y from dataframes

1 Answers