How did the number of elements in each array end up being less than they were in their parent array (array of arrays)

user_14533818 0
Viewed 62

This problems seems to go away when I dont use shuffle(train_data). I also tried doing shuffle(train_data_1) and shuffle(train_data_2) seperately but in that case too, the distribution seems to change.

import numpy as np
import pandas as pd
from collections import Counter
from random import shuffle

train_data_1 = np.load('training_data_v1.npy', allow_pickle=True)
train_data_2 = np.load('training_data_v2.npy', allow_pickle=True)

train_data = np.vstack((train_data_1, train_data_2))

df = pd.DataFrame(train_data)
print(df.head())
print(Counter(df[1].apply(str)))

Output :
                                                 0          1
0  [[33, 255, 255, 255, 255, 255, 255, 255, 255, ...  [0, 1, 0]
1  [[33, 255, 255, 255, 255, 255, 255, 255, 255, ...  [0, 1, 0]
2  [[33, 255, 255, 255, 255, 255, 255, 255, 255, ...  [0, 1, 0]
3  [[33, 255, 255, 255, 255, 255, 255, 255, 255, ...  [0, 1, 0]
4  [[33, 255, 255, 255, 255, 255, 255, 255, 255, ...  [0, 1, 0]
Counter({'[0, 1, 0]': 6064, '[1, 0, 0]': 542, '[0, 0, 1]': 394})

Seperating the forwards, lefts and rights into their respective arrays

shuffle(train_data)

lefts = []
rights = []
forwards = []

for data in train_data:
    img = data[0]
    choice = data[1]

    if choice == [1,0,0]:
        lefts.append([img,choice])
    elif choice == [0,1,0]:
        forwards.append([img,choice])
    elif choice == [0,0,1]:
        rights.append([img,choice])
    else:
        print('no matches')

print (len(forwards), len(lefts), len(rights))

Output:
6086 763 151

At start, the distribution of forward, left, right was (6064, 542, 394)

Now, it is (6096, 763, 151)

How has the distribution changed??

What am I doing wrong!?

pandas dataframe numpy arrays python
1 Answers

Dont use shuffle(train_data) instead after the formation of forwards, lefts and rights array, use shuffle() on them individually.

shuffle(forwards)
shuffle(lefts)
shuffle(rights)

Now there isn't any problem with the distribution and shuffling is also performed correctly.

user_145338180
Related