Divide a big numpy matrix into 10 independent submatrices in a stratified manner

user_2163392 0
Viewed 461

I have two numpy variables with 32000 entries that are as follows:

>>> files
array(['GAN_0.npy', 'GAN_1.npy', 'GAN_10.npy', ..., 'GAN_822.npy',
       'GAN_8220.npy', 'GAN_8221.npy'], dtype='<U13')
>>> files.shape
(32000,)

>>> labels
array([1, 1, 1, ..., 1, 1, 1])
>>> np.unique(labels)
array([0, 1])
>>> labels.shape
(32000,)

In other words, the first variable is a NumPy matrix of strings, while the other is a NumPy matrix of integers. In the first matrix, I have a string list of images names and, in the other matrix, integers as the labels that I am using to identify them (images that start with names GAN_ are 1, whereas images with names RAW_ are 0).

I would like to know if I can split these 32000 entries into 10 subsets of 3200 images each without repetition, being 1600 entries which names start with RAW_ and other 1600 which names start with GAN_. The images' names and labels are shuffled, therefore I cannot simply split the big matrix into 10 submatrices. Therefore, is there any method in Numpy that could create 10 stratified and independent submatrices of samples from 2 classes in Python?

P.s: these two files can be found here

matrix numpy python
3 Answers

Here is a possible solution:

arrays = [np.concatenate((g, r))
          for g, r in zip(np.array_split(files[labels==1], 10), 
                          np.array_split(files[labels==0], 10))]

This solution keeps the relative order of 'GAN*' and 'RAW*' files. Also, arrays are created such that the initial positions are filled with 'GAN*' files, the remaining with 'RAW*' files. if you're not happy with this sorting you can always shuffle each array after you created them.

user_52961060

Here is a solution without loops (suggested by @Crazy Coder in other answer's comments):

labels = np.array(labels, dtype=bool)
np.split(np.vstack((files[labels],files[~labels])).T.reshape(-1,1), 10)
user_49759810

My own solution (based on @CrazyCoder suggestion)

import numpy as np

#Read file names
file_names=np.genfromtxt('test_files.txt',dtype='str')
   
raw_vector=[]
gan_vector=[]

for i in range(0,file_names.shape[0]):

    image_name=file_names[i]

    #Separate RAW and GAN files
    if("RAW_" in image_name):
        raw_vector.append(image_name)

    if("GAN_" in image_name):
        gan_vector.append(image_name)
    
raw_vector=np.array(raw_vector)
gan_vector=np.array(gan_vector)

#Split into 10 subsets each
raw_vector_divided=np.split(raw_vector,10)
gan_vector_divided=np.split(gan_vector,10)

for j in range(0,10):
    x=raw_vector_divided[j]
    y=gan_vector_divided[j]
    
    x=x.reshape(x.shape[0],1)
    y=y.reshape(y.shape[0],1)
    
    #merge
    experiment_data=np.vstack((x,y))
    
    #Save subset as file
    np.savetxt( 'experiment-' + str(j+1) + '-data.txt',  experiment_data, fmt='%s')

print("finished")
user_21633920
Related