How to load a Test Dataset and Train Dataset without using the sklearn test_train_split

Viewed 19

I've been playing around with sklearn's train_test_split function to provide values for the various variables: X_train, X_test, y_train and y_test.

I understand this is useful when you have one big dataset and want to split it accordingly but say I have a separate test directory and training directory. How would I go about loading those to the various variables mentioned? Would it be just like any other directory with images?

Purpose is that I want to test my model on specific sets of data that I have gathered.

For example:

classes = ["train_fire", "smoke_small", "train_nosmokefire"] # names of the classes
DIR = ("../Code & Dataset") # datapath Directory

new_size = 200 # setting the image shape
training_data = []

for classType in classes:
    path = os.path.join(DIR,classType) # joins the directory with each class's sub-folder 
    class_num = classes.index(classType) # determines the number of classes
    
    for image in os.listdir(path):
        try: 
            img_arr = cv2.imread(os.path.join(path,image), cv2.IMREAD_GRAYSCALE) # reads in an image as a grayscale
            new_array = cv2.resize(img_arr, (new_size, new_size)) # make all images the same size (200x200)
            training_data.append([new_array, class_num]) # append image to array of training data
            
        except Exception as e:
            pass # if the image cannot be found or read throw an error

I wasn't sure how much code I should show to get a good understanding of how I am currently going about loading in the dataset as a dataframe so I've shown the rest of the code up until the splitting of the dataset.

random.shuffle(training_data) # shuffle data

X = []
y = []

for features, label in training_data:
    X.append(features) # pixels 
    y.append(label) # class labels such as fire, no fire and smoke
    
X = np.array(X).reshape(-1, new_size, new_size, 1)
y = np.array(y).reshape(-1,1)

num_fire = 0
num_smoke = 0
num_neither = 0

for img, classType in training_data:
    if classType == 0:
        num_fire += 1
        
    if classType == 1:
        num_smoke += 1
        
    if classType == 2:
        num_neither += 1

data = pd.DataFrame({'Class Type':["Fire","Smoke","NoSmokeNoFire"],
                     'Count Pics':[num_fire, num_smoke , num_neither]}) # table containing data information
0 Answers
Related