I've been playing around with sklearn's train_test_split function to provide values for the various variables: X_train, X_test, y_train and y_test.
I understand this is useful when you have one big dataset and want to split it accordingly but say I have a separate test directory and training directory. How would I go about loading those to the various variables mentioned? Would it be just like any other directory with images?
Purpose is that I want to test my model on specific sets of data that I have gathered.
For example:
classes = ["train_fire", "smoke_small", "train_nosmokefire"] # names of the classes
DIR = ("../Code & Dataset") # datapath Directory
new_size = 200 # setting the image shape
training_data = []
for classType in classes:
path = os.path.join(DIR,classType) # joins the directory with each class's sub-folder
class_num = classes.index(classType) # determines the number of classes
for image in os.listdir(path):
try:
img_arr = cv2.imread(os.path.join(path,image), cv2.IMREAD_GRAYSCALE) # reads in an image as a grayscale
new_array = cv2.resize(img_arr, (new_size, new_size)) # make all images the same size (200x200)
training_data.append([new_array, class_num]) # append image to array of training data
except Exception as e:
pass # if the image cannot be found or read throw an error
I wasn't sure how much code I should show to get a good understanding of how I am currently going about loading in the dataset as a dataframe so I've shown the rest of the code up until the splitting of the dataset.
random.shuffle(training_data) # shuffle data
X = []
y = []
for features, label in training_data:
X.append(features) # pixels
y.append(label) # class labels such as fire, no fire and smoke
X = np.array(X).reshape(-1, new_size, new_size, 1)
y = np.array(y).reshape(-1,1)
num_fire = 0
num_smoke = 0
num_neither = 0
for img, classType in training_data:
if classType == 0:
num_fire += 1
if classType == 1:
num_smoke += 1
if classType == 2:
num_neither += 1
data = pd.DataFrame({'Class Type':["Fire","Smoke","NoSmokeNoFire"],
'Count Pics':[num_fire, num_smoke , num_neither]}) # table containing data information