How should the data folder be to take input as (x-train, y-train), (x-test, y-test) in a cnn model

user_14400294 0
Viewed 720

I am new to machine learning and deep learning. I have tried a multi-class classification model using CNN algorithm. I first tried it using the CIFAR-10 data set which is provided by the keras. In there, we give the input as follows to load the data set,

(x-train, y-train), (x-test, y-test) = tf.keras.datasets.cifar10.load_data()

This worked for me. However, I am now trying it with my manual data set instead of the built-in data set. I don't know how the data set folder should be, and even how to access it.

For now, I have a data set folder arrangement as follows,

Dataset=> Training_set => 10 different classes folders with corresponding images within
          Test_set => 10 different classes folders with corresponding images within

I have no idea how to use it in the code while calling the load_data. If I use it in the normal way, like, flow_from_directory('../Dataset/Training_set') I get the following error - Too many values to unpack(expected 2). Kindly help with this issue. It would be a great help for learning.

tf.keras conv-neural-network keras tensorflow python
1 Answers

You can try using tf.keras.utils.image_dataset_from_directory.

Create dummy data:

import os
import numpy
from PIL import Image

os.mkdir('Training_set')
for i in range(10):
  os.mkdir('Training_set/class{}'.format(i))

for i in range(10):
  for j in range(2):
    imarray = numpy.random.rand(100,100,3) * 255
    im = Image.fromarray(imarray.astype('uint8')).convert('RGB')
    im.save('Training_set/class{}/result_image{}.png'.format(i, j))

Folder structure:

- Training_set/
    - class9/
        - result_image1.png
        - result_image0.png
    - class8/
        - result_image1.png
        - result_image0.png
    - class7/
        - result_image1.png
        - result_image0.png
    - class0/
        - result_image1.png
        - result_image0.png
    - class2/
        - result_image1.png
        - result_image0.png
    - class5/
        - result_image1.png
        - result_image0.png
    - class4/
        - result_image1.png
        - result_image0.png
    - class3/
        - result_image1.png
        - result_image0.png
    - class1/
        - result_image1.png
        - result_image0.png
    - class6/
        - result_image1.png
        - result_image0.png

Load data with validation_split=0.2 (80% train data, 20% validation data):

import tensorflow as tf

train_ds = tf.keras.utils.image_dataset_from_directory(
  'Training_set',
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(100, 100),
  batch_size=2)

val_ds = tf.keras.utils.image_dataset_from_directory(
  'Training_set',
  validation_split=0.2,
  subset="validation",
  seed=123,
  image_size=(100, 100),
  batch_size=2)

for x, y in train_ds.take(1):
  print(x.shape, y.shape)

Found 20 files belonging to 10 classes.
Using 16 files for training.
Found 20 files belonging to 10 classes.
Using 4 files for validation.
(2, 100, 100, 3) (2,)

You can also choose if you want the labels to be sparse or categorical. See the docs for more information.

These datasets can be fed directly to your model like this:

epochs=10
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs)

And you can convert your tensorflow datasets to numpy arrays if you really want to:

x_train, y_train = np.concatenate(list(train_ds.map(lambda x, y: x))), np.concatenate(list(train_ds.map(lambda x, y: y)))
x_test, y_test = np.concatenate(list(val_ds.map(lambda x, y: x))), np.concatenate(list(val_ds.map(lambda x, y: y)))
user_96578610
Related