Is there a way to create my own dataset in tensorflow python neural networks?

user_18312234 0
Viewed 67

I am trying to figure out how to train my neural network based off of what I would like to it to do. Currently I am using the mnist.npz dataset, which trains the neural network with recognizing digits. I would I to switch to my own dataset, but cant fiqure out how. I tried different method, but it none of them worked for myself. my training preperation code currently looks like this:

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras import backend as K
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def show_min_max(array, i):
  random_image = array[i]
  print(random_image.min(), random_image.max())

def plot_image(array, i, labels):
  plt.imshow(np.squeeze(array[i]))
  plt.title(" Class " + str(labels[i]))
  plt.xticks([])
  plt.yticks([])
  plt.show()

img_rows, img_cols = 28, 28  
num_classes = 10 

(train_images, train_labels), (test_images, test_labels) = mnist.load_data(path="mnist.npz")  #training data
(train_images_backup, train_labels_backup), (test_images_backup, test_labels_backup) = mnist.load_data(path="mnist.npz") #untrained backup data

print(train_images.shape) 
print(test_images.shape) 

train_images = train_images.reshape(train_images.shape[0],  img_rows, img_cols, 1) 
test_images = test_images.reshape(test_images.shape[0], img_rows, img_cols, 1) 
input_shape = (img_rows, img_cols, 1) 

plot_image(train_images, 1, train_labels)
show_min_max(train_images, 100) 

train_images[3000:]=255-train_images[3000:] 

train_images = train_images.astype('float32') 
test_images = test_images.astype('float32') 

train_images /= 255 
test_images /= 255 

plot_image(train_images, 100, train_labels) 
show_min_max(train_images, 100) 

train_labels = keras.utils.to_categorical(train_labels, num_classes) 
test_labels = keras.utils.to_categorical(test_labels, num_classes)

Things I Did:

  1. Searched through code of tensorflow and mnist database, and which up words to redirect it to mine, but I need my dataset (a folder) to be a npz file.
  2. I tried to use the function image_dataset_from_directory() but it didn't return what I expected / needed.
  3. turned all of my images into npz files, yet that still didn't matter.

those are just 3 of the many things I've tried. It would be amazing if I could get help with either turning my dataset into one npz file, or just to just restart from the basics (from the code up above(it is the base code with non of my chnages))

mnist.load_data() is loading is a npz file which contains 4 data values, x_train, y_train, x_test, y_test. I for 1 do not know the significance of each of these items, aswell as the fact that I dont know how to turn my dataset (a folder) into an npz file. I was following a tutorial during the making of this so therefore I have the backup variables lol.

this is my data currently, its not very big lol. just waiting until I know I can actually use it https://drive.google.com/drive/folders/1Cvs7ie6232BLI9ZicaUJdyhxUu-JB4kV?usp=sharing

keras tensorflow neural-network numpy python
1 Answers

Making your own dataset can be as simple as:

  1. Iterate through every picture in a folder and appending to a list, making sure all images are the same shape. They should probably have their class as part of the name for ease (class0.png, class1.png). This assumes you're working with color images. If they're grayscale, use cv2.IMREAD_GRAYSCALE
path = "folder/path/to/images"
images = []
labels = []
for i in os.path(path):
    label = int(i[6])
    i = os.path.join(path, i)
    i = cv2.imread(i, cv2.IMREAD_COLOR)
    # i = cv2.imread(i, cv2.IMREAD_GRAYSCALE)
    cv2.resize(i, (desired_shape))
    images.append(i)
    labels.append(label)
  1. Convert lists to arrays
images = np.asarray(images)
labels = np.asarray(labels)
  1. Then save them to disk or you could add them to an array so you'd have a single dataset instead of one with images and one with labels
arr_path = "some/path"
np.save("images.npy", images)
np.save("labels.npy", labels)

# or optionally
np.savez("dataset.npz", images, labels)

Just make sure you have enough memory. You can then load them using np.load() and use like normal.

mnist is already split into train and test sets, that's why you get four total sets, two for training (images and labels) and two for testing (images and labels). You can do the same using scikit-learn's train_test_split.

user_16765890
Related