How to train a tensorflow model to accurately classify open eyes and closed eyes?

Question

I am trying to train a tensorflow model to classify open eyes and closed eyes for my school project.

I used OpenCV library to extract my eyes from the video recorded by webcam. I have approximately 4000 images with open eyes and 4000 images of closed eyes. half of them are left eyes and half of them are right eyes.

Dataset Looks Like This:

Open Eyes:

Closed Eyes:

I have 8000 images like this with different light conditions and angles.

MY CODE:

I used this code to import datasets as arrays.

import numpy as np
import cv2
import os
from random import shuffle
from tqdm import tqdm
import matplotlib.pyplot as plt
Training_Data = []
Eyes_Open_Data = []
Eyes_Closed_Data=[]
Labels_Data=[]
only_eyes_open = 'C:/Users/Ibrahim/Desktop/Staj 2020/only_eyes_open'
only_eyes_closed = 'C:/Users/Ibrahim/Desktop/Staj 2020/only_eyes_closed'


for item in os.listdir('only_eyes_open'):
    img_array= cv2.imread(os.path.join(only_eyes_open, item), cv2.IMREAD_GRAYSCALE)
    new_array = cv2.resize(img_array, (50,50))
    Eyes_Open_Data.append(new_array)
    Labels_Data.append(1)


for item in os.listdir('only_eyes_closed'):
    img_array= cv2.imread(os.path.join(only_eyes_closed, item), cv2.IMREAD_GRAYSCALE)
    new_array = cv2.resize(img_array, (50,50))
    Eyes_Closed_Data.append(new_array)
    Labels_Data.append(0)

I used this code to merge the left eye and right eye data as well as the labels. Then I shuffled it and separated it as training data and labels.

Training_Data = Eyes_Open_Data + Eyes_Closed_Data

Training_Data = list(zip(Training_Data, Labels_Data))


shuffle(Training_Data)

Labels_Data = [b for a,b in Training_Data]

Training_Data = [a for a,b in Training_Data]

X = []
y = []

X = Training_Data
y = Labels_Data

X = np.array(X).reshape(-1, 50, 50, 1)

I created my model with the following code:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Activation, Flatten, Conv2D, MaxPooling2D


X = np.array(X/255.0)
y= np.array(y)

model = Sequential()

model.add(Conv2D(32, (3, 3), input_shape=X.shape[1:]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())  # this converts our 3D feature maps to 1D feature vectors

model.add(Dense(128))

model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X, y, batch_size=32, epochs=1, verbose=1, validation_split=0.1, shuffle=True)

Output of the code above:

226/226 [==============================] - 11s 47ms/step - loss: 0.0908 - accuracy: 0.9582 - val_loss: 0.0028 - val_accuracy: 1.0000

Everything seems okay so far until I actually test the model with different data. I captured four more images of my eyes just like the images in my dataset. First two of them are open eyes and the last two of them are closed eyes.

So I am expecting the prediction to be like: [1,1,0,0]

I used this code to import the testing data:

test_data = 'C:/Users/Ibrahim/Desktop/Staj 2020/testing'

test_array = []
for item in os.listdir(test_data):
    img_array = cv2.imread(os.path.join(test_data,item), cv2.IMREAD_GRAYSCALE)
    new_array = cv2.resize(img_array,(50,50))
    plt.imshow(new_array)
    test_array.append(new_array)

test_array = np.array(test_array).reshape(-1,50,50,1)

I ran this code to predict:

model.predict(test_array)

Here is the output:

array([[1.],
       [1.],
       [1.],
       [1.]], dtype=float32)

If I am not mistaken, this means that it predicted all of them as open eyes. So I decided to try the prediction with the images that I used in training.

I changed the test data location to my dataset location:

test_data = 'C:/Users/Ibrahim/Desktop/Staj 2020/only_eyes_closed'

Output is correct:

array([[0.],
       [0.],
       [0.],
       ...,
       [0.],
       [0.],
       [0.]], dtype=float32)

After this point, no matter what data I use for testing, it only predicts 1. Only exception is the data I used for training.

user_7018323 · Answer

Okay, possible scenarios to look into:

Train for multiple epochs.
Tune your hyperparameters use decaying learning rate function.
Try tf.keras.layers.BatchNormalization and tf.keras.layers.Dropout.
Look into tf.keras.preprocessing.image.ImageDataGenerator for Image augmentation techniques.
If the problem, still persists, try giving closed eyes data more weightage.
Fine-tuning a predefined architecture (VGG19/EfficientNet) i.e. keep its starting layers frozen (as they capture basic features).
As this is a binary classification you should look into other metrics like ROC/Precision/Recall etc.
At least give a proper look into your real-world dataset and training dataset and see how much they differ.

user_12505894 · Answer

I solved the problem. The issue was that when I took pictures of my closed eyes, OpenCV would fail to detect my eyes and rewrite the previous image. This created a huge imbalance in my dataset. So I removed the duplicate images from closed eyes folder and removed same amount of random images from open eyes folder. Also I changed my model a bit.

Here is my new model:

lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_steps=10000,
    decay_rate=0.9)
optimizer = tf.keras.optimizers.SGD(learning_rate=lr_schedule)
X = np.array(X/255.0)
y= np.array(y)

model = Sequential()

model.add(Flatten()) 

model.add(Dense(16))
model.add(Activation('relu'))

model.add(Dense(16))
model.add(Activation('relu'))

model.add(Dense(1))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

model.fit(X, y, batch_size=1, epochs=4, verbose=1, validation_split=0.1, shuffle=True)

When I tried to predict a testing set with 6 open and 6 closed eyes:

array([[1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [1.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.],
       [0.]], dtype=float32)

Output is 100% correct.

How to train a tensorflow model to accurately classify open eyes and closed eyes?

2 Answers