Keras: verbose (value 1) in model.fit shows less training data

user_9837978 0
Viewed 1667

I'm currently using the latest version of Keras 2.4.2 and Tensorflow 2.2.0 to implement a simple matrix factorization model with Movielens-1M dataset (which contains 1 million rows). However, I noticed that the amount of training data is reduced while training.

from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
import keras

dataset = pd.read_csv('ratings.dat', skiprows=1,sep='::',names="userId,itemId,rating,timestamp".split(","))

dataset.userId = dataset.userId.astype('category').cat.codes.values
dataset.itemId = dataset.itemId.astype('category').cat.codes.values

train, test = train_test_split(dataset, test_size=0.2)

#model architecture can be found link below

x = [train.userId, train.itemId]
y = train.rating

print(x[0].shape, x[1].shape, y.shape)
history = model.fit(x, y, epochs=25,
                verbose=1, validation_split=0.25)

enter image description here

Click here to see model architecture.

As you can see, the number of training data should be 600124 (after validation split). But instead, the verbose progress bar shows a much lesser value (18754). What is happening here?

keras tensorflow python
1 Answers

Everything is as expected here. 18754 is not the number of training data. This is the number of steps to complete one epoch. The whole training data breaks into a number of groups and each group is called a batch. The default batch_size is 32. This means, your whole training data will be N number of groups where each group contains 32 training data.

So what will be the size of N?

Simple, number of steps (N) = total_training_data/batch_size.

Now you can calculate by yourself.

Btw, this batch is being used because your memory is limited and you can't load the whole training data into your GPU memory. You can change the batch size depending on your memory size.

user_30770370
Related