I have a tensorflow model which is trained with a rating dataset like this:
| User | Movie |
|---|---|
| 0 | 2 |
| 1 | 2 |
| 0 | 1 |
which means that user 0 rate movie 2, user 1 rate movie 0 and user 0 rate movie 1. In each batch the models needs the list of ratings, unique users and unique movies. Taking a batch of 2 elements, the rating would be:
| User | Movie |
|---|---|
| 0 | 2 |
| 1 | 2 |
| Unique user |
|---|
| 0 |
| 1 |
| Unique movie |
|---|
| 2 |
I want to highlight with these example that each batch could have a different length of elements, that's why i tried the following.
I create a Spark DataFrame with the ratings dataset.
df = spark.read.parquet('s3a://path/to/dataset')
next, create a SparkDatasetConverter with the dataframe:
conv_train = make_spark_converter(df)
now with the idea of get 3 equals batches from the same converter and apply to each batch different transformations i used these code
with conv_train.make_tf_dataset(transform_spec=transformation0, shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as rating, \
conv_train.make_tf_dataset(transform_spec=transformation1, shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as unique_user, \
conv_train.make_tf_dataset(transform_spec=transformation2, shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as unique_movie:
hist = model.fit(rating, unique_user, uynique_movie)
The problem is that these code shuffle each batch independently even using the same seed. When i run the next code:
with conv_train.make_tf_dataset(shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as train0, \
conv_train.make_tf_dataset(shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as train1, \
conv_train.make_tf_dataset(shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as train2:
#Print train0
#Print train1
#Print train2
the elements of train0, train1 and train2 are not the same, which should not happen due im using the same seed.
How can I achive that train0, train1 and train2 be equals?