Avoiding or syncing shuffling in make_tf_dataset

Viewed 20

I have a tensorflow model which is trained with a rating dataset like this:

User Movie
0 2
1 2
0 1

which means that user 0 rate movie 2, user 1 rate movie 0 and user 0 rate movie 1. In each batch the models needs the list of ratings, unique users and unique movies. Taking a batch of 2 elements, the rating would be:

User Movie
0 2
1 2
Unique user
0
1
Unique movie
2

I want to highlight with these example that each batch could have a different length of elements, that's why i tried the following.

I create a Spark DataFrame with the ratings dataset.

df = spark.read.parquet('s3a://path/to/dataset')

next, create a SparkDatasetConverter with the dataframe:

conv_train = make_spark_converter(df)

now with the idea of get 3 equals batches from the same converter and apply to each batch different transformations i used these code

with conv_train.make_tf_dataset(transform_spec=transformation0, shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as rating,  \
     conv_train.make_tf_dataset(transform_spec=transformation1, shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as unique_user, \
     conv_train.make_tf_dataset(transform_spec=transformation2, shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as unique_movie:
    hist = model.fit(rating, unique_user, uynique_movie)

The problem is that these code shuffle each batch independently even using the same seed. When i run the next code:

with conv_train.make_tf_dataset(shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as train0,  \
     conv_train.make_tf_dataset(shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as train1, \
     conv_train.make_tf_dataset(shuffling_queue_capacity=None, batch_size=2, num_epochs=1, seed=1) as train2:
   #Print train0
   #Print train1
   #Print train2

the elements of train0, train1 and train2 are not the same, which should not happen due im using the same seed.

How can I achive that train0, train1 and train2 be equals?

0 Answers
Related