Let us say we have some Relational data. Making a simple example for a retail store chain:
- Dataset 1 --> Store_id, Daily_sales
- Dataset 2 --> Customer_id, store_id, Time in, Time out
Let us say the task is to predict Daily_sales.
I know how how to create data batches for one single CSV. I can use tf.data.experimental.make_csv_dataset and iterate over the dataset iterable that it returns to read the batches lazily.
However, I want to read in the batches from Dataset 1 and Dataset 2 described above where the common id is store_id such that the batch reads the rows with same store_ids from both the datasets. I want to do this because I will run two networks (RNN on Dataset 2 and a single Fully connected layer on Dataset 1) on both datasets and then merge them in the final fully connected layer.
Can you please guide me on how to approach this problem in scenarios where:
- The datasets can fit into the memory
- The datasets can not fit into the memory
Here is a concrete example of the consistent batch creation I am looking for:
import pandas as pd
Dataset_1 = pd.DataFrame({'id':['a','b','c','d'],'col1':[1,2,3,4]})
print(Dataset_1)
id col1
0 a 1
1 b 2
2 c 3
3 d 4
Dataset_2 = pd.DataFrame({'id':['a','a','b','c','c','c','d'],'col1':[10,11,12,13,14,15,16]})
print(Dataset_2)
id col1
0 a 10
1 a 11
2 b 12
3 c 13
4 c 14
5 c 15
6 d 16
#Let us say i want to create 2 batches. The following dataframes are how i want my batches to look like
batch_1 = (pd.DataFrame({'id':['a','b'],'col1':[1,2]}),pd.DataFrame({'id':['a','a','b'],'col1':[10,11,12]}))
print(batch_1[0])
id col1
0 a 1
1 b 2
print(batch_1[1])
id col1
0 a 10
1 a 11
2 b 12
batch_2 = (pd.DataFrame({'id':['c','d'],'col1':[3,4]}),pd.DataFrame({'id':['c','c','c','d'],'col1':[13,14,15,16]}))
print(batch_2[0])
id col1
0 c 3
1 d 4
print(batch_2[1])
id col1
0 c 13
1 c 14
2 c 15
3 d 16