Pardon me if use the terms in the wrong sense. I am still grappling with many spark and distributed related things.
Here is my use case and I am not able to get a complete picture of the implementation.
I have time-series data of 40 columns and 100 timesteps saved in parquet format.
I learned that to do distributed training on big data we can use petastorm for data injection and Horovod for training. But it is unclear to me how the data needs to be partitioned (one partition per ID? what row groups are?) and how to convert the data to sequences that LSTM expects?
Any pointers in this direction will be of great help. Thanks!