How to improve the performance when no shuffling of dataloader is needed?

I'm currently doing some researches on video recognition. What I'm trying to do is like this paper.
Because it needs to sequentially read consecutive video frames (section 4.3), the dataloader will output the data like this: (the number is timestamp, and each row represents a batch)

[1,  9, 18, 27, 36]
[2, 10, 19, 28, 37]
[3, 11, 20, 29, 38]
[4, 12, 21, 30, 39]
...

The thing is, due to the sequential reading, I couldn't do shuffling for the dataloader, which unsurprisingly results in a pretty bad performance.

For such streaming loading problem, are there any tips to improve the bad performance caused by no shuffling?

I have done the following experiments to verify that the problem is really caused by no shuffling:

1. original dataloader with shuffling: good accuracy
2. original dataloader without shuffling: worst accuracy
3. the dataloader I mentioned in the question: medium accuracy but still bad