I'm trying to train a model that finds road lanes. I have 191070 training images and 191070 validation images. I'm trying my model on this dataset using PyTorch. But when I start the training process (5 epochs, batch_size=256, num_workers=4, pin_memory=True), it always crashes something like an hour later. Here is the recent error that I got:
PIL.UnidentifiedImageError: cannot identify image file '/content/drive/MyDrive/path_to_dataset/data/train/image-188463.jpg'
When I check the image, nothings wrong. I'm using Google Colab's Pro version. I have more powerful GPU's than standard Colab version. Why do I get this error while training my model? Is it because the dataset is too big? If so, what should I do to prevent this error? Also I'm getting images from Google Drive. I tried to download these files to Colab session but because the images are in a folder, I couldn't download the images.