"A Google Drive timeout has occoured" error while loading a folder with a large number of small files in Google Colab

user_9801811 0
Viewed 406

I'm working on a folder with 150 thousand files (~64 kb size each) stored on my Google Drive. In Google Colab, I'm using the following function to get the list of all files. Colab throws the drive timeout error when I ran the function on the folder with 150k files. I tried remounting the drive multiple times, restarting Colab, and using glob to obtain the list of files. None of these approaches worked for me.

Do you know how I can effectively work with 150k files?

def list_all_files_with_an_extension(pth_folder: str, file_extension: list = [".png"]) -> list:
"""
Get files from a folder with an extension
"""
   # Placeholder
   list_files = []
   # Walk through dir and sub dire
   for root, dirs, files in os.walk(pth_folder):
       for file in files:
           for extension in file_extension:
               if file.endswith(extension):
                   list_files.append(os.path.join(root, file))
   # Return the list of paths
   return list_files
google-colaboratory python
1 Answers

I solved this problem by trying several times to load the data.

list_of_files = []
while len(list_of_files)<1:
   list_of_files  = list_all_files_with_an_extension(pth_folder)

After several failed attempts, Colab will finally fetch all files from Google Drive.

user_98018110
Related