What would be the best approach in python to check the existence of many millions of file paths, identifying paths that do not exist? I am currently using a single thread, something like:
paths_not_existing = set()
all_file_paths = [long_list]
for path_name in all_file_paths:
if os.path.isfile(path_name) == False:
paths_not_existing.add(path_name)
Is it likely that multi-threading will give any speed up? Specifically, since I assume that this is I/O bound, I was wondering if there a way to look up multiple paths at the same time?
(in case of reference, the hard-drive that i am using is not solid state).