How to speed up several nested loops

user_16359302 0
Viewed 71

I have nested for loops which are causing the execution of my operation to be incredibly slow. I wanted to know if there is another way to do this.

The operation is basically going through files in 6 different directories and seeing if there is a file in each directory that is the same before opening each file up and then displaying them.

My code is:

original_images = os.listdir(original_folder)
ground_truth_images = os.listdir(ground_truth_folder)
randomforest_images = os.listdir(randomforest)
ilastik_images = os.listdir(ilastik)
kmeans_images = os.listdir(kmeans)
logreg_multi_images = os.listdir(logreg_multi)
random_forest_multi_images = os.listdir(randomforest_multi)


for x in original_images:
  for y in ground_truth_images:
    for z in randomforest_images:
      for i in ilastik_images:
        for j in kmeans_images:
          for t in logreg_multi_images:
            for w in random_forest_multi_images:
              if x == y == z == i == j == w == t:
                   *** rest of code operation ***
nested-loops python
3 Answers

If the condition is that the same file must be present in all seven directories to run the rest of the code operation, then it's not necessary to search for the same file in all directories. As soon as the file is not in one of the directories, you can forget about it and move to the next file. So you can build a for loop looping through the files in the first directory and then build a chain of nested if statements: If the file exists in the next directory, you move forward to the directory after that and search there. If it doesn't, you move back to the first directory and pick the next file in it.

user_181030120

Convert all of them to sets and iterate through the last one, checking membership for all of the others:

original_images = os.listdir(original_folder)
ground_truth_images = os.listdir(ground_truth_folder)
randomforest_images = os.listdir(randomforest)
ilastik_images = os.listdir(ilastik)
kmeans_images = os.listdir(kmeans)
logreg_multi_images = os.listdir(logreg_multi)

files = set()

# add folder contents to the set of all files here
for folder in [original_images, ground_truth_images, randomforest_images, ilastik_images, kmeans_images, logreg_multi_images]:
    files.update(folder)

random_forest_multi_images = set(os.listdir(randomforest_multi))

# find all common items between the sets
for file in random_forest_multi_images.intersection(files):
    # rest of code

The reason this works is that you are only interested in the intersection of all sets, so you only need to iterate over one set and check for membership in the rest

user_78679680

You should check x == y before going in the nest loop. Then y == z etc. Now you are going over each loop way too often.

There is also another approach:

You can create a set of all your images and create an intersection over each set so the only elements which will remain are the ones that are equal. If you are sure that the files are the same you can skip that step.

If x is in all other list you can create your paths on the go:

import pathlib
original_images = os.listdir(original_folder)
ground_truth_images = pathlib.Path(ground_truth_folder) #this is a folder
randomforest_images = pathlib.Path(randomforest)

for x in original_images:
    y = ground_truth_images / x
    i = randomforest_images / x
    # And so on for all your files

    # check if all files exist:
    for file in [x, y, i, j, t ,w]:
        if not file.exists():
            continue # go to next x
    # REST OF YOUR CODE USING x, y, i, j, t, w,
    # y, i, j, t, w, are now pathlib object, you can get s string (of its path using str(y), str(i) etc.
user_57753580
Related