Parallel loading images in Julia

Question

I am trying to create mosaic images of simulation data where each tile is in .jpg format and has a fixed number of pixels. I combine hundreds of these into a larger image for easier parameter analysis. So far I was able to parallelize getting the tiles to create a larger image with the following code:

using Distributed
addprocs(8)
# add @everywhere macro before every function and variable so it magically works

@everywhere big_image = zeros(RGB, 300, 1000); #each tile has 100x100 pixels for simplicity
function createBigFrame(big_image)
    @sync for row = 1:10 # @sync waits until all the images are fetched and then continue with plotting
        for col = 1:3
            i = #range for x
            j = #range for y (or vice versa)
            image_path = #get the path
            Threads.@spawn big_image[i, j] = load(image_path);
        end
     end
     plot(...) # add axes and ticks to the image 
     savefig(...) # save the figure on the disk
end

Although I used this on small data, it gave me a 20% increase in performance. Higher performance will be seen with larger data since there are more tile images to parallel. However, it has been told me that this is not the proper way of parallelizing things. I am very curious to know the right way to load images in parallel (not concurrently since the load() function is not thread-safe) and improve the code and performance further. I am very grateful for your help.

EDIT: The following code is supposed to be a minimal working example, but since you don't have the files to load(), the situation is a little different.

using Distributed
addprocs(4) #do not re-run
@everywhere using Colors, Images
@everywhere img = Images.zeros(RGB, 300, 1000);

function createBigFrame(image)
    @sync for row = 1:10
        for col = 1:3
            j = 100*(col-1)+1:100*col;
            i = 100*(row-1)+1:100*row;
            Threads.@spawn image[j, i] .= RGB(rand(3)...)
        end
     end
    return image
end
createBigFrame(img)

user_10050565 · Answer

I managed to run my code as the second suggestion of the selected answer in here. However, I'm not sure I fully understand how the code works under the hood.

I'll put this example picture here so we are on the same page:

There are 3 rows and 10 columns of tiles in this image. Now, in my first attempt, I used the following way to get the images to row 1 in parallel.

arr = SharedArray{Float32,3}(ones(3, 512*3, 512*10))

@sync @distributed for row = 1:3
    for col = 1:10
        row_index = #index range for the height of the image
        col_index = #index range for the width of the image
        arr[:, row_index, col_index] = channelview(testimage("lake_color"))
    end
 end

If my understanding is correct, @sync is used to wait for each row iteration to complete while workers fetch images in that row in parallel. I thought if this is the case, there should be a waste of time between rows waiting for the previous row to complete. So I flattened the 2D fetching loop to 1D in order to have more space to move and fetch the images freely:

@sync @distributed for s = 1:30
    row_index = #index range for the height of the image
    col_index = #index range for the width of the image
    arr[:, row_index, col_index] = channelview(testimage("lake_color"))
end

But I was wrong? The second code came out 25% slower than the first one (150 seconds to 190 seconds), so I wonder what is going on here?

Parallel loading images in Julia

1 Answers