I have a large list and I used to create one celery task with apply_async to process it. But I want to split it into multiple tasks with chunks. For example large_list = [1, 2, 3, 4, 5, 6, 7, 8]
So here is what I do:
I run process_function.chunks(zip(large_list), CHUNK_SIZE).apply_asnyc()
Then I checked the queue and I saw tit creates the tasks correctly. It creates more than one task and each task has a subset of the large_list item in it. For example if CHUNK_SIZE=2 it create subtask for [1, 2] & [2, 3], [4, 5], [6, 7], [8]
However, when it calls the process_function, it calls the function 8 times. every time with one item in the list. Isn't supposed to call the function with the subset created with chunk?
If not what is the benefit of using chunk?