How to implement Multiprocessing in Azure Databricks - Python

user_11247054 0
Viewed 325

I need to get details of each file from a directory. It is taking longer time. I need to implement Multiprocessing so that it's execution can be completed early.

My code is like this:

from pathlib import Path
from os.path import getmtime, getsize
from multiprocessing import Pool, Process

def iterate_directories(root_dir):
  
  for child in Path(root_dir).iterdir():
    
    if child.is_file():
        modified_time = datetime.fromtimestamp(getmtime(file)).date()
        file_size = getsize(file)
         # further steps...
      
    else:
      iterate_directories(child) ## I need this to run on separate Process (in Parallel)

I tried to do recursive call using below, but it is not working. It comes out of loop immediately.

else:
    p = Process(target=iterate_directories, args=(child))
    Pros.append(p) # declared Pros as empty list.
    p.start()

for p in Pros:
  if not p.is_alive():
     p.join()

What am I missing here? How can I run for sub-directories in parallel.

azure-databricks python-multiprocessing python-asyncio python-multithreading python-3.x
2 Answers

You have to get the directories list first and then you have to use multiprocessing pool to call the function.

something like below.

from pathlib import Path
from os.path import getmtime, getsize
from multiprocessing import Pool, Process
Filedetails = ''

def iterate_directories(root_dir):
  
  for child in Path(root_dir).iterdir():
    
    if child.is_file():
        modified_time = datetime.fromtimestamp(getmtime(file)).date()
        file_size = getsize(file)
         Filedetails = Filedetails + '\n' + '{add file name details}' + modified_time + file_size
 else:
      iterate_directories(child) ## I need this to run on separate Process (in Parallel)

return Filesdetails #file return from that particular directory 

   pool = multiprocessing.Pool(processes={define how many processes you like to run in parallel})
    results = pool.map(iterate_directories, {explicit directory list })
    print(results) #entire collection will be printed here. it basically a list you can iterate individual directory level

.

pls let me know, how it goes.

user_28717230

The problem is this line:

if not p.is_alive():

What this translates to is that if the process is already complete, only then wait for it to complete, which obviously does not make much sense (you need to remove the not from the statement). Also, it is completely unnecessary as well. Calling .join does the same thing internally that p.is_alive does (except one blocks). So you can safely just do this:

for p in Pros:
    p.join()

The code will then wait for all child processes to finish.

user_163107410
Related