How to copy list of files in parallel to list of destinations? Python

user_913098 0
Viewed 215

I have src_list and dst_list, two lists of the same length.

src_list contains paths to existing files.

dst_list contains paths to maybe existing files to maybe overwrite (not folders!).

src_list[i] should correspond to dst_list[i].

I want to copy every src_list[i] to dst_list[i], in (multiprocessing, not threading) parallel.

I want the call to be blocking, meaning all processes should be joined before moving on.

Examples of lists:

src_list = [

r"/data/one/f1.txt",
r"/data/one/f2.txt",
r"/data/two/f3.txt",
r"/data/three/f4.txt",
r"/data/four/five/f5.txt",
]

dst_list = [

r"/out1/f1.txt",
r"/out2/two/three/pook.txt",
r"/data/foo/bar/baz/f333.txt",
r"/kiko/f4",
r"/data/four/five/f5.txt",
]

really, just just paths to files. Please assume the lists are ~1000-5000 elements long.

What I seem to lack is the correct API call to parallelize over two iterables.

multiprocessing python
3 Answers
from concurrent.futures import ProcessPoolExecutor
from shutil import copyfile

with ProcessPoolExecutor() as executor:
    executor.map(copyfile, src_list, dst_list)

If you want to limit the number of processes that are working on this you can pass a max_workers argument inside the ProcessPoolExecutor() method.

user_146195010

you can try something like that:

from multiprocessing import Process
import shutil

def parallel_copy(src_lst, dst_list):
    if not src_lst or not dst_list or len(src_lst) != len(dst_list):
        raise ValueError('Cannot process inputs.')

    processes = [Process(target=shutil.copyfile, args=(src, dst)) for src, dst in zip(src_lst, dst_list)]
    [p.start() for p in processes]
    [p.join() for p in processes]
user_140704140

Full answer, based on @tibipin's. This one also creates the dst folder structure if needed.

from concurrent.futures import ProcessPoolExecutor
import shutil 
from typing import Iterable


def copy_single_file_create_dirs(src, dst):
    os.makedirs(os.path.dirname(dst), exist_ok=True)
    shutil.copyfile(src, dst)


def copy_src_to_dst_overwrite_parallel(
        existing_src_files: Iterable[str],
        dst_files: Iterable[str],
        max_workers=None,
):
    with ProcessPoolExecutor(max_workers=max_workers) as executor:
        executor.map(copy_single_file_create_dirs, existing_src_files, dst_files)

The copy_single_file_create_dirs function can't be defined as an inner function because then it needs to be pickled which doesn't work. This is the cleanest I could get.

user_9130980
Related