I've got a set of videos from which I'm trying to pull random frames. There are a lot of videos, so I want to work in parallel, and ffmpeg can splice out the frames for me, so here's the important part of the code:
import os
from tqdm import tqdm
from joblib import Parallel, delayed
from multiprocessing import current_process
from subprocess import Popen
vids_dir = 'just a string'
out_dir = 'another string'
def process_each(f):
temp = 'temp' + str(current_process()._identity[0])
os.mkdir(temp)
proc = Popen(['ffmpeg -i ' + vids_dir + '/' + f + ' ' + temp + '/' + f[:-4] + '_%03d.jpg &> /dev/null'], shell=True) # convert to frames
proc.wait()
# do stuff
os.system('rm -rf ' + temp) # clean up
Parallel(n_jobs=10)(delayed(process_each)(f) for f in tqdm(os.listdir(vids_dir)))
I can print out the command being passed to Popen and execute it in a shell, and it works. I can open a python3 session and call the command from Popen or subprocess.call or even os.system, and it works. I can even set n_jobs=1 in my Parallel, and it works.
But the moment I actually parallelize this, I find ffmpeg doesn't flush its full results to the temporary folders; it only gets the first one or few frames.
What on earth could be going on? subprocess and multiprocessing should be able to mix this way.