I've written a Luigi pipeline to extract 1.2 mio files and then do some sed work on them - see https://gist.github.com/wkerzendorf/395c85a2955002412be302d708329f7f.
If I run this through Luigi on a few thousand files it schedules fine. But running this on the whole dataset it complains with Failed connecting to remote scheduler. Not sure if I do this the right way.