The most effective way to run parallel CLI processes from PHP application

user_2018578 0
Viewed 104

I'm trying to find the best way to run multiple ImageMagick CLI processes from the PHP code.

I'm using ImageMagick as a CLI utility because I have to do complex and intense image processing. It's more convenient than writing tons of corresponding PHP code using the Imagick extension.

The task is to take a huge GIF file containing many (50-100) frames and process all these frames as fast as possible. So, if we have a server with 16 CPU cores, we need to process 16 frames simultaneously.

Solutions I see:

  1. The most simple: to perform all processing using the PHP exec function with an ampersand at the end of the commands, e.g.
exec('convert frame1.gif ...some_params... frame1_output.gif &');
exec('convert frame2.gif ...some_params... frame2_output.gif &');
exec('convert frame3.gif ...some_params... frame3_output.gif &');
...

After this, we only have to wait for the required amount of output files in the output directory. Will this approach utilize CPU cores correctly?

  1. To use some sort of event-driven asynchronous processing PHP framework, e.g. Amp. Internally it uses libev and libevent. I'm not sure if it's a good idea to start CLI processes using an event loop.

  2. To use Parallel

  3. To use some sort of task queue with the producer and many consumers (workers). Each worker will be a separate PHP process. This approach leads to overhead with the message broker or some other kind of queue implementation, deploying, network delays while processing etc...

  4. To use some process manager e.g. PHP-PM or Supervisor just for spawning ImageMagick processes.

Which approach seems to be the most correct for you? What other ways are there to solve this problem?

parallel-processing image-processing imagemagick multithreading php
1 Answers

For tasks like that... if you don't have to use PHP i would go for NodeJS here. Simple async task would do that for you

user_48665290
Related