I am calling an External API from my Laravel App via queues, but for some reason, when I dispatch the calls from the queue, the API call takes significantly longer than when called outside of a queue.
I am using php8.1 and Laravel 8.
Here's a class I have that makes the call to the external API:
use Illuminate\Support\Facades\Http;
class ProcessApiData
{
public function run(App\Models\User $user)
{
info("{$user->id} -------> STARTED PROCESSING AT:" . now());
try{
$responses = Http::pool(fn(Pool $pool) => [
$pool->as('requestOne')
->acceptJson()
->timeout(90)
->post('https://requestOne.com', ['username' => $user->username]),
$pool->as('requestTwo')
->acceptJson()
->timeout(90)
->post('https://requestTwo.com', ['username' => $user->username])
]);
info("REQUEST ONE TOTAL TIME: " . $responses['requestOne']->handlerStats()['total_time'])
info("REQUEST TWO TOTAL TIME: " . $responses['requestTwo']->handlerStats()['total_time'])
return [
'reqOne' => $responses['requestOne']->json(),
'reqTwo' => $responses['requestTwo']->json()
];
} catch(\Exception $e){
return $e->getMessage();
}
}
}
Then in my Job class, I simply call this API caller class:
class RunUserJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public function __construct(public User $user)
{
$this->onQueue(QueueNames::USER_QUEUE);
}
public function __invoke()
{
// call the API class
$result = App\Actions\ProcessApiData::run($this->user);
$this->user->update(['processed' => true]);
}
public function failed(\Exception $e)
{
$this->user->updated(['processed' => false]);
}
}
Then I dispatch the jobs as follows:
foreach(App\Models\User::all() as $user){
RunUserJob::dispatch($user);
}
When I dispatch this job, say for 10 users, I can get an average processing time of 10 seconds for the first 7 and then the last 3 take about 3 minutes to run and eventually timeout. I am running many workers and I can see that all the jobs called the ProcessApiData action about 1 second after they are queued (because I have info("{$user->id} -------> STARTED PROCESSING AT:" . now()); in the ProcessApiData, so there is no delay in the workers processing the jobs.
However, when I execute the following in tinker, I get the responses for all the API calls all in less than 10 seconds:
foreach(App\Models\User::all() as $user){
ProcessApiData::run($user);
}
Is there anything that calling the ProcessApiData class in queued jobs would make the call take longer than when called directly without a queue?
NB: I have stress-tested the external API using JMeter and there are no issues with even 100 concurrent requests.