How to avoid many connections when using high number of queue workers?

user_1064428 0
Viewed 1128

Probably I'm missing something obvious but I had an architecture problem.

I'm building uptime monitoring service.

I have monitors table, each monitor represents a URL. My task it to send http request to the url. I send http requests from multiple servers. Each servers will handle like thousands of urls per minute.

So, in every server I have supervisord which spawns, say, 50-100 queue workers. Once per minute I push to the queue all monitors, and then these monitors are being checked by the queue workers.

However, each queue worker creates a connection to MySQL. I think this is because I pass Monitor model to the queue Job, and Laravel puts only monitor id into the queue. Then it retrieves whole monitor model from database when the job is being handled.

Or probably I'm mistaken and the reason is just because laravel connects by default when running any kind of code, including queued Jobs.

But 10 servers * 100 workers = 1000 connections, which is probably bad (I don't know for sure, but I assume that it's not scalable).

My queues based on local redis.

So I thought that it's reasonable to put serialized model data in redis. Then, the result of http requests I put in redis. Then push bunch of results at once in MySQL.

So how to achieve that?

laravel redis mysql
1 Answers

Before dispatching the jobs you fetch all the related models(in chunks) and dispatch the jobs with the model instance that you need in the job. So you won't need to fetch the model again from the database while processing the job.

One of the key thing is that, you shouldn't use SerializesModels trait inside your job class if you are going to send the model instance. As it is stated in the documentation

If your queued job accepts an Eloquent model in its constructor, only the identifier for the model will be serialized onto the queue. When the job is actually handled, the queue system will automatically re-retrieve the full model instance and its loaded relationships from the database.

If the handle method doesn't have any database related dependency, then it won't open a connection to your database. If you are using mysql you can then try SHOW PROCESSLIST to examine it. I checked on my local(with sleep), it doesn't show any mysql process.

user_21889220
Related