Gmail SMTP Connection "unexpectedly closed" after sending some mails w/ multithreading

dizzydizzy 0
Viewed 21

I'm trying to use a multithreading approach to speed up sending multiple individual emails using the Django framework and the Gmail SMTP, but I'm running into issues with my connection being dropped

It is important to know that I can manage to send about 50 or 55 emails untill my connection is dropped with this error:

smtplib.SMTPServerDisconnected: Connection unexpectedly closed

I can't use Django send_mass_email method because I need to be able to register if each mail is sent or failed in a database for auditing purposes

My code looks like this:

class Threading(Thread):
def __init__(self, emails):
    self.emails = emails
    Threading.thread(self)

def thread(self):
    queue = Queue()

    #4 worker threads
    for x in range(4):

        worker = SenderWorker(queue)
        worker.daemon = True
        worker.start()

    
    for email in self.emails:
        queue.put( email )

    queue.join()

And the SenderWorker class:

class SenderWorker(Thread):

def __init__(self, queue):
    Thread.__init__(self)
    self.queue = queue

def run(self):
    start = datetime.now()
    while True:
        email = self.queue.get()
        try:
            Email.sendEmail( email ) #method for sending and logging a single email sent
        finally:
            self.queue.task_done()

My SMTP config looks like this:

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_USE_TLS = True
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 587
EMAIL_HOST_USER = 'anon@anonymous.com'
SERVER_EMAIL = 'anon@anonymous.com'
EMAIL_HOST_PASSWORD = '*******'
DEFAULT_FROM_EMAIL = EMAIL_HOST_USER

I've tried so far:

  • Changing number of threads
  • Switching from ports/authentication from SSL to TSL and viceversa
  • Sending less emails / trying in batches

I'm not able to find limits on Gmail's SMTP regarding emails sent per second, or connections stablished per second. Is there any workaround? I'd like to avoid adding sleep times after each mail. Thank you

multithreading smtp email django python
1 Answers

After playing around I finally found the answer, if anyone else needs it:

The connection was failing because I was opening and closing several of them per second. When I instantiate a single SMTP connection and pass that one so all emails are sent through that one, the problem goes away (I just need to deal with the other limits now)

Threading class looks like this now:

class Threading(Thread):
def __init__(self, emails):
    self.emails = emails
    Threading.thread(self)

def thread(self):
    queue = Queue()
    connection = mail.get_connection()
    connection.open()

    for x in range(4):

        worker = SenderWorker(queue)
        worker.daemon = True
        worker.start()

    for email in self.emails:
        email['connection'] = connection
        queue.put( email )

    queue.join()
dizzydizzy0
Related