Passing flask.request to another thread

Question

I'm a bit confused about whether I can or cannot send Flask's request object to another thread for processing. I'm trying to store request related data in Elasticsearch using after_request. To handle the request as fast as possible, I'm trying to offload the processing to a separate thread:

from datetime import datetime
import socket
from threading import Thread

from elasticsearch import Elasticsearch
from flask import current_app, request

def after_request(response):
    Thread(
        target=_process_data, args=(
            current_app._get_current_object(),
            request._get_current_object()
        )
    ).start()

    return response

def _process_data(app, request):
    data = {
        'timestamp': datetime.utcnow(),
        'scheme': request.environ.get('REQUEST_SCHEME'),
        'protocol': request.environ.get('SERVER_PROTOCOL'),
        'method': request.environ.get('REQUEST_METHOD'),
        'uri': request.environ.get('REQUEST_URI'),
        'remote_addr': request.environ.get('REMOTE_ADDR'),
        'remote_port': int(request.environ.get('REMOTE_PORT')),
        'server_name': request.environ.get('SERVER_NAME'),
        'server_port': int(request.environ.get('SERVER_PORT')),
        'app_server': socket.gethostname(),
        'request_headers': dict(request.headers)
    }

    elastic = Elasticsearch('localhost:9200')
    elastic.index(index=app.config['TRACKING_ES_INDEX'], doc_type='_doc', body=data)

However, what I started seeing in Elasticsearch are documents with empty request_headers field ({}), especially when load-testing the app. When I tried to move filling the data dictionary to after_request and passing data to _process_data instead, everything works as expected.

The documentation states:

The context is unique to each thread (or other worker type). request cannot be passed to another thread, the other thread will have a different context stack and will not know about the request the parent thread was pointing to.

However, couple of paragraphs later, it states:

The reference to the proxied object is needed in some situations, such as sending Signals or passing data to a background thread.

So I assumed that using request._get_current_object() to get the actual object and passing it to the separated thread would work. But apparently, there is something that I'm missing...

Passing flask.request to another thread

0 Answers