Serialize Django Queryset to JSON for Celery before execution

user_876180 0
Viewed 1783

I am given a Queryset from an external library and - since Querysets are lazy - I'd like to serialize it to JSON before it is accessed, and thus executed, in the DB, so it may be executed in an asynchronous Celery task instead.

My question is, is there a way of representing the essential elements of a Queryset as JSON so I don't have to use Pickle?

I know I could get the raw query with queryset.query, but since I'd have to execute a raw query at the other end, I don't much like that idea.

Details

The specific use-case here is evaluating the Queryset in a task so the results can then be exported to a file and dumped on a server for later access.

The result set can be quite large, so doing the export as part of a standard request/response cycle can often timeout.

The request is generated from a Django Admin change view list filter. I've tried going back through the list filter code, but it seems like it just generates filter objects - not suitable for JSON serialization.

I could look at the list filter query string and re-evaluate all of the keys into a Queryset filter, but many of the filters are SimpleListFilter classes that also just return filter objects, and don't reveal real Queryset filters in the URL. I'd need to recreate logic for these filters just to get a kwarg-style filter that can be JSON serialized.

celery django-queryset django
4 Answers

At time of writing, using Python 3.7+ and Django 2.2 the pickle dumps/loads seems to work. It is missing just a queryset.update() in the end.

I tried something like:

# packing
queryset = MyModel.objects.filter(...)
serialized_queryset = pickle.dumps(queryset.query)

# unpacking
restored = MyModel.objects.all()
restored.query = pickle.loads(serialized_queryset)
restored.update()

and it works fine. Not tested across celery tasks yet, but I expect it should work.

user_152743400

You need to decode and encode query. With Django 2.1 and Celery 4.2 works with this code:

query_dump = pickle.dumps(queryset.query).decode('latin1')

query = pickle.loads(query_dump.encode('latin1'))
user_87665680
Related