We are using Django 2.2, python 3.6 and mysql 5.6 for scheduling data intensive jobs.
Memory is increasing over time for a long running job. DEBUG=False in settings.py
why use ignore_conflicts?
we set up a unique key for the table, so ignore_conflicts can filter out those records already in the table.
simple code like this
for record_batch in readFromSomewhere(batch_size):
for record in record_batch:
product = parse(record)
product_list.append(product)
# memory increase and leak
Product.objects.bulk_create(product_list, ignore_conflicts=True)
# memory does not increase
#Product.objects.bulk_create(product_list)
#db.reset_queries()
#gc.collect()
I read a lots of stackoverflow post and put gc.collect() and django.db.reset_query(), but it does not prevent the increase.
If I use Product.objects.bulk_create(products), memory does not increase.
but if I use Product.objects.bulk_create(products,ignore_conflicts=True) the memory increase over time.
the batch size is very small, around 100. I notice if the batch size is smaller, which means the number of bulk_create calls is larger, memory increases faster. If the batch size is larger, then memory increase slower.
Any thoughts to release the memory after batch created(ignore_conflict=True) into db?