I'm importing large amounts of data to django from csv. I created script with django-extensions which is creating objects like this:
def process_row(row):
return {"foo": row[0], "bar": row[1]}
data = []
for row csv.reader(csvfile):
if len(data) > 5000:
MyModel.objects.bulk_create(data)
data = []
data.append(process_row(row))
Basically, this works really good and fast, but the problem is still rising RAM usage by docker container. It never goes down. At the beggining container uses ~1.5GB, then after tens of bulk_create() it uses up to 10GB and I have to restart Docker Desktop to reset that.
P.S. docker stats says it only uses ~3GB, so I think the problem is with Docker Desktop