Flask application memory leakage with sql alchemy

user_3629658 0
Viewed 540

I am processing a large http response content(1 million records) by streaming it. After reading 10000 elements from the request I am calling a method to insert them into database and then continuing processing the response. I have noticed that the RAM usage is continuously increasing. Below is the sample code. If I comment the db.session.add() part tried to add the data to temporary list the memory leak is not happening. So SQLAlchemy is not properly releasing the memory after insertion to database? Please suggest.

Sample Code.

for chunk in response.iter_content(2048):
    #decode chunk
    xyz_json_list = list()
    xyz_json = json.loads(chunk_string)
    xyz_json_list.append(xyz_json)
    if len(xyz_json_list) == 10000:
        add_elements_to_db(xyz_json_list)
        xyz_json_list = list()
    ...



func add_elements_to_db(xyz_list):
    for xyz in xyz_list:
       db.sesssion.add(create_object_from_json(xyz))
    db.session.commit()
    db.session.expunge_all()
    del xyz_list
    gc.collect()

After each call to add_elements_to_db memory is increasing exponentially.

flask-sqlalchemy flask python
1 Answers

as of v1.0.0, SQLAlchemy supports Bulk Opertions

  • Session.bulk_save_objects()
  • Session.bulk_insert_mappings()
  • Session.bulk_update_mappings()
s = Session()
objects = [
    User(name="u1"),
    User(name="u2"),
    User(name="u3")
]
s.bulk_save_objects(objects)

so you can make little bit changes in add_elements_to_db() function:

[..]

def add_elements_to_db(xyz_list):

    objects = []
   
    for xyz in xyz_list:
        objects.append(create_object_from_json(xyz))

    db.session.bulk_save_objects(objects)

[..]

just curious, why you are not using Redis for asynchronous tasks, you can delegate the execution of add_elements_to_db() function as an asynchronous task to Redis since it's a blocking task and notably slow and for sure you'll see a good performance.

user_123684190
Related