I am trying to understand the best way to deal with Sessions. I am scraping data from a website (and using scrapy, though kind of irrelevant here) and, when facing new data, saving it to my database, or when it match something that already exists, updates some values.
My issue is that each webpage I am scraping concerns only one entry in my table, and may need an update in some other table.
For instance, let's assume I am scraping cars and some features (gearbox type, engine type, owner, ...) are within linked tables.
At the moment, I am doing something like :
parse_cars_page(self, response) # note that this is a standard scrapy method, response holds the scraped html data
# do some stuff here to find the car, its parameters, among which its gearbox and motor types
engine = create_engine("connection_string")
with Session(engine):
stmt_cars = select(model.Cars).where(model.Cars.id == car.id)
current_car = session.execute(stmt_cars).scalar_one_or_none()
if current_car != None:
# update some data
else:
stmt_motor = select(model.Motors).where(model.Motors.type == car.motor_type)
current_motor = session.execute(stmt_motor).scalar_one_or_none()
if current_motor == None:
current_motor = model.Motors(create_it_using_my_data)
current_car.motor = current_motor
# then do the same for the gearbox, owner, .........
session.commit()
This works fine. But it's slow, very slow. The whole with statement takes about a minute to execute.
As the info is contained within a single page, and that I am opening thousands of them, getting the data is tedious and takes an enormous amount of time.
I tried finding best practices from the SQL Alchemy documentation but I can't find what I am looking for, and I understand that keeping a Session opened and share it within my whole app isn't a good idea.
My app is the only thing that will update the data in the database. Other processes may read it but not write to it.
Is there a way to open a Session, copy in memory a snapshot of the database, update this snapshot while the Session is closed, and only open a Session for synchronization once I generated X new data ?