Efficiently copy data between databases with sqlalchemy

Viewed 112

I'm trying to mirror a postgresql + PostGIS database that I defined with sqlalchemy to a sqlite (spatialite) file database. The session.merge() method appears to work for adding the instances queried from the first session to the other session, but it does not scale for nearly a million rows. See the example below that copies data from an in-memory sqlite database to another memory database for the sake of easy reproducibility. I'm looking for an approach (potentially completely different from what I'm doing now) to efficiently move all the data from one database to another.

    from sqlalchemy import create_engine
    from sqlalchemy import Table, Column, Integer, ForeignKey, String
    from sqlalchemy.orm import declarative_base, sessionmaker
    from sqlalchemy.orm import relationship, joinedload
    
    
    engine_0 = create_engine('sqlite:///:memory:', echo=True)
    engine_1 = create_engine('sqlite:///:memory:', echo=True)
    Base = declarative_base()
    Session0 = sessionmaker(bind=engine_0)
    Session1 = sessionmaker(bind=engine_1)
    
    # Define ORM models
    association_table = Table('association', Base.metadata,
        Column('parent_id', ForeignKey('parent.id'), primary_key=True),
        Column('child_id', ForeignKey('child.id'), primary_key=True)
    )
    
    class Parent(Base):
        __tablename__ = 'parent'
        id = Column(Integer, primary_key=True)
        name = Column(String)
        children = relationship(
            "Child",
            secondary=association_table,
            back_populates="parents")
    
    class Child(Base):
        __tablename__ = 'child'
        id = Column(Integer, primary_key=True)
        name = Column(String)
        parents = relationship(
            "Parent",
            secondary=association_table,
            back_populates="children")
    
    # Create schema
    Base.metadata.create_all(engine_0)
    Base.metadata.create_all(engine_1)
    
    # Create some example instances
    # Children
    bart = Child(name='Bart')
    lisa = Child(name='Lisa')
    maggie = Child(name='Maggie')
    milhouse = Child(name='Milhouse')
    # Parents
    homer = Parent(name='Homer',
                   children=[bart, lisa, maggie])
    marge = Parent(name='Marge',
                   children=[bart, lisa, maggie])
    flanders = Parent(name='Ned')
    kirk = Parent(name='Kirk', children=[milhouse])
    
    # Insert data into first database
    session_0 = Session0()
    session_0.add_all([homer, marge, flanders, kirk])
    session_0.commit()
    
    # Query the data and insert it into the second database
    all_obj = session_0.query(Parent).options(joinedload('*')).all()
    session_0.expunge_all()
    session_1 = Session1()
    for obj in all_obj:
        session_1.merge(obj)
    session_1.commit()
    
    # MAke sure that 4 instance of child are present in the second database
    print(session_1.query(Child).all())

One alternative approach I have tried (unsuccessfully) is to make the parent objects transient using the sqlalchemy.orm.make_transient() function and use session.add_all() instead of session.merge() to insert the objects into the second session. However, this does not propagate to the relationships and only Parent objects are made transient.

0 Answers
Related