SQLAlchemy copy/insert table from source db to target db

Viewed 27

How to copy table from source db to a target db?

So requirements are:

  1. Create source schema to target db if not exists.
  2. Copy table definition from source db to target db (so datatypes are the same).
  3. Insert data from source table to target table.

What I've tried so far:

import sqlalchemy as sa
import os
import pandas as pd

def copy_to_db(schema, list_of_tables, source_engine, target_engine):

    # 1. Create source schema to target db if not exists.
    if not target_engine.dialect.has_schema(target_engine, schema):
        target_engine.execute(sa.schema.CreateSchema(schema))

    # 2. Copy table definition from source db to target db (so datatypes are the same).
    smeta = sa.MetaData(bind=source_engine)
    for table in list_of_tables:
        sa.Table(table, smeta, schema=schema, autoload=True)
    smeta.create_all(target_engine)

    # 3. Insert data from source table to target table.
    for table in list_of_tables:
        df = pd.read_sql_table(table_name=table, con=source_engine, schema=schema)
        df.to_sql(name=table, con=target_engine, schema=schema, if_exists="replace", index=False)

I believe I have requirements 1 and 2.

3 doesn't sit well with me because reading into a dataframe could infer datatypes/dtypes which could raise errors on insert to target (also worried if data is too large that this method would slow the script down). Is there a more efficient/intuitive way of doing this in SQLAlchemy?

Working with a PostgreSQL database (version 14)

0 Answers
Related