How to copy table from source db to a target db?
So requirements are:
- Create source schema to target db if not exists.
- Copy table definition from source db to target db (so datatypes are the same).
- Insert data from source table to target table.
What I've tried so far:
import sqlalchemy as sa
import os
import pandas as pd
def copy_to_db(schema, list_of_tables, source_engine, target_engine):
# 1. Create source schema to target db if not exists.
if not target_engine.dialect.has_schema(target_engine, schema):
target_engine.execute(sa.schema.CreateSchema(schema))
# 2. Copy table definition from source db to target db (so datatypes are the same).
smeta = sa.MetaData(bind=source_engine)
for table in list_of_tables:
sa.Table(table, smeta, schema=schema, autoload=True)
smeta.create_all(target_engine)
# 3. Insert data from source table to target table.
for table in list_of_tables:
df = pd.read_sql_table(table_name=table, con=source_engine, schema=schema)
df.to_sql(name=table, con=target_engine, schema=schema, if_exists="replace", index=False)
I believe I have requirements 1 and 2.
3 doesn't sit well with me because reading into a dataframe could infer datatypes/dtypes which could raise errors on insert to target (also worried if data is too large that this method would slow the script down). Is there a more efficient/intuitive way of doing this in SQLAlchemy?
Working with a PostgreSQL database (version 14)