how to copy pandas dataframe to vertica with airflow?

user_12557235 0
Viewed 88

when I run the following function:

def write_df_to_vertica(conn: Cursor, df: pd.DataFrame):
    query = """COPY temp_data.test(coumn1, coumn2, coumn3) FROM STDIN DELIMITER ',' """
    conn.copy(query, df.to_csv(index=False, header=False))

it works, and the data is written to vertica db.

However when I call this function from PythonOperator in a AirFlow dag, the data isn't written to the db.

Why? and how to fix it?

vertica-python vertica airflow pandas python
1 Answers

try this maybe?

`pandas_to_vertica(df, 
              cursor=None, 
              name: str = "", 
              schema: str = "", 
              dtype: dict = {}, 
              parse_n_lines: int = 10000, 
              temp_path: str = "", 
              insert: bool = False,)`

this is the format for it i guess.

user_190070510
Related