Change the file format used by to_sql method

Viewed 117

This works as expected and creates a new table. But the data is stored in a format that only spark can read. How do I store the data in csv format?

from pyathena.pandas.util import to_sql

to_sql(
    mrdf,
    "mrdf_table3",
    conn,
    "s3://" + bucket + "/tutorial/s3dir3/",
    schema="hunspell",
    index=False,
    if_exists="replace",
)

I tried flavor="csv" or flavor="textfile" but the file that is generated is still not readable.


Update: Connection string

from pyathena import connect
bucket = "hunspell"

conn = connect(
    aws_access_key_id="XXX",
    aws_secret_access_key="XXX",
    s3_staging_dir="s3://" + bucket + "/tutorial/staging/",
    region_name="us-east-1",
)
0 Answers
Related