Importing data from postgresql with Dask

user_10791600 0
Viewed 335

So I have a large (7GB) dataset stored in postgres that I'm trying to import into Dask. I'm trying the read_sql_table function, but keep getting ArgumentErrors.

My info in postgres is the following:

  • database is "my_database"
  • schema is "public"
  • data table is "table"
  • username is "fred"
  • password is "my_pass"
  • index in postgres is 'idx'

I am trying to get this piece of code to work:

df = dd.read_sql_table('public.table', 'jdbc:postgresql://localhost/my_database?user=fred&password=my_pass', index_col='idx')

Am I formatting something incorrectly?

dask postgresql python
1 Answers

I was finally able to figure it out by using psycopg2. The answer is below:

df = dd.read_sql_table('table', 'postgresql+psycopg2://postgres:fred@localhost/my_database', index_col = 'idx')

Additionally, I had to create a different index in the postgres table. The original index needed to be a whole separate column. I did this with the following line in Postgres:

alter table table add idx serial;
user_107916000
Related