dplyr copy_to large data

Viewed 444

Hi I would like to know.

  1. what is the best way to copy large data into an sql database using dplyr?

copy_to fails as data is larger than memory. I am using code:

library(DBI)
library(dplyr)


db <- dbConnect(
  RPostgreSQL::PostgreSQL(),
  dbname = "postgres",
  host="localhost",
  user="user",
  password="password")


Data<-rio::import("Data/data.feather")
Data <- copy_to(db, Data,temporary=FALSE)

This results in error:

Error in postgresqlExecStatement(conn, statement, ...) : 
  RS-DBI driver: (could not Retrieve the result : ERROR:  out of memory
DETAIL:  Cannot enlarge string buffer containing 0 bytes by 1305608043 more bytes.
)
  1. is there a way to do this without having to import the data first?
  2. is there a way to do this though sparkly since this is using only one core?
0 Answers
Related