Hi I am trying to read csv file data from S3 using R. (28G data, using m5.24xlarge cluster with 348G memory) But met error:
Error in curl::curl_fetch_disk(url, x$path, handle = handle): Failure writing output to destination Traceback:
- s3read_using(FUN = read.csv, object = "s3://communication-attribution/test_data/uidisplayemailopenformatteddata/part-00000-84c15b36-9cfa-4944-91e6-fb35145c6f09-c000.csv")
- save_object(bucket = bucket, object = object, file = tmp)
- s3HTTP(verb = "GET", bucket = bucket, path = paste0("/", object), . headers = headers, write_disk = httr::write_disk(path = file, . overwrite = overwrite), ...)
- httr::GET(url, H, query = query, write_disk, show_progress, ...)
- request_perform(req, hu$handle$handle)
- request_fetch(req$output, req$url, handle)
- request_fetch.write_disk(req$output, req$url, handle)
- curl::curl_fetch_disk(url, x$path, handle = handle)
Here is my code:
dfsmall <- s3read_using(FUN = read.csv, object = "s3://communication-attribution/test_data/uidisplayemailopenformatteddata/part-00000-84c15b36-9cfa-4944-91e6-fb35145c6f09-c000.csv")