Read tab delimited text file in Spark R

user_9153761 0
Viewed 810

I have a tab delimited file that is saved as a .txt with " " around the string variables. The file can be found here.

I am trying to read it into Spark-R (version 3.1.2), but cannot successfully bring it into the environment. I've tried variations of the read.df code, like this:

df <- read.df(path = "FILE.txt", header="True", inferSchema="True", delimiter = "\t", encoding="ISO-8859-15")

df <- read.df(path = "FILE.txt", source = "txt", header="True", inferSchema="True", delimiter = "\t", encoding="ISO-8859-15")

I have had success with bringing in CSVs with read.csv, but many of the files I have are over 10GB, and is not practical to convert them to CSV before bring them into Spark-R.

EDIT: When I run read.df I get a laundry list of errors, starting with this:

enter image description here

I am able to bring in csv files used in a previous project with both read.df and read.csv, so I don't think it's a java issue.

sparkr txt apache-spark tabs r
1 Answers

If you don't need to specifically use Spark R, then base R read.table should work just fine for the .txt you provided. Note that it is tab-delimited, and so this should be specified.

Something like this should work:

dat <- read.table("FILE.TXT",  
                  sep="\t",
                  header=TRUE)
user_68060510
Related