R data.table problem when read file with inconsistent column

user_13480887 0
Viewed 833

When I use R data.table(fread) to read dat file (3GB) a problem occurs:

Stopped early on line 3169933. Expected 136 fields but found 138. Consider fill=TRUE and comment.char=. First discarded non-empty line:

enter image description here

My code:

library(data.table)
file_path = 'data.dat' # 3GB
fread(file_path,fill=TRUE)

The problem is that my file has ~ 5 million rows. In detail:

  • From row 1 to row 3169933 it has 136 columns
  • From row 3169933 to row 5000000 it has 138 columns

fread() only reads my file to row 3169933 due to this error. fill = TRUE did not help in this case. Could anyone help me ?

R version: 3.6.3 data.table version: 1.13.2

Note about fill=TRUE in this case:

[Case 1- not my case] if part 1 of my file (50% rows) have 138 columns and part 2 have 136 columns then the fill=TRUE will help (it will fill two column in part 2 with NA)

[Case 2- my case] if part 1 of my file (50% rows) have 136 columns and part 2 have 138 columns then the fill =TRUE will not help in this case.

data.table fread r
2 Answers

Not sure why you still have the problem even with fill=T... But if nothing helps, you can try playing with something like this:

tryCatch(
  expr    = {dt1 <<- fread(file_path)},
  warning = function(w){
    cat('Warning: ', w$message, '\n\n');
    n_line <- as.numeric(gsub('Stopped early on line (\\d+)\\..*','\\1',w$message))
    if (!is.na(n_line)) {
      cat('Found ', n_line,'\n')
      dt1_part1 <- fread(file_path, nrows=n_line)
      dt1_part2 <- fread(file_path, skip=n_line)
      dt1 <<- rbind(dt1_part1, dt1_part2, fill=T)
    }
  },
  finally = cat("\nFinished. \n")
);

tryCatch() construct catches warning message so you can extract the line number and process it accordingly.

user_17072780

Try to read them separately, combine them after creating two extra columns for the first part.

first_part = fread('data.dat', nrows = 3169933) %>%
  mutate(extra_1 = NA, extra_2 = NA)

second_part = fread('data.dat', skip = 3169933)
df = bind_rows(first_part, second_part)
user_23440110
Related