fread() run time is taking longer than reported speed

user_4556186 0
Viewed 370

I am attempting to read a large file into R on an EC2 instance. However, I have been experiencing run times which are far longer than the amount of time reported by fread after reading in some of the data.

Below, for instance, I have the verbose=TRUE output of fread when reading in only the first row of data for my csv file. As you can see, the reported run time is much shorter than the actual run time. Do you have any idea why this is happening? Is there any way I will be able to speed up the process so it is more in line with the runtime that fread reports after reading in the data?

> start_time <- Sys.time()
> fread(file_name_1, nrows=1, verbose=TRUE)
Input contains no \n. Taking this to be a filename to open
File opened, filesize is 68.770914 GB.
Memory mapping ... ok
Detected eol as \n only (no \r afterwards), the UNIX and Mac standard.
Positioned on line 1 after skip or autostart
This line is the autostart and not blank so searching up for the last non-blank ... line 1
Detecting sep ... ','
Detected 55 columns. Longest stretch was from line 1 to line 30
Starting data input on line 1 (either column names or first row of data). First 10 characters: bank_num,b
All the fields on line 1 are character fields. Treating as the column names.
nrow set to nrows passed in (1)
Type codes (point  0): 1114434134111034444411333333333333333333333333333311111
Type codes: 1114434134111034444411333333333333333333333333333311111 (after applying colClasses and integer64)
Type codes: 1114434134111034444411333333333333333333333333333311111 (after applying drop or select (if supplied)
Allocating 55 column slots (55 - 0 dropped)
Read 1 rows and 55 (of 55) columns from 68.771 GB file in 00:00:27
Read 1 rows. Exactly what was estimated and allocated up front
  26.480s (100%) Memory map (rerun may be quicker)
   0.000s (  0%) sep and header detection
   0.000s (  0%) Count rows (wc -l)
   0.000s (  0%) Column type detection (100 rows at 10 points)
   0.000s (  0%) Allocation of 1x55 result (xMB) in RAM
   0.000s (  0%) Reading data
   0.000s (  0%) Allocation for type bumps (if any), including gc time if triggered
   0.000s (  0%) Coercing data already read in type bumps (if any)
   0.000s (  0%) Changing na.strings to NA
  26.480s        Total
> end_time <- Sys.time()
> end_time - start_time
Time difference of 9.695263 mins
data.table fread amazon-ec2 r
1 Answers

Please always state version numbers; e.g. output of sessionInfo(). But I can tell you're probably using the CRAN version.

Please always check NEWS before asking on Stack Overflow.

Item 3 (amongst many other fread improvements) :

Memory maps lazily; e.g. reading just the first 10 rows with nrow=10 is 12s down to 0.01s from cold for a 9GB file. Large files close to your RAM limit may work more reliably too. The progress meter will commence sooner and more consistently.

The latest version from dev can easily be tried with this install command . You wrote EC2, so presumably Linux but any Windows users can use the Windows.zip from dev with no tools needed.

Since you have a 68GB csv, you should definitely benefit greatly from data.table v1.10.5+. Please update here how you get on with it.

user_4033100
Related