skipping first x and last y lines of file

user_2751573 0
Viewed 136

I'm doing some simple parsing on text files (which could get up into the 1GB range). How would I go about skipping the first N rows, and more importantly, the last (different) N rows? I'm sure I could open the file and count the rows, and do something with $_ < total_row_count -N, but that seems incredibly inefficient. I'm pretty much a perl newb, btw.

perl
2 Answers

A file is a sequence of bytes, without the notion of "lines." Some of those bytes are considered as "line" separators (linefeeds), which is how software gives us our "logical" lines to work with. So there is no way to know how many lines there are in a file -- without having read it and counted them, that is.

A simple and naive way is to read line-by-line and count

open my $fh, '<', $file  or die "Can't open $file: $!";

my $cnt;
++$cnt while <$fh>;

with a little faster version using $. variable

1 while <$fh>;
my $cnt = $.;

These take between 2.5 and 3 seconds for a 1.1 Gb text file on a reasonable desktop.

We can speed this up a lot by reading in larger chunks and counting newline characters

open my $fh, '<', $file  or die "Can't open $file: $!";

my $cnt; 
NUM_LINES: {
    my $len = 64_000; 
    my $buf;

    $cnt += $buf =~ tr/\n// 
        while read $fh, $buf, $len;

    seek $fh, 0, 0;
};

This goes in barely over half a second, on same hardware and Perl versions.

I've put it in a block to scope unneeded variables but it should be in a sub, where you can then check where the filehandle is when you get it and return it there after counting (so we can count the "rest" of lines from some point in the file and the processing can then continue), etc. It should also include checks on read operation, at each invocation.

I'd think that a half a second overhead on a Gb large file isn't that bad at all.

Still, you can go for faster yet, at the expense of it being far messier. Get the file size (metadata, so no reading involved) and seek to a position estimated to be the wanted number of lines before the end (no reading involved). That most likely won't hit the right spot so read to the end to count lines and adjust, seeking back (further or closer). Repeat until you reach the needed place.

open my $fh, "<", $file; 
my $size = -s $file;

my $estimated_line_len = 80;
my $num_last_lines     = 100;

my $pos = $size - $num_last_lines*$estimated_line_len;

seek $fh, $pos, 0; 

my $cnt;    
++$cnt while <$fh>; 

say "There are $cnt lines from position $pos to the end"; 

# likely need to seek back further/closer ...

I'd guess that this should get you there in under 100 ms. Note that $pos is likely inside a line.

Then once you know the number of lines (or the position for desired number of lines before the end) do seek $fh, 0, 0 and process away. Or really have this in a sub which puts the filehandle back where it was before returning, as mentioned.

user_46533790

I think you need a circular buffer to avoid reading entire file on your memory.

skip-first-last.pl

#!/usr/bin/perl
use strict;
use warnings;

my ($first, $last) = @ARGV;

my @buf;
while (<STDIN>) {
    my $mod = $. % $last;
    print $buf[$mod] if defined $buf[$mod];
    $buf[$mod] = $_ if $. > $first;
}

1;

Skip first 5 lines and last 2 lines:

$ cat -n skip-first-last.pl | ./skip-first-last.pl 5 2
     6
     7  my @buf;
     8  while (<STDIN>) {
     9      my $mod = $. % $last;
    10      print $buf[$mod] if defined $buf[$mod];
    11      $buf[$mod] = $_ if $. > $first;
    12  }
user_4825190
Related