Line-oriented streaming in Ruby (like grep)

user_157237 0
Viewed 1348

By default Ruby opens $stdin and $stdout in buffered mode. This means you can't use Ruby to perform a grep-like operation filtering text. Is there any way to force Ruby to use line-oriented mode? I've seen various solutions including popen3 (which does buffered-mode only) and pty (which doesn't separately handle $stdout and $stderr, which I require).

How do I do this? Python seems to have the same lack.

pipe ruby
4 Answers

The accepted answer by user208769 is good, but has one flaw: under certain conditions you will loose last line. I'll show how to reproduce it and how to fix it below:

To reproduce the "last line lost" bug:

mkdir deleteme
touch deleteme/1 deleteme/2 deleteme/3
ls deleteme/ | ./rgrep.rb ''
Looking for
1
2

as you can see the "3" file is missing from the rgrep output. Surprisingly for the different filename length it would work differently though! Look:

rm -fr deleteme/
mkdir deleteme
touch deleteme/11 deleteme/22 deleteme/33
ls deleteme/ | ./rgrep.rb ''
Looking for
11
22
33

Now the third file is present! What a bug! Isn't she a beauty!!?
One can only imagine how much damage this random behaviour can cause.

To fix the bug we'd modify rescue portion slightly:

#!/usr/bin/env ruby
STDIN.sync = true
STDOUT.syswrite "Looking for #{ARGV[0]}\n"

def next_line
  mybuff = @overflow || ""
  until mybuff[/\n/]
    mybuff += STDIN.sysread(8)
  end
  overflow = mybuff.split("\n")
  out, *others = overflow
  @overflow = others.join("\n")
  out
rescue EOFError => e
  if @overflow.to_s.size > 0
    leftover_line = @overflow
    @overflow = ''
    return leftover_line
  else
    false
  end
end

line = next_line
while line
  STDOUT.syswrite "#{line}\n" if line =~ /#{ARGV[0]}/i
  line = next_line
end

I'll leave the "why" portion out of this post as an exercise for curious ones as otherwise it wont be digested properly (and this post is already way too long for my 1st post ever;) heh..

user_70220620
Related