Quick unix command to display specific lines in the middle of a file?

Viewed 420919

Trying to debug an issue with a server and my only log file is a 20GB log file (with no timestamps even! Why do people use System.out.println() as logging? In production?!)

Using grep, I've found an area of the file that I'd like to take a look at, line 347340107.

Other than doing something like

head -<$LINENUM + 10> filename | tail -20 

... which would require head to read through the first 347 million lines of the log file, is there a quick and easy command that would dump lines 347340100 - 347340200 (for example) to the console?

update I totally forgot that grep can print the context around a match ... this works well. Thanks!

19 Answers

I found two other solutions if you know the line number but nothing else (no grep possible):

Assuming you need lines 20 to 40,

sed -n '20,40p;41q' file_name

or

awk 'FNR>=20 && FNR<=40' file_name

When using sed it is more efficient to quit processing after having printed the last line than continue processing until the end of the file. This is especially important in the case of large files and printing lines at the beginning. In order to do so, the sed command above introduces the instruction 41q in order to stop processing after line 41 because in the example we are interested in lines 20-40 only. You will need to change the 41 to whatever the last line you are interested in is, plus one.

with GNU-grep you could just say

grep --context=10 ...

No there isn't, files are not line-addressable.

There is no constant-time way to find the start of line n in a text file. You must stream through the file and count newlines.

Use the simplest/fastest tool you have to do the job. To me, using head makes much more sense than grep, since the latter is way more complicated. I'm not saying "grep is slow", it really isn't, but I would be surprised if it's faster than head for this case. That'd be a bug in head, basically.

What about:

tail -n +347340107 filename | head -n 100

I didn't test it, but I think that would work.

I'd first split the file into few smaller ones like this

$ split --lines=50000 /path/to/large/file /path/to/output/file/prefix

and then grep on the resulting files.

If your line number is 100 to read

head -100 filename | tail -1

With sed -e '1,N d; M q' you'll print lines N+1 through M. This is probably a bit better then grep -C as it doesn't try to match lines to a pattern.

print line 5

sed -n '5p' file.txt
sed '5q' file.txt

print everything else than line 5

`sed '5d' file.txt

and my creation using google

#!/bin/bash
#removeline.sh
#remove deleting it comes move line xD

usage() {                                 # Function: Print a help message.
  echo "Usage: $0 -l LINENUMBER -i INPUTFILE [ -o OUTPUTFILE ]"
  echo "line is removed from INPUTFILE"
  echo "line is appended to OUTPUTFILE"
}
exit_abnormal() {                         # Function: Exit with error.
  usage
  exit 1
}

while getopts l:i:o:b flag
do
    case "${flag}" in
        l) line=${OPTARG};;
        i) input=${OPTARG};;
        o) output=${OPTARG};;
    esac
done

if [ -f tmp ]; then
echo "Temp file:tmp exist. delete it yourself :)"
exit
fi

if [ -f "$input" ]; then
   re_isanum='^[0-9]+$'
   if ! [[ $line =~ $re_isanum ]] ; then
      echo "Error: LINENUMBER must be a positive, whole number."
      exit 1
   elif [ $line -eq "0" ]; then
      echo "Error: LINENUMBER must be greater than zero."
      exit_abnormal
   fi
   if [ ! -z $output ]; then
      sed -n "${line}p" $input >> $output
   fi
   if [ ! -z $input ]; then
      # remove this sed command and this comes move line to other file
      sed "${line}d" $input > tmp && cp tmp $input
   fi
fi

if [ -f tmp ]; then
rm tmp
fi

I am surprised only one other answer (by Ramana Reddy) suggested to add line numbers to the output. The following searches for the required line number and colours the output.

file=FILE
lineno=LINENO
wb="107"; bf="30;1"; rb="101"; yb="103"
cat -n ${file} | { GREP_COLORS="se=${wb};${bf}:cx=${wb};${bf}:ms=${rb};${bf}:sl=${yb};${bf}" grep --color -C 10 "^[[:space:]]\\+${lineno}[[:space:]]"; }
Related