How to use sed to print from the beginning of the file till all the matching elements. The file size is greater than 25GB

user_9149501 0
Viewed 336

I am not being able to use sed to print the content of the file from the beginning until the matching pattern as to when it finds the first occurrence of the pattern it stops and does not print all the matching patterns.

As the file size is greater than 25GB. However, below is a small example of the problem.

Eg: The content of the file is:

2010T10:11:12 some data.
2012T10:11:12 some data.
2013T10:11:12 They all are different data
2014T10:11:12 Logs basically
2014T10:11:12 Error Logs
2014T10:11:12 Any Data
2014T10:11:12 Data
2015T10:11:12 Some fields
2016T10:11:12 etc

Basically, when I give the range from 2010T10:11:12 - 2014T10:11:12 it should print till the 7th line of the file.

The command I am using for printing is:

sed -n '1,/2014T10:11:12/p' File-1.txt

Output:

2010T10:11:12 some data.
2012T10:11:12 some data.
2013T10:11:12 They all are different data
2014T10:11:12 Logs basically

Expected Output:

2010T10:11:12 some data.
2012T10:11:12 some data.
2013T10:11:12 They all are different data
2014T10:11:12 Logs basically
2014T10:11:12 Error Logs
2014T10:11:12 Any Data
2014T10:11:12 Data

This command duplicates the first line of the match pattern:

sed -n '1,/2014T10:11:12/p;/2014T10:11:12/p' File-1.txt

Output:

2010T10:11:12 some data.
2012T10:11:12 some data.
2013T10:11:12 They all are different data
2014T10:11:12 Logs basically <- Duplicate line. Need to
2014T10:11:12 Logs basically <- remove any one of them
2014T10:11:12 Error Logs
2014T10:11:12 Any Data
2014T10:11:12 Data

Another issue is that the content of the file changes every second so we cannot give any range like 1-7 or 5-7. It has to be on the basis of the pattern like 2010T10:11:12 - 2014T10:11:12 or 2015T10:11:12 - 2016T10:11:12.

ubuntu-18.04 sed awk shell bash
4 Answers

An alternative version of awk would be:

awk '($1 > "2014T10:11:12"){exit}1' file

This is useful when processing big files as it will stop reading the file when the first field is lexicographical bigger than "2014T10:11:12".

If you want to print a range, you can do:

awk '($1 > "2014T10:11:12"){exit}($1 >= "2013T12:12:12")' file

And when you want to overoptimse it:

awk '($1 >= "2013T12:12:12") { if($1 > "2014T10:11:12"){exit}; print}' file
user_83440600

Try this:

awk '($1 >= "2010T10:11:12") && ($1 <= "2014T10:11:12")' File-1.txt
user_3371490
user_19215460

You need an address range in sed:

begin='^2010T10:11:12'
end='^2014T10:11:12'

sed -n "
    /$begin/,/$end/{ p; d; }
    /$end/p
" file

This assumes the input file is sorted by the first field (the date and time).
The second command (/$end/p) is required since you want to print all lines matching the $end. The range address (/$begin/,/$end/) matches lines starting from where the $begin matches, and continues until the first line matching the $end (inclusively).

The below may be more efficient since it gives up reading the input after the last line matching the $end (the input must be sorted in order this to work).

begin='^2010T10:11:12'
end='^2014T10:11:12'

sed -n "
    /$begin/,/$end/{
        /$end/{
            :a
            p
            n
            /$end/!q
            ba
        }
        p
    }
" file
user_138090010
Related