Stream filter large number of lines that are specified by line number from stdin

Keeping with OP's current idea:

xz -dcq huge.txt.xz | awk '!firstfile_proceed { nums[$1]; next } (FNR in nums)' select.txt firstfile_proceed=1 -

Where the - (at the end of the line) tells awk to read from stdin (in this case the output from xz that's being piped to the awk call).

Another way to do this (replaces all of the above code):

awk '
FNR==NR { nums[$1]; next }             # process first file
FNR in nums                            # process subsequent file(s)
' select.txt <(xz -dcq huge.txt.xz)

Comments removed and cut down to a 'one-liner':

awk 'FNR==NR {nums[$1];next} FNR in nums' select.txt <(xz -dcq huge.txt.xz)

Adding some logic to implement Ed Morton's comment (exit processing once FNR > largest value from select.txt):

awk '
# process first file

FNR==NR      { nums[$1]
               maxFNR= ($1>maxFNR ? $1 : maxFNR)
               next
             }

# process subsequent file(s):

FNR > maxFNR { exit }
FNR in nums
' select.txt <(xz -dcq huge.txt.xz)

NOTES:

• keeping in mind we're talking about scanning millions of lines of input ...
• FNR > maxFNR will obviously add some cpu/processing time to the overall operation (though less time than FNR in nums)
• if the operation routinely needs to pull rows from, say, the last 25% of the file then FNR > maxFNR is likely providing little benefit (and probably slowing down the operation)
• if the operation routinely finds all desired rows in, say, the first 50% of the file then FNR> maxFNR is probably worth the cpu/processing time to keep from scanning the entire input stream (then again, the xz operation, on the entire file, is likely the biggest time consumer)
• net result: the additional NFR > maxFNR test may speed-up/slow-down the overall process depending on how much of the input stream needs to be processed in a typical run; OP would need to run some tests to see if there's a (noticeable) difference in overall runtime

To clarify my previous comment. I'll show a simple reproducible sample:

linelist content:

10
15858
14
1499

To simulate a long input, I'll use seq -w 100000000.

Comparing sed solution with my suggestion, we have:

#!/bin/bash

time (
    sed 's/$/p/' linelist > selector
    seq -w 100000000 | sed -nf selector
)
time (
    sort -n linelist | sed '$!{s/$/p/};$s/$/{p;q}/' > my_selector
    seq -w 100000000 | sed -nf my_selector
)

output:

000000010
000000014
000001499
000015858

real    1m23.375s
user    1m38.004s
sys 0m1.337s
000000010
000000014
000001499
000015858

real    0m0.013s
user    0m0.014s
sys 0m0.002s

Comparing my solution with awk:

#!/bin/bash

time (
    awk '
# process first file

FNR==NR      { nums[$1]
               maxFNR= ($1>maxFNR ? $1 : maxFNR)
               next
             }

# process subsequent file(s):

FNR > maxFNR { exit }
FNR in nums
' linelist <(seq -w 100000000)
)

time (
    sort -n linelist | sed '$!{s/$/p/};$s/$/{p;q}/' > my_selector
    sed -nf my_selector <(seq -w 100000000)
)

output:

000000010
000000014
000001499
000015858

real    0m0.023s
user    0m0.020s
sys 0m0.001s
000000010
000000014
000001499
000015858

real    0m0.017s
user    0m0.007s
sys 0m0.001s

In my conclusion, seq using q is comparable with awk solution. For readability and maintainability I prefer awk solution.

Anyway, this test is simplistic and only useful for small comparisons. I don't know, for example, what the result would be if I test this against the real compressed file, with heavy disc I/O.

EDIT by Ed Morton:

Any speed test that results in all output values that are less than a second is a bad test because:

1. In general no-one cares if X runs in 0.1 or 0.2 secs, they're both fast enough unless being called in a large loop, and
2. Things like cache-ing can impact the results, and
3. Often a script that runs faster for a small input set where execution speed doesn't matter will run slower for a large input set where execution speed DOES matter (e.g. if the script that's slower for the small input spends time setting up data structures that will allow it to run faster for the larger)

The problem with the above example is it's only trying to print 4 lines rather than the 1000s of lines that the OP said they'd have to select so it doesn't exercise the difference between the sed and the awk solution that causes the sed solution to be much slower than the awk one which is that the sed solution has to test every target line number for every line of input while the awk solution just does a single hash lookup of the current line. It's an order(N) vs order(1) algorithm on each line of the input file.

Here's a better example showing printing every 100th line from a 1000000 line file (i.e. will select 1000 lines) rather than just 4 lines from any size file:

$ cat tst_awk.sh
#!/usr/bin/env bash

n=1000000
m=100
awk -v n="$n" -v m="$m" 'BEGIN{for (i=1; i<=n; i+=m) print i}' > linelist

seq "$n" |
    awk '
        FNR==NR {
            nums[$1]
            maxFNR = $1
            next
        }
        FNR in nums {
            print
            if ( FNR == maxFNR ) {
                exit
            }
        }
    ' linelist -

$ cat tst_sed.sh
#!/usr/bin/env bash

n=1000000
m=100
awk -v n="$n" -v m="$m" 'BEGIN{for (i=1; i<=n; i+=m) print i}' > linelist

sed '$!{s/$/p/};$s/$/{p;q}/' linelist > my_selector
seq "$n" |
    sed -nf my_selector

$ time ./tst_awk.sh > ou.awk

real    0m0.376s
user    0m0.311s
sys     0m0.061s

$ time ./tst_sed.sh > ou.sed

real    0m33.757s
user    0m33.576s
sys     0m0.045s

As you can see the awk solution ran 2 orders of magnitude faster than the sed one, and they produced the same output:

$ diff ou.awk ou.sed
$

If I make the input file bigger and select 10,000 lines from it by setting:

n=10000000
m=1000

in each script, which is probably getting more realistic for the OPs usage, the difference becomes really impressive:

$ time ./tst_awk.sh > ou.awk

real    0m2.474s
user    0m2.843s
sys     0m0.122s

$ time ./tst_sed.sh > ou.sed

real    5m31.539s
user    5m31.669s
sys     0m0.183s

i.e. awk runs in 2.5 seconds while sed takes 5.5 minutes!

If you have a file of line numbers, add p to the end of each and run it as a sed script.

If linelist contains

10
14
1499
15858

then sed 's/$/p/' linelist > selector creates

10p
14p
1499p
15858p

then

$: for n in {1..1500}; do echo $n; done | sed -nf selector
10
14
1499

I didn't send enough lines through to match 15858 so that one didn't print.

This works the same with a decompression from a file.

$: tar xOzf x.tgz | sed -nf selector
10
14
1499