Subsetting a CSV based on a percentage of unique values

Any suggestions on how to improve the efficiency?

Avoid sort in 1st step as 2nd and 3rd do not care about order, you might do your whole 1st step using single awk command as follows:

awk 'BEGIN{FS=","}!arr[$11]++{print $11}' File1.csv > uniqueValues.txt

Explanation: I inform GNU AWK that field separator (FS) is comma, then for each line I do arr[$11]++ to get number of occurence of value in 11th column and use ! to negate it, so 0 becomes true, whilst 1 and greater becomes false. If this hold true I print 11th column.

Please test above against your 1st step for you data and then select one which is faster.

As for 3th step you might attemp using not-GNU AWK if you are allowed to install tools at your machine. For example author of article¹ Don’t MAWK AWK – the fastest and most elegant big data munging language! found nawk faster than GNU AWK and mawk faster than nawk. After installing prepare test data and measure times for

gawk -F, 'NR==FNR{a[$1]=$0;next}($11 in a){print}' uniqueValues.10pct.txt File1.csv > File1_subsample.csv
nawk -F, 'NR==FNR{a[$1]=$0;next}($11 in a){print}' uniqueValues.10pct.txt File1.csv > File1_subsample.csv
mawk -F, 'NR==FNR{a[$1]=$0;next}($11 in a){print}' uniqueValues.10pct.txt File1.csv > File1_subsample.csv

then use one which proved by fastest.

¹be warned that values shown pertains to versions available at September 2009, you might get different times with version available at June 2022.

You might find this to be faster (untested since no sample input/output provided):

cut -f11 -d',' File1.csv |
    sort -u > uniqueValues.txt
numUnq=$(wc -l < uniqueValues.txt)
shuf -n "$(( numUnq / 10 ))" uniqueValues.txt |
    awk -F',' 'NR==FNR{a[$1]; next} $11 in vals' - File1.csv

You could try replacing that first cut | sort; numUnq=$(wc...) with

numUnq=$(awk -F',' '!seen[$11]++{print $11 > "uniqueValues.txt"; cnt++} END{print cnt+0}' File1.csv)

to see if that's any faster but I doubt it since cut, sort, and wc are all very fast while awk has to do regexp-based field splitting and store all $11 values in memory (which can get slow as the array size increases due to how dynamic array allocation works).

Create a sample *.csv file:

for ((i=1;i<=100;i++))
do
    for ((j=1;j<=100;j++))
    do
        echo "a,b,c,d,e,f,g,h,i,j,${j},k,l,m"
    done
done > large.csv

NOTES:

• 1,000 total lines
• 100 unique values in the 11th field
• each unique value shows up 10 times in the file

We'll look at a couple awk ideas that:

• keep track of unique values as we find them
• apply the random percentage check as we encounter a new (unique) value
• require just a single pass through the source file
• NOTE: both of these awk scripts (below) replace all of OP's current code (cat/cut/sort/uniq/cat/awk/awk)

First idea applies our random percentage check each time we find a new unique value:

awk -F',' '
BEGIN        { srand() }
!seen[$11]++ { if (rand() <= 0.10)              # if this is the 1st time we have seen this value and rand() is <= 10% then ...
                  keep[$11]                     # add the value to our keep[] array
             }
$11 in keep                                     # print current line if $11 is an index in the keep[] array
' large.csv > small.csv

NOTES:

• one drawback to this approach is that the total number of unique values is not guaranteed to always be exactly 10% since we're at the mercy of the rand() function, for example ...
• a half dozen sample runs generated 70, 110, 100, 140, 110 lines (ie, 7, 11, 10, 14 and 11 unique values) in small.csv

A different approach where we pre-generate a random set of modulo 100 values (ie, 0 to 99); as we find a new uniq value we check the count (of uniq values) modulo 100 and if we find a match to our pre-generated set then we print the row:

awk -F',' -v pct=10 '
BEGIN        { srand()
               delete mods                      # force awk to treat all "mods" references as an array and not a scalar
               while (length(mods) < pct)       # repeat loop until we have "pct" unique indices in the mods[] array
                     mods[int(rand() * 100)]    # generate random integers betwen 0 and 99
             }
!seen[$11]++ { if ((++uniqcnt % 100) in mods)   # if this is the 1st time we have seen this value then increment our unique value counter and if "modulo 100" is an index in the mods[] array then ...
                  keep[$11]                     # add the value to our keep[] array
             }
$11 in keep                                     # print current line if $11 is an index in the keep[] array
' large.csv > small.csv

NOTES:

• for a large pct this assumes the rand() results are evenly distributed between 0 and 1 so that the mods[] array is populated in a timely manner
• this has the benefit of printing lines that represent exactly 10% of the possible unique values (depending on number of unique values the percentage will actually be 10% +/- 1%)
• a half dozen sample runs all generated exactly 100 lines (ie, 10 unique values) in small.csv

If OP still needs to generate the two intermediate (sorted) files (uniqueValues.txt and uniqueValues.10pct.txt) then this could be done in the same awk script via an END {...} block, eg:

END { PROCINFO["sorted_in"]="@ind_num_asc"          # this line of code requires GNU awk otherwise OP can sort the files at the OS/bash level
      for (i in seen)
          print i > "uniqueValues.txt"
      for (i in keep)
          print i > "uniqueValues.10pct.txt"        # use with 1st awk script
        # print i > "uniqueValues." pct "pct.txt"   # use with 2nd awk script
    }