most efficient way to join array with separators in awk

j = a[1]
for ( i=2; i<=max; i++ ) 
    j = j sep a[i]

is the clearest and most common approach so IMHO it's what should be used even if a tiny performance improvement could be squeezed out of a slightly less clear script.

The time when you typically see a ternary being used is when printing the contents of the array as:

for ( i=1; i<=max; i++ ) 
    printf "%s%s", a[i], (i<max ? OFS : ORS)

instead of the slightly more efficient but lengthier and requiring the printf formatting string and associated data values to be duplicated before the loop for the first index and inside the loop for all of the other indices:

printf "%s", a[1]
for ( i=2; i<=max; i++ ) 
    printf "%s%s", OFS, a[i]
print ""

Just yesterday I was surprised by a huge performance improvement using a recursive descent function instead of a loop, see https://stackoverflow.com/a/68389244/1745001, so I thought I'd give that a try but it failed miserably:

$ head -50 tst*.awk
==> tstLoop.awk <==
BEGIN {
    max = 64
    for (i=1; i<=max; i++) {
        a[i] = sprintf("%50s",i)
    }
    for (n=1; n<=100000; n++) {
        j = a[1]
        for (i=2; i<=max; i++) {
            j = j OFS a[i]
        }
    }
}

==> tstRec.awk <==
function join(i){
    if( i == 1 ) return a[i]
    return join(i-1) OFS a[i]
}

BEGIN {
    max = 64
    for (i=1; i<=max; i++) {
        a[i] = sprintf("%50s",i)
    }
    for (n=1; n<=100000; n++) {
        j = join(max)
    }
}

$ time awk -f tstLoop.awk

real    0m2.596s
user    0m2.561s
sys     0m0.030s

$ time awk -f tstRec.awk

real    0m3.743s
user    0m3.733s
sys     0m0.000s

I do not know if it is better in the sense that the overall performance is in any way better than the for loop, but it is more clean for regular record processing use cases where the array is implicitly given from the record numbers:

awk 'FNR==1 {printf $1} FNR>1 {printf " "$1} END {printf "\n"}'

I understand that this is not what the Author really asked for, but it will likely be what people finding this post are really looking for.

UPDATE 2 : quick note on joining direction

I've observed that gawk and mawk each have a preferred joining direction that's much much faster than the other direction (as in roughly 1 order of magnitude), but the directions are also reversed between the 2 :

• mawk : much faster when continuously prepend-ing to the string
• gawk : much faster when continuously append-ing to the string

================================================================

@jhnc : i'm not too sure how well your suggested method scales :

• 75,369 rows | 13.3 MB utf-8 .txt file,

  splitting and join by \n

echo; fg;fg; f="${m2m}"; xxh128sum "${f}" | ecp; echo; 
sleep 1; 
( time ( pvE0 < "${f}" | 
       LC_ALL=C mawk2x 'BEGIN { FS=RS="^$"; OFS=ORS=_="" 
                        } END { __=split($-_,___,____="\n"); 
                                print \
                                fastjoin(___,____,!_,__) }' ) |
pvE9 ) | xxh128sum| lgp3 | ecp
sleep 1
( time ( pvE0 < "${f}" | 

  LC_ALL=C mawk2 'BEGIN { FS=RS="^$"; OFS=ORS=_="" 
                  } END { __=split($-_,___,____="\n"); 
                          _____=___[__]_^=_<_;
                          while (_<__) { _____=___[--__](____)_____ }
                          print _____ }' ) | 
pvE9 ) | xxh128sum | lgp3 | ecp

fg: no current job
fg: no current job
 
c023ca75edcabbdd1253859a27e08c5f  /………./m2vid_main.txt     
 

      in0: 13.3MiB 0:00:00 [ 138MiB/s] [ 138MiB/s] [=============>] 100%            
     out9: 13.3MiB 0:00:00 [97.7MiB/s] [97.7MiB/s] [<=> ]

( pvE 0.1 in0 < "${f}" | LC_ALL=C mawk2x ; )  

0.13s user 0.04s system 109% cpu 0.159 total

c023ca75edcabbdd1253859a27e08c5f  stdin


      in0: 13.3MiB 0:00:00 [2.27GiB/s] [2.27GiB/s] [========>] 100%            
     out9: 13.3MiB 0:00:38 [ 356KiB/s] [ 356KiB/s] [<=>]
( pvE 0.1 in0 < "${f}" | LC_ALL=C mawk2 ; )  

15.51s user 22.63s system 99% cpu 38.152 total

c023ca75edcabbdd1253859a27e08c5f  stdin

For a 2nd test file a few more orders of magnitude in size :

• 8.12 mn rows | 153MB ASCII-only .txt file,

  split and join by \n

699ddf5bda2530a07d9651fe0b43a624  /……/m2map_main.txt     
 
  8125949 160950671 160950671     /……/m2map_main.txt
 

      in0:  153MiB 0:00:00 [1.10GiB/s] [1.10GiB/s] [============>] 100%            
     out9:  153MiB 0:00:01 [ 108MiB/s] [ 108MiB/s] [<=> ]
( pvE 0.1 in0 < "${f}" | LC_ALL=C mawk2x ; )  

1.09s user 0.33s system 98% cpu 1.438 total

699ddf5bda2530a07d9651fe0b43a624  stdin

 
# it's still running after 23 mn 10 secs, so I halted it
 
      in0:  153MiB 0:00:00 [2.86GiB/s] [2.86GiB/s] [============>] 100%            
     out9: 0.00 B **0:23:10** [0.00 B/s] [0.00 B/s] [<=>  

What I realized was that for really large inputs, regardless of which awk flavor, neither pure linear or pure recursion is ideal, but a combination of both is closer to optimal :

  —- fewer than, say, 100-200 bytes, then join them linearly

  —- otherwise, keep recursing downwards

UPDATE 1 ::

9.94 secs to split out a 12.49 mn cell array then join it back into a single 
1.85 GB string should at least be semi-respectable speed for scripting code
   

|

    {m,g}awk '
     BEGIN {
     1      FS=RS = "^$"
     1       ____ = "="
     1      _____ = ORS
     1       split( ORS = OFS = _="",___)
    } END {
     1      print six4join(___,_____,!_,_=\
                     split($+_,___,(_____)))
     1      printf(\
               "\n\n\n %.f \n\n\n",_) >("/dev/stderr")
    }
524287  function six4join(__,_____,_,____,___,______)
        {
524287      if ((______="") == ___) { # 1
     1          for (______ in __)  { break  }
     1          if(!(______ in __)) { return }
            }
524287         _=    _!=(______="")? +_    :+_ in __?+_:++_
524287      ____= ____!= ______    ? +____ : length(__)-!_
524287       ___=  ___!= ______    ? +___  : ____-_

524287      if((______*=______+=++______)^--______<___ || ___<______) {
262143          return ______<___\
                ? six4join(__,_____,_,___=_+int(___/--______), ___-_)\
                  (_____) six4join(__,_____,++___,____,____-___) \
                : +___ == --______\
                ? __[_](___=_____) __[--____]___ __[++____] \
                : +___ <-___ ?"":__[_] (-___<+___ ?(_____) __[____]:""))
            }
262144          ___ = __[_]
262144       ______+= ++______
262144         ____ = (_____) substr("", _____=____)

1746372     while((_____-_++) % ______) {
1746372         ___ = (___)____ __[_]
            }

1310720     while(_<+_____) {
1310720         ___ = (___)____ __[_++]____ __[_++]____ __[_++]____\
                    __[_++]____ __[_++]____ __[_++]____ __[_++]____ __[_++]
            }

262144      return ___
        }

f9d2e18d22eb58e5fc2173863cff238e stdin

rows = 12494275. | UTF8 chars = 1285316715. | bytes = 1983544693.


   in0:  295MiB 0:00:00 [2.88GiB/s] [2.88GiB/s] [>] 15% ETA 0:00:00
   in0: 1.85GiB 0:00:00 [3.00GiB/s] [3.00GiB/s] [===>] 100%
  out9: 1.65GiB 0:00:09 [2.85GiB/s] [ 174MiB/s] [ <=> ]
  out9: 1.85GiB 0:00:09 [ 190MiB/s] [ 190MiB/s] [ <=> ]
  

 12494276   # this figure is always expected to be 
            # 1 more than row count whenever 
            # there is a trailing `\n`

( pvE 0.1 in0 < "${f}" | LC_ALL=C mawk2 ; ) 

5.42s user 4.22s system 96% cpu 9.942 total

f9d2e18d22eb58e5fc2173863cff238e stdin