How to combine all files into one file, as I want

Admin 1
Viewed 154

There are 1000 txt files in a folder. The contents of the files are as follows:

("a1", "b1")

I want to combine all the files into one file. However, I have to separate the contents of each file with commas.

("a1", "b1"), ("a2", "b2")

Then I need to add a fixed text at the beginning and end as below.

START ("a1", "b1"), ("a2", "b2") END

However, as long as the number of files to be added is 100, it should create a new line.

Example:

START ("a1", "b1"), ("a2", "b2") ... END
...
START ("a900", "b900"), ("a901", "b901") ... END

The values ​​I have given in the example are just examples to explain in a simpler way.

I need to write this as a bash script to run on the server.

I was able to do something like the following, but I don't think it's good code. Unfortunately I was unable to create a new line in every 100 files.

echo "START " > all.txt
cat *.txt >> all.txt
sed -i 's/)(/),(/g' all.txt
echo " END" >> all.txt

Frankly, I need your help as it is a very complex code for me.

I am open to your valuable comments and suggestions.

Thank you.

sh ubuntu awk linux bash
4 Answers

Here an attempt to do it with an awk one-liner (I used 3 columns for easier demonstration):

awk 'BEGIN{printf "START "} NR == 1 { printf "%s", $0; next} NR % 3 == 1 { printf " END\nSTART %s", $0;next}  {printf ", %s", $0} END { print " END"}' file*

$ awk 'BEGIN{printf "START "} NR == 1 { printf "%s", $0; next} NR % 3 == 1 { printf " END\nSTART %s", $0;next}  {printf ", %s", $0} END { print " END"}' file*
START (10,  0), (11,  0), (12,  0) END
START (13,  0), (14,  0), (15,  0) END
START (16,  0), (17,  0), (18,  0) END
START (19,  0), (20,  0), (21,  0) END
START (22,  0), (23,  0), (24,  0) END
START (25,  0), (26,  0), (27,  0) END
START (28,  0), (29,  0), (30,  0) END
START (31,  0), (32,  0), (33,  0) END
START (34,  0), (35,  0), (36,  0) END
START (37,  0), (38,  0), (39,  0) END
START (40,  0), (41,  0), (42,  0) END
START (43,  0), (44,  0), (45,  0) END
START (46,  0), (47,  0), (48,  0) END
START (49,  0), (50,  0), (51,  0) END
START (52,  0), (53,  0), (54,  0) END
START (55,  0), (56,  0), (57,  0) END
START (58,  0), (59,  0), (60,  0) END
START (61,  0), (62,  0), (63,  0) END
START (64,  0), (65,  0), (66,  0) END
START (67,  0), (68,  0), (69,  0) END
START (70,  0), (71,  0), (72,  0) END
START (73,  0), (74,  0), (75,  0) END
START (76,  0), (77,  0), (78,  0) END
START (79,  0), (80,  0), (81,  0) END
START (82,  0), (83,  0), (84,  0) END
START (85,  0), (86,  0), (87,  0) END
START (88,  0), (89,  0), (90,  0) END
START (91,  0), (92,  0), (93,  0) END
START (94,  0), (95,  0), (96,  0) END
START (97,  0), (98,  0), (99,  0) END

Or in a more readable format:

BEGIN { printf "START " }
NR == 1 { printf "%s", $0; next }
NR % 3 == 1 { printf " END\nSTART %s", $0; next }
{ printf ", %s", $0 }
END { print " END" }

If you really have a large list of files, you could get argument list too long error. To fix this, use xargs instead.

echo file* | xargs awk '......' > joined

This looks like it would have the same issue, but because echo is builtin to bash, it is not bound by the argument length limitation.

user_8281930

Something like this (untested) should do it:

awk '
    { rec = (rec=="" ? "" : rec ", ") $0 }
    (NR%100) == 0 {
        print "START", rec, "END"
        rec = ""
    }
' *
user_17450010

I know you asked for it in shell - but technically, if you prepend the Python solution with a python -c ", it is shell, is not it? [wink]

#! /usr/bin/env python3
import os

contents = []

with ('all.txt', 'wt') as output:
    for filename in os.listdir('.'):
       if filename == 'all.txt': continue
       contents.append(open(filename).read().strip())
       if len(contents) >= 100:
           output.write(f'''START {', '.join(contents)} END\n''')
           contents = []
    output.write(f'''START {', '.join(contents)} END\n''')
user_1082050

Bash variant

n=1; while read -r line; do
    ((n==1))  &&   printf 'start '
                   printf "$line"
    ((n>=10)) && { printf ' end\n';   n=1  ; } \
              || { printf ', '    ; ((n++)); }
done < <(cat file*)
user_126074430
Related