I am trying to split a text file for every 1000th word.
awk -v RS='[[:space:]]+' 'END{print NR+0}' filename
with awk I can count the words in a file but I don't know how I can split it.
final output= filename(1).txt, filename(2).txt
I am trying to split a text file for every 1000th word.
awk -v RS='[[:space:]]+' 'END{print NR+0}' filename
with awk I can count the words in a file but I don't know how I can split it.
final output= filename(1).txt, filename(2).txt
This totally sick solution should work for files which are less than 10000 words:
. <(echo -e 'uno due tre\nquattro\ncinque sei sette otto\nnove dieci undici dodici tredici' | sed -zE '
s/^/\x0/
:a
y/012345678/123456789/
s/\x0(([^ \n]+[ \n]+){4})/cat > file0 <<EOF\n\1\nEOF\n\x0/
ta
s/\x0(.*)/cat > file0 <<EOF\n\1\nEOF\n\x0/
s/\n+/\n/g')
Essentially, it intersperses some code at the points where the splits have to occur in such a way that the outcoming file is a bash script which is a sequence of cat commands which read from a heredocument and write to a file (a maximum of 10 files is allowed!). This script is sourced (. file is just source file, just uglier). You can see the script by removing the leading . <( and the trailing ).
The nice thing is that it splits the big file in the middle of lines if necessary, without altering the lines where no split occurs.
The ugliest thing is that it numbers the files backward.
The limitation on the number of words is because I am implementing only a one-digit addition on the filenames; it can be removed by implementing an addition in a similar way as done here or here.
You can do it with awk without too much trouble. It helps keep the clutter down if you write a function to actually handle outputting the words from an array to your file. Keep a counter to number the output file names, e.g. wordsfile_1 (first 1000 words), wordsfile_2 (next 1000 words) and so on. Then it is just a matter of keeping track of how many words you add to your array and call your output function when you hit 1000 words. Then delete the array, to make it ready to hold the next 1000 words, reset your word counter and keep going.
For example you could do something like:
awk '
function writefile() {
fname="wordsfile_" ++c + 0
for (j=1; j<=n; j++)
print a[j] > fname
delete a
n = 0
}
{
for (i=1; i<=NF; i++) {
a[++n] = $i
if (n == 1000)
writefile()
}
}
END {
writefile()
}' input_file
The function writefile() handles writing the output to your 1000 word files and deleting the array and resetting the counter n. The END rule just calls the function once more to output any words collected since the last output.
Let me know if you have further questions.
#!/bin/bash
for FILE in *.txt
do
#FILE="FILENAME.txt"
read -p "HOW MANY WORDS SHOULD BE IN YOUR FILES? (~ APPROXIMATE) " BUFFER
#BUFFER=1000 # APPROXIMATE NUMBER OF WORDS IN A FILE
NW=$(wc -w $FILE | awk '{print $1}') #NW=NUMBER OF WORDS IN YOUR FILE
if [[ $NW -gt $BUFFER ]]
then
LINENUMBER=$(wc -l $FILE | awk '{print $1}')
WCOUNT=0
FL=1 #FIRST LINE NUMBER OF EVERY NEW FILE
FN=1 #FILE NUMBER
for j in $(eval echo "{1..$LINENUMBER}")
do
INC=$(sed -n "${j}p" $FILE | wc -w)
WCOUNT=$(( WCOUNT + INC ))
if [[ $WCOUNT -gt $BUFFER ]];
then
sed -n "${FL},${j}p" $FILE > ${FILE%%.*}_${FN}.txt
FL=$(( j + 1))
(( FN++ ))
WCOUNT=0
fi
done
sed -n "${FL},\$p" $FILE > ${FILE%%.*}_${FN}.txt
fi
done
I found a different solution, It generates files that have roughly 1000 words in each.