Splitting a file with a very long comma separated string of numbers into multiple files

user_1491371 0
Viewed 218

I have a single text file that contains a very long string of comma separated number, e.g.:

15,0,-475,-1243,1,0,0,1456,....

The file contains over a million number like this, all on a single line. I would like to split the file into multiple files, with each file containing a fixed number of comma-separated numbers (17577 numbers per file). So the first file will contain the first 17577 comma separated number (from the source file), and the second file will pick up where the first file ended. Ideally each file will end with a number (not the trailing comma) although this isn't critical. If it matters, all of the numbers will be between -2000 and 2000.

I've been trying out methods using bash, like replacing all the commas with line breaks, then using split -l to split up the file, and then replacing all the line breaks back to commas. But I wasn't able to get that to work and am hoping there is a better way. Is there an efficient way to do this?

awk bash split
3 Answers

The file contains over a million number like this, all on a single line.

Ideally each file will end with a number (not the trailing comma) although this isn't critical.

Then I'd use GNU split for that.

split -t , -l 17577 file{,_}
user_102486780

You may try this awk for splitting based on record count after splitting input with record separator as comma:

awk -v n=17577 -v RS=, '{fn = "file-" int((NR-1) / n) + 1 ".txt"} NF {printf "%s", $0 (NR%n ? RS : ORS) > fn} END {print "" > fn}' file

A more readable version:

awk -v n=17577 -v RS=, '{
   fn = "file-" int((NR-1) / n) + 1 ".txt"
}
NF {
   printf "%s", $0 (NR%n ? RS : ORS) > fn
}
END {
   print "" > fn
}' file

Only caveat is that the very last split file may have a trailing comma.

user_5482250

the split solution of Oguz is again a wonderful shorthand. Here is still another approach using Gnu awk. You need Gnu awk as we will exploit the record separator:

$ awk -v RS='([-0-9]+,?){,17577}' '{print substr(RT,1,length(RT)-1) > (FILENAME FNR) }' file

unfortunately, we need to use substr here because of the ending , in the record RT.

You could also use:

$ awk -v RS='([-0-9]+,){,17576}[-0-9]+' '{print RT > (FILENAME FNR) }' file

Both these solutions work nicely, however, the last file will always be empty. This can still be solved, but it just makes the solution aesthetically, unenjoyable.

$ awk -v RS='([-0-9]+,){,17576}[-0-9]+' '!(!NF && c++){print RT > (FILENAME FNR) }' file
user_83440600
Related