Using awk substr on fixed width file

user_5596534 0
Viewed 253

I have a fixed width file where I would like to split rows by a value defined by the first four characters into files called file_1985.dat & file_1986.dat. Take this minimal file: 

$ cat foo.dat
1985tiny dancer
1986largechicken
1985hey  jude

I would like to end up with this:

$ cat file_1985.dat
1985tiny dancer
1985hey  jude

and this:

$ cat file_1986.dat
1986largechicken

I am pretty sure I need to do something like this:

awk -F, '{if(???)print > "file_1985.dat";else print > "file_1986.dat"}' foo.dat

where ??? involves some use of substr. Can anyone advise here?

substr awk
3 Answers

Use substr($0, 1, 4) to get the 4 characters that you want to go in the filename (a substring starting at character 1 of length 4):

awk '{ 
  out = "file_" substr($0, 1, 4) ".dat" # set filename
  if (out != prev) close(prev)          # close previous file
  print >> out                          # write to file
  prev = out                            # remember filename to check on next line
}' foo.dat

The filename out comes from the result of concatenating the string literals with the result of substr. This variable is used to determine the file that print's output ends up in.

>> opens a file in "append mode", which means that if you re-open the same file, you don't lose the previous content.

Testing it out:

$ awk '{ out = "file_" substr($0, 1, 4) ".dat"; if (out != prev) close(prev); print >> out; prev = out  }' foo.dat
$ cat file_1985.dat 
1985tiny dancer
1985hey  jude
$ cat file_1986.dat 
1986largechicken

The use of close is a precaution to prevent opening too many files, but if your input isn't too big then you can simplify to just:

awk '{ print > ("file_" substr($0, 1, 4) ".dat") }' foo.dat

Regarding performance, you could try sorting the input, to avoid opening and closing the same files repeatedly (although sorting would itself take time):

sort -s -k1.1,1.4 foo.dat | awk '{ out = "file_" substr($0, 1, 4) ".dat"; if (out != prev) close(prev); print > out; prev = out  }'

Here I also changed the >> to > since awk will only open each file once.

You could also avoid repeating the same string concatenation once per line by caching the results:

{
  ss = substr($0, 1, 4)
  if (!(ss in outs)) {
    outs[ss] = "file_" ss ".dat"
  }
  out = outs[ss]
  if (out != prev) close(prev)
  print >> out
  prev = out
}

Put it in a script like script.awk and run it like awk -f script.awk foo.dat.

user_20881350 
gawk -v FIELDWIDTHS="4 200" '{ print $2 > "file_" $1 ".dat" }' foo.dat

from the man-page: If the FIELDWIDTHS variable is set to a space-separated list of numbers, each field is expected to have fixed width, and gawk splits up the record using the specified widths. Each field width may optionally be preceded by a colon-separated value specifying the number of characters to skip before the field starts. The value of FS is ignored. Assigning a new value to FS or FPAT overrides the use of FIELDWIDTHS.

user_7240390

If you're using GNU awk or you'll have fewer than about 12 output files then:

awk '{print > ("file_"substr($0,1,4)".dat")}' foo.dat

otherwise to avoid a "too many open files" error:

awk '{out="file_"substr($0,1,4)".dat"; print >> out; close(out)}' foo.dat
user_17450010
Related