replacing newlines with the string '\n' with POSIX tools

user_6646161 0
Viewed 484

Yes I know there are a number of questions (e.g. (0) or (1)) which seem to ask the same, but AFAICS none really answers what I want.

What I want is, to replace any occurrence of a newline (LF) with the string \n, with no implicitly assumed newlines... and this with POSIX only utilities (and no GNU extensions or Bashisms) and input read from stdin with no buffering of that is desired.

So for example:

  • printf 'foo' | magic should give foo
  • printf 'foo\n' | magic should give foo\n
  • printf 'foo\n\n' | magic should give foo\n\n

The usually given answers, don't do this, e.g.:

  • awk
    printf 'foo' | awk 1 ORS='\\n gives foo\n, whereas it should give just foo
    so adds an \n when there was no newline.
  • sed
    would work for just foo but in all other cases, like:
    printf 'foo\n' | sed ':a;N;$!ba;s/\n/\\n/g' gives foo, whereas it should give foo\n
    misses one final newline.
    Since I do not want any sort of buffering, I cannot just look whether the input ended in an newline and then add the missing one manually.
    And anyway... it would use GNU extensions.
    sed -z 's/\n/\\n/g'
    does work (even retains the NULs correctly), but again, GNU extension.
  • tr
    can only replace with one character, whereas I need two.

The only working solution I'd have so far is with perl:
perl -p -e 's/\n/\\n/'
which works just as desired in all cases, but as I've said, I'd like to have a solution for environments where just the basic POSIX utilities are there (so no Perl or using any GNU extensions).

Thanks in advance.

sh sed posix awk shell
3 Answers

The following will work with all POSIX versions of the tools being used and with any POSIX text permissible characters as input whether a terminating newline is present or not:

$ magic() { { cat -u; printf '\n'; } | awk -v ORS= '{print sep $0; sep="\\n"}'; }

$ printf 'foo' | magic
foo$

$ printf 'foo\n' | magic
foo\n$

$ printf 'foo\n\n' | magic
foo\n\n$

The function first adds a newline to the incoming piped data to ensure that what awk is reading is a valid POSIX text file (which must end in a newline) so it's guaranteed to work in all POSIX compliant awks and then the awk command discards that terminating newline that we added and replaces all others with "\n" as required.

The only utility above that has to process input without a terminating newline is cat, but POSIX just talks about "files" as input to cat, not "text files" as in the awk and sed specs, and so every POSIX-compliant version of cat can handle input without a terminating newline.

user_17450010

You can (I think) do this with pure POSIX shell. I am assuming you are working with text, not arbitrary binary data that can include null bytes.

magic () {
  while read x; do
      printf '%s\\n' "$x"
  done
  printf '%s' "$x"
}

read assumes POSIX text lines (terminated with a newline), but it still populates x with anything it reads until the end of its input when no linefeed is seen. So as long as read succeeds, you have a proper line (minus the linefeed) in x that you can write back, but with a literal \n instead of a linefeed.

Once the loop breaks, output whatever (if anything) in x after the failed read, but without a trailing literal \n.

$ [ "$(printf foo | magic)" = foo ] && echo passed
passed
$ [ "$(printf 'foo\n' | magic)" = 'foo\n' ] && echo passed
passed
$ [ "$(printf 'foo\n\n' | magic)" = 'foo\n\n' ] && echo passed
passed
user_11268410

Here is a tr + sed solution that should work on any POSIX shell as it doesn't call any gnu utility:

printf 'foo' | tr '\n' '\7' | sed 's/\x7/\\n/g'
foo

printf 'foo\n' | tr '\n' '\7' | sed 's/\x7/\\n/g'
foo\n

printf 'foo\n\n' | tr '\n' '\7' | sed 's/\x7/\\n/g'
foo\n\n

Details:

  • tr command replaces each line break with \x07
  • sed command replace each \x07 with \\n
user_5482250
Related