Why does splitting my $PATH with `read -r -a line` work but not with `while read -r line`?

Viewed 122

Just noticed something strange which I can't quite explain:

When I split my $PATH variable using read -a everything works fine

IFS=: read -r -a lines <<< "$PATH"
for line in "${lines[@]}"; do echo "$line"; done

But when I try to do the same using while ... read loop, only the first line is printed

while IFS=: read -r line; do echo "$line"; done <<< "$PATH"
4 Answers

You can make this work; switch from using IFS=: to using -d:, and append a : to the end of your input stream:

while IFS= read -r -d: line; do echo "$line"; done <<< "$PATH:"

The difference is that IFS is used to find boundaries between words, but read -r line reads into exactly one variable, line, so it's not looking for multiple words at all. By contrast, -d tells each invocation of read which character to stop at; by default that's a newline, but you can replace it with any other single character. (If that character isn't found, read exits with a nonzero status; that's why the standard/idiomatic while read loop idiom skips the last line of your file if it isn't correctly terminated by a newline, and why we use $PATH: as our input here).

If you ran IFS=: read -r first second rest, on the other hand, it would put your first PATH entry into $first, the second one into $second, and the remainder of the line into $rest; whereas with IFS: read -r line, it's as if you only had a single item, $rest.

Your while loop processes 1 line, it is not a loop. So the complete path is stored in the field line.
When you had given more fields, the path would be divided to those fields (and the last field gets the remainder):

while IFS=: read -r line field2 field3 otherfields; do echo "$line"; done <<< "$PATH"

When you want to avoid an array, you can use

while read -r line; do echo "$line"; done <<< "${PATH//:/$'\n'}"

It works fine.

Splitting into an array gives an open-ended number of elements, so does what you expect.

Splitting into a single variable does the same thing, but when it runs out of supplied variable names into which to put the data, it's stops splitting and puts the rest into the last one.

Try this:

$: IFS=: read -r a b c <<< "$PATH"
$: printf "[%s]\n" "$a" "$b" "$c"

You'll get the first PATH element in $a, the second in $b, and the rest ALL in $c.

Does that make it clearer?

c.f. this guide

Why does splitting my $PATH with read -r -a line work but not with while read -r line?

Because read -r line reads the whole line and then after reading the single whole line then the line is spitt on IFS. Because you provided only one variable to read, all the line is in that one variable. You could like split the line on the first element and rest of elements:

IFS=: read -r part1 rest_of_parts <<<"$line"

See read 1p read the If there are fewer vars than fields, part. Note that still IFS=: read -r -a lines <<< "$PATH" will fail when PATH contains a newline, like so:

$ export PATH=/usr/bin       # reset PATH to something short
$ cd /tmp/
$ mkdir temp$'\n'dir         # create a directory with a newline in the name
$ ls -d tem*
'temp'$'\n''dir'/
$ cd temp$'\n'dir
$ printf "%s\n" '#!/bin/bash' 'echo hello world' > script.sh
$ chmod +x ./script.sh       # add a script in that directory
$ export PATH="$PATH:$PWD"   # add that directory to path
$ ./script.sh                # yes. yes, it works
hello world
$ IFS=: read -r -a lines <<< "$PATH"
$ declare -p lines
declare -a lines=([0]="/usr/bin" [1]="/tmp/temp")
#                                              ^^^^ newline and 'dir' is missing
# That is because `read` reads _one line_ and one line only
# _after_ reading that one line that _one line_ is split on IFS
# so any more lines are ignored.

You could use a bash extension to read -d that makes read not read the whole line, but up until a character (but I needed to ignore read exit status, dunno why):

$ while IFS= read -r -d':' line || [[ -n "$line" ]]; do declare -p line; done < <(printf "%s" "$PATH")
declare -- line="/usr/bin"
declare -- line="/tmp/temp
dir"

Note that <<< adds a trailing newline, so using that will result in the last element of PATH having a newline - as a workaround, in bash you may use process substitution < <(printf "%s" "$PATH").

The real safe solution if using bash is just using mapfile/readarray:

$ mapfile -d: -t lines < <(printf "%s" "$PATH")
$ declare -p lines
declare -a lines=([0]="/usr/bin" [1]=$'/tmp/temp\ndir')
Related