bash variable only retaining last line of grep

user_3783002 0
Viewed 100

I'm trying to write a git hook which would allow me to collect unit testing reports post commit.

I'm on a windows machine but using bash for the hook itself. It seems to run fine. Here's what I have so far:

#!/bin/sh
test_summary=$(pytest -v | grep -E ".+\.py::.+[A-Z]+")
commit_id=$(git rev-parse HEAD)

The issue is that the variable test_summary retains only the last line from the grep match. I know my piped pytest/grep is fine since running it on an interactive msys terminal gives the correct output with all the matches:

$ pytest -v | grep -E ".+.py::.+[A-Z]+"
test_main.py::test_main PASSED                                           [ 33%]
test_main.py::test2 FAILED                                               [ 66%]
test_main.py::test3 PASSED                                               [100%]

test_summary contains the following:

$ echo $test_summary
 test_main.py::test3 PASSED [100%]3%]

Why does it only retain the last line plus some mumbo jumbo from the first line?

I don't get it....

windows bash
2 Answers

tl;dr

OP is running these commands in a windows environment so (in this case) the variable is being loaded with \r\n line endings.

One idea for addressing this issue is to use parameter substitution:

$ test_summary=$(pytest -v | grep -E ".+\.py::.+[A-Z]+")
$ test_summary="$(test_summary//$'\r'/}"                   # strip out '\r' characters

$ echo $test_summary
test_main.py::test_main PASSED                                           [ 33%]
test_main.py::test2 FAILED                                               [ 66%]
test_main.py::test3 PASSED                                               [100%]

Continue for details ...

We can simulate this behavior with the following:

$ cat pytest.out
test_main.py::test_main PASSED                                           [ 33%]
test_main.py::test2 FAILED                                               [ 66%]
test_main.py::test3 PASSED                                               [100%]

# convert to windows/dos line endings:

$ unix2dos pytest.out

# load variable:

$ test_summary=$(cat pytest.out)

# verify '\r\n' lines endings loaded into variable:

$ od -c <<< "$test_summary"
... snip ...  3   3   %   ]  \r \n  t   e   s   t ... snip ...

# eumlate OP's echo/output:

$ echo $test_summary
 test_main.py::test3 PASSED [100%]3%]     <== '\r' causes lines to overwrite the same one line

Wrapping $test_summary in double quotes helps:

$ echo "$test_summary"
test_main.py::test_main PASSED                                           [ 33%]
test_main.py::test2 FAILED                                               [ 66%]
test_main.py::test3 PASSED                                               [100%]

Alternatively we can strip the \r line endings from the variable, eg:

$ test_summary="$(test_summary//$'\r'/}"
$ od -c <<< "$test_summary"
... snip ...  3   3   %   ]  \n  t   e   s   t ... snip ...

$ echo $test_summary
test_main.py::test_main PASSED [ 33%] test_main.py::test2 FAILED [ 66%] test_main.py::test3 PASSED [100%]

# still need to wrap in double quotes to properly display all spaces and line endings:

$ echo "$test_summary"
test_main.py::test_main PASSED                                           [ 33%]
test_main.py::test2 FAILED                                               [ 66%]
test_main.py::test3 PASSED                                               [100%]

A couple other ideas for stripping out the \r characters before saving the output in the variable:

$ test_summary=$(pytest -v | grep -E ".+\.py::.+[A-Z]+" | tr -d '\r')
$ test_summary=$(pytest -v | grep -E ".+\.py::.+[A-Z]+" | sed 's/\r//g')
$ test_summary=$(pytest -v | grep -E ".+\.py::.+[A-Z]+" | dos2unix)

The removal of the \r characters could also be applied after the pytest call and before the grep call (eg, pytest -v | dos2unix | grep -E ...).

user_73661000

Give it a try with mapfile:

...
mapfile -t test_summary <(pytest -v | grep -E ".+\.py::.+[A-Z]+") 
...

or

mapfile -t test_summary <<< $(pytest -v | grep -E ".+\.py::.+[A-Z]+")
user_77876540
Related