Invalid string: control characters from U+0000 through U+001F must be escaped using Bash?

Viewed 18909

I have a function like:

getServers() {
    curl -s -X GET ...
}

The output is a complex JSON string. I can parse the JSON like this:

serverId=$(getServers | jq -r ".[] | select(whatever...)")
echo $serverId

But if I store the output of the function call in a variable, I get an error:

servers=$(getServers)
echo $servers | jq .
# jq can not process this
# parse error: Invalid string: control characters from U+0000 through             
echo "$servers" | jq .
# does not work either

U+001F must be escaped at line ...

Even though I $servers contain the same value as function call, jq fails to process it. What's happening here?

$ jq --version
jq-1.5


This is not a duplicate question! This issue roots in existence of \r\n in the string. Adding quotes does not fix the problem.

Answer

Pipe the string into tr '\r\n' ' ':

servers=$(getServers)
echo $servers | tr '\r\n' ' ' | jq .

This script and its output show how adding " does not fix the problem, but tr does:

#!/bin/env bash

get() {
    x=$'multi\r\nline'
    echo $x
}


g=$(get)

echo 'Without "'
echo $g
echo 'With "'
echo "$g"
echo 'With tr'
echo $g | tr '\r\n' '\n'

Output:

Without "
 line
With "
 line
With tr
multi
 line

I hope community votes to reopen this question.

0 Answers
Related