jq truncates ENV variable after whitespace

Question

Trying to write a bash script that replaces values in a JSON file we are running into issues with Environment Variables that contain whitespaces.

Given an original JSON file.

{
  "version": "base",
  "myValue": "to be changed",
  "channelId": 0
}

We want to run a command to update some variables in it, so that after we run:

CHANNEL_ID=1701 MY_VALUE="new value" ./test.sh

The JSON should look like this:

{
  "version": "base",
  "myValue": "new value",
  "channelId": 1701
}

Our script is currently at something like this:

#!/bin/sh

echo $MY_VALUE
echo $CHANNEL_ID

function replaceValue {
  if [ -z $2 ]; then echo "Skipping $1"; else jq --argjson newValue \"${2}\" '. | ."'${1}'" = $newValue' build/config.json > tmp.json && mv tmp.json build/config.json; fi
}

replaceValue channelId ${CHANNEL_ID}
replaceValue myValue ${MY_VALUE}

In the above all values are replaced by string and strings are getting truncated at whitespace. We keep alternating between this issue and a version of the code where substitutions just stop working entirely.

This is surely an issue with expansions but we would love to figure out, how we can: - Replace values in the JSON with both strings and values. - Use whitespaces in the strings we pass to our script.

user_5291015 · Answer

You don't have to mess with --arg or --argjson to import the environment variables into jq's context. It can very well read the environment on its own. You don't need a script separately, just set the values along with the invocation of jq

CHANNEL_ID=1701 MY_VALUE="new value" \
     jq '{"version": "base", myValue: env.MY_VALUE, channelId: env.CHANNEL_ID}' build/config.json

Note that in the case above, the variables need not be exported globally but just locally to the jq command. This allows you to not export multiple variables into the shell and pollute the environment, but just the ones needed for jq to construct the desired JSON.

To make the changes back to the original file, do > tmp.json && mv tmp.json build/config.json or more clearly download the sponge(1) utility from moreutils package. If present, you can pipe the output of jq as

| sponge build/config.json

user_9072753 · Answer

Pass variables with --arg. Do:

jq --arg key "$1" --arg value "$2" '.[$key] = $value'

Notes:

#!/bin/sh indicates that this is posix shell script, not bash. Use #!/bin/bash in bash scripts.
function replaceValue { is something from ksh shell. Prefer replaceValue() { to declare functions. Bash obsolete and deprecated syntax.
Use newlines in your script to make it readable.
--argjson passes a json formatted argument, not a string. Use --arg for that.
"${2}" doesn't quote $2 expansion - it only appends and suffixes the string with ". Because the expansion is not qouted, word splitting is performed, which causes your input to be split on whitespaces when creating arguments for jq.
Remember to quote variable expansions.
Use http://shellcheck.net to check your scripts.
. | means nothing in jq, it's like echo $(echo $(echo))). You could jq '. | . | . | . | . | .' do it infinite number of times - it passes the same thing. Just write the thing you want to do.

Do:

#!/bin/bash

echo "$MY_VALUE"
echo "$CHANNEL_ID"

replaceValue() {
    if [ -z "$2" ]; then
         echo "Skipping $1"
    else
        jq --arg key "$1" --arg value "$2" '.[$key] = $value' build/config.json > tmp.json &&
        mv tmp.json build/config.json
   fi
}

replaceValue channelId "${CHANNEL_ID}"
replaceValue myValue "${MY_VALUE}"

@edit Replaced ."\($key)" with easier .[$key]

user_4470895 · Answer

jq allows you to build new objects:

MY_VALUE=foo;
CHANNEL_ID=4
echo '{
  "version": "base",
  "myValue": "to be changed",
  "channelId": 0
}' | jq ". | {\"version\": .version, \"myValue\": \"$MY_VALUE\", \"channelId\": $CHANNEL_ID}"

The . selects the whole input, and inputs that (|) to the construction of a new object (marked by {}). For version is selects .version from the input, but you can set your own values for the other two. We use double quotes to allow the Bash variable expansion, which means escaping the double quotes in the JSON.

You'll need to adapt my snippet above to scriptify it.

jq truncates ENV variable after whitespace

3 Answers