Sed!

Given template.txt:

The number is ${i}
The word is ${word}

we just have to say:

sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt

Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation.

Use /bin/sh. Create a small shell script that sets the variables, and then parse the template using the shell itself. Like so (edit to handle newlines correctly):

File template.txt:

the number is ${i}
the word is ${word}

File script.sh:

#!/bin/sh

#Set variables
i=1
word="dog"

#Read in template one line at the time, and replace variables (more
#natural (and efficient) way, thanks to Jonathan Leffler).
while read line
do
    eval echo "$line"
done < "./template.txt"

Output:

#sh script.sh
the number is 1
the word is dog

Here's a robust Bash function that - despite using eval - should be safe to use.

All ${varName} variable references in the input text are expanded based on the calling shell's variables.

Nothing else is expanded: neither variable references whose names are not enclosed in {...} (such as $varName), nor command substitutions ($(...) and legacy syntax `...`), nor arithmetic substitutions ($((...)) and legacy syntax $[...]).

To treat a $ as a literal, \-escape it; e.g.:\${HOME}

Note that input is only accepted via stdin.

Example:

$ expandVarsStrict <<<'$HOME is "${HOME}"; `date` and \$(ls)' # only ${HOME} is expanded
$HOME is "/Users/jdoe"; `date` and $(ls)

Function source code:

expandVarsStrict(){
  local line lineEscaped
  while IFS= read -r line || [[ -n $line ]]; do  # the `||` clause ensures that the last line is read even if it doesn't end with \n
    # Escape ALL chars. that could trigger an expansion..
    IFS= read -r -d '' lineEscaped < <(printf %s "$line" | tr '`([$' '\1\2\3\4')
    # ... then selectively reenable ${ references
    lineEscaped=${lineEscaped//$'\4'{/\${}
    # Finally, escape embedded double quotes to preserve them.
    lineEscaped=${lineEscaped//\"/\\\"}
    eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
  done
}

^{The function assumes that no 0x1, 0x2, 0x3, and 0x4 control characters are present in the input, because those chars. are used internally - since the function processes text, that should be a safe assumption.}

Here is a way to get the shell to do the substitution for you, as if the contents of the file were instead typed between double quotes.

Using the example of template.txt with contents:

The number is ${i}
The word is ${word}

The following line will cause the shell to interpolate the contents of template.txt and write the result to standard out.

i='1' word='dog' sh -c 'echo "'"$(cat template.txt)"'"'

Explanation:

• i and word are passed as environment variables scopped to the execution of sh.
• sh executes the contents of the string it is passed.
• Strings written next to one another become one string, that string is:
  • 'echo "' + "$(cat template.txt)" + '"'
• Since the substitution is between ", "$(cat template.txt)" becomes the output of cat template.txt.
• So the command executed by sh -c becomes:
  • echo "The number is ${i}\nThe word is ${word}",
  • where i and word are the specified environment variables.

If you are open to using Perl, that would be my suggestion. Although there are probably some sed and/or AWK experts that probably know how to do this much easier. If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily, but you might just as well put it into a standard Perl script at that point.

perl -p -e 's/\$\{dbName\}/testdb/s' yourfile | mysql

A short Perl script to do something slightly more complicated (handle multiple keys):

#!/usr/bin/env perl
my %replace = ( 'dbName' => 'testdb', 'somethingElse' => 'fooBar' );
undef $/;
my $buf = <STDIN>;
$buf =~ s/\$\{$_\}/$replace{$_}/g for keys %replace;
print $buf;

If you name the above script as replace-script, it could then be used as follows:

replace-script < yourfile | mysql

It can be done in bash itself if you have control of the configuration file format. You just need to source (".") the configuration file rather than subshell it. That ensures the variables are created in the context of the current shell (and continue to exist) rather than the subshell (where the variable disappear when the subshell exits).

$ cat config.data
    export parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA
    export parm_user=pax
    export parm_pwd=never_you_mind

$ cat go.bash
    . config.data
    echo "JDBC string is " $parm_jdbc
    echo "Username is    " $parm_user
    echo "Password is    " $parm_pwd

$ bash go.bash
    JDBC string is  jdbc:db2://box7.co.uk:5000/INSTA
    Username is     pax
    Password is     never_you_mind

If your config file cannot be a shell script, you can just 'compile' it before executing thus (the compilation depends on your input format).

$ cat config.data
    parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA # JDBC URL
    parm_user=pax                              # user name
    parm_pwd=never_you_mind                    # password

$ cat go.bash
    cat config.data
        | sed 's/#.*$//'
        | sed 's/[ \t]*$//'
        | sed 's/^[ \t]*//'
        | grep -v '^$'
        | sed 's/^/export '
        >config.data-compiled
    . config.data-compiled
    echo "JDBC string is " $parm_jdbc
    echo "Username is    " $parm_user
    echo "Password is    " $parm_pwd

$ bash go.bash
    JDBC string is  jdbc:db2://box7.co.uk:5000/INSTA
    Username is     pax
    Password is     never_you_mind

In your specific case, you could use something like:

$ cat config.data
    export p_p1=val1
    export p_p2=val2
$ cat go.bash
    . ./config.data
    echo "select * from dbtable where p1 = '$p_p1' and p2 like '$p_p2%' order by p1"
$ bash go.bash
    select * from dbtable where p1 = 'val1' and p2 like 'val2%' order by p1

Then pipe the output of go.bash into MySQL and voila, hopefully you won't destroy your database :-).

In place perl editing of potentially multiple files, with backups.

  perl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' \
    -i.orig \
    -p config/test/*

I created a shell templating script named shtpl. My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:

$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}


$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4: 

More info on my github

To me this is the easiest and most powerful solution, you can even include other templates using the same command eval echo "$(<template.txt):

Example with nested template

1. create the template files, the variables are in regular bash syntax ${VARIABLE_NAME} or $VARIABLE_NAME

you have to escape special characters with \ in your templates otherwhise they will be interpreted by eval.

template.txt

Hello ${name}!
eval echo $(<nested-template.txt)

nested-template.txt

Nice to have you here ${name} :\)

2. create source file

template.source

declare name=royman 

3. parse the template

source template.source && eval echo "$(<template.txt)"

4. the output

Hello royman!
Nice to have you here royman :)

envsubst

please don't use anything else (ie. don't eval)