Terraform interpreting locals expression when it shouldn't be

Question

I have a Kafka client container running on AWS ECS, which should receive a regular expression for which topics to subscribe to from the containers environment variables. In my terraform I define the environment variable like so:

..
      {
        "name": "KAFKA_TOPIC_REGEX",
        "value": "${local.kafka_topic_regex[local.environment]}"
      }
..

With a corresponding value in the locals.tf file like so:

..
  kafka_topic_regex = {
    development = "^(ab\..*\..*)$|^(foo\.bar)$"
  }
..

However I'm having difficulty getting this to work as expected. When I terraform plan, the following error is shown:

The symbol "." is not a valid escape sequence selector.

Unfortunately I need the backward slash there for the regex to use the literal dot as a character in the topic name. I can't escape the backward slash as I actually require it be there for the regex.

I have equally tried the regex like so:

^(ab[.].*[.].*)$|^(foo[.]bar)$

Again, Terraform does some strange interpretation of this and in the plan the value appears as:

^ab.(.*)

This will mean that the literal dot I require isn't in the regex as it is required to be escaped in the regex.

It appears Terraform is interpreting my string and causing a failed plan / apply, when it shouldn't be interpreting it.

Any help would be appreciated, thank you

user_281848 · Answer

Unfortunately this is a typical sort of situation that results when source code in one language is written as a string in another language: you need to deal with multiple levels of escaping. Although it may seem generous to talk about regular expressions as "another language", from Terraform's perspective that's exactly what they are, albeit a very terse language with a very specific featureset.

The requirement here is to write an expression that Terraform can evaluate to produce a string containing a regular expression that the regular expression engine can then interpret. The resulting regular expression string must contain literal backslashes, as you noted, because backslashes are part of the regular expression language.

We can produce a string containing a literal backslash in the Terraform language by escaping the backslash with an additional backslash:

"\"

The above would, from Terraform's standpoint, produce a string containing just a single backslash.

We can extend that idea to your regular expression problem by escaping all of the backslashes in your regular expression:

  kafka_topic_regex = {
    development = "^(ab\..*\..*)$|^(foo\.bar)$"
  }

When Terraform evaluates that quoted string it will respond to the escaped backslashes by producing a single literal backslash, and so the resulting string will be ^(ab\..*\..*)$|^(foo\.bar)$ as the regular expression engine is expecting. You can then pass that result on to the regular expression engine for processing, at which point it should see the \. sequences and (as you intended) treat it as a literal . rather than as the "any single character" wildcard.

One "trap" to watch out for is that Terraform uses a variant of its own quoted string syntax when rendering string values in the UI for situations like describing a planned change. In doing so, it will re-introduce the backslash escaping necessary to unambiguously render the result on-screen, showing it as ^(ab\..*\..*)$|^(foo\.bar)$. Generally when reading values Terraform shows on-screen in its output we should understand them as representations of constant values in Terraform's own language, not as the literal values that will be used. This is similar to how Terraform will show numbers on-screen as sequences of decimal digits, like 1234, rather than showing the raw binary value that really represents that value in memory inside Terraform.

user_11579537 · Answer

It seems the plan / apply breakdown of what is actually applied is wrong. The regex which escapes the dot using square brackets will apply with the correct value but is displayed incorrectly on plan and apply. Working version: ^(ab[.].*[.].*)$|^(foo[.]bar)$

Terraform interpreting locals expression when it shouldn't be

2 Answers