Invalid pattern in vectorizer standardization

user_19376850 0
Viewed 27

I am trying to separate unique characters from words so I can threat them as separate tokens.

This is my vectorizer:

vectorizer = tf.keras.layers.TextVectorization(
                standardize=code_standaridization,
                split="whitespace",
                output_mode="int",
                output_sequence_length=params["input_dim"],
            )

And this is my function:

def code_standaridization(input_data):
    """
    Helps with finding correct embedding.
    """
    input_data = tf.strings.regex_replace(input_data, "-", " - ")
    input_data = tf.strings.regex_replace(input_data, "_", " _ ")
    input_data = tf.strings.regex_replace(input_data, "(", " ( ")
    input_data = tf.strings.regex_replace(input_data, ")", " ) ")
    input_data = tf.strings.regex_replace(input_data, "{", " { ")
    input_data = tf.strings.regex_replace(input_data, "}", " } ")
    input_data = tf.strings.regex_replace(input_data, "[", " [ ")
    input_data = tf.strings.regex_replace(input_data, "]", " ] ")
    input_data = tf.strings.regex_replace(input_data, '"', ' " ')
    input_data = tf.strings.regex_replace(input_data, "'", " ' ")
    input_data = tf.strings.regex_replace(input_data, ".", " . ")
    input_data = tf.strings.regex_replace(input_data, ",", " , ")

    return input_data

But I am getting this error:

Node: 'StaticRegexReplace_2'
Invalid pattern: (, error: missing ): (
     [[{{node StaticRegexReplace_2}}]] [Op:__inference_adapt_step_133]
data-preprocessing keras tensorflow python
1 Answers

regex_replace() uses the RE2 syntax defined here, which notes:

The simplest regular expression is a single literal character. Except for the metacharacters like *+?()|, characters match themselves. To match a metacharacter, escape it with a backslash: \+ matches a literal plus character.

So, you should add backslashes for replacing parentheses:

input_data = tf.strings.regex_replace(input_data, "\(", " \( ")
input_data = tf.strings.regex_replace(input_data, "\)", " \) ")
user_177698150
Related