Process escape sequences in a string in Python

user_319821 0
Viewed 125253

Sometimes when I get input from a file or the user, I get a string with escape sequences in it. I would like to process the escape sequences in the same way that Python processes escape sequences in string literals.

For example, let's say myString is defined as:

>>> myString = "spam\\neggs"
>>> print(myString)
spam\neggs

I want a function (I'll call it process) that does this:

>>> print(process(myString))
spam
eggs

It's important that the function can process all of the escape sequences in Python (listed in a table in the link above).

Does Python have a function to do this?

escaping python string
8 Answers

The (currently) accepted answer by Jerub is correct for python2, but incorrect and may produce garbled results (as Apalala points out in a comment to that solution), for python3. That's because the unicode_escape codec requires its source to be coded in latin-1, not utf-8, as per the official python docs. Hence, in python3 use:

>>> myString="špåm\\nëðþ\\x73"
>>> print(myString)
špåm\nëðþ\x73
>>> decoded_string = myString.encode('latin-1','backslashreplace').decode('unicode_escape')
>>> print(decoded_string)
špåm
ëðþs

This method also avoids the extra unnecessary roundtrip between strings and bytes in metatoaster's comments to Jerub's solution (but hats off to metatoaster for recognizing the bug in that solution).

user_55834430

This is a bad way of doing it, but it worked for me when trying to interpret escaped octals passed in a string argument.

input_string = eval('b"' + sys.argv[1] + '"')

It's worth mentioning that there is a difference between eval and ast.literal_eval (eval being way more unsafe). See Using python's eval() vs. ast.literal_eval()?

user_111508470

Quote the string properly so that it looks like the equivalent Python string literal, and then use ast.literal_eval. This is safe, but much trickier to get right than you might expect.

It's easy enough to add a " to the beginning and end of the string, but we also need to make sure that any " inside the string are properly escaped. If we want fully Python-compliant translation, we need to account for the deprecated behaviour of invalid escape sequences.

It works out that we need to add one backslash to:

  • any sequence of an even number of backslashes followed by a double-quote (so that we escape a quote if needed, but don't escape a backslash and un-escape the quote if it was already escaped); as well as

  • a sequence of an odd number of backslashes at the end of the input (because otherwise a backslash would escape our enclosing double-quote).

Here is an acid-test input showing a bunch of difficult cases:

>>> text = r'''\\ \ \" \\" \\\" \'你好'\n\u062a\xff\N{LATIN SMALL LETTER A}"''' + '\\'
>>> text
'\\\\ \\ \\" \\\\" \\\\\\" \\\'你好\'\\n\\u062a\\xff\\N{LATIN SMALL LETTER A}"\\'
>>> print(text)
\\ \ \" \\" \\\" \'你好'\n\u062a\xff\N{LATIN SMALL LETTER A}"\

I was eventually able to work out a regex that handles all these cases properly, allowing literal_eval to be used:

>>> def parse_escapes(text):
...     fixed_escapes = re.sub(r'(?<!\\)(\\\\)*("|\\$)', r'\\\1\2', text)
...     return ast.literal_eval(f'"{fixed_escapes}"')
...

Testing the results:

>>> parse_escapes(text)
'\\ \\ " \\" \\" \'你好\'\nتÿa"\\'
>>> print(parse_escapes(text))
\ \ " \" \" '你好'
تÿa"\

This should correctly handle everything - strings containing both single and double quotes, every weird situation with backslashes, and non-ASCII characters in the input. (I admit it's a bit difficult to verify the results by eye!)

user_5236120
Related