How to find multi-line comments wrapped in quotes?

Viewed 224

I am parsing Python code, and I need to remove all possible comments/docstrings. I have successfully been able to remove "comments" of the form:

#comment
"""comment""" 
'''comment''' 

However, I have found some samples where people write comments of the form:

"'''comment'''" 
"\"\"\"\n comment  \"\"\""

I am struggling to successfully remove these comments (three single quotes surrounded by a double quote, and double quotes with line breaks). The expression I tried was:

p = re.compile("([\'\"])\1\1(.*?)\1{3}", re.DOTALL)
code = p.sub('', code)

But this did not work for either of the second two cases. Does anyone have any suggestions?

3 Answers

You could try using strip(). It works by removing the characters you place in between the brackets. If nothing is in the brackets it removes spaces but you want to remove the three single quotes surrounded by a double quote, and double quotes with line breaks. So an example is:

txt = ",,,,,rrttgg.....banana....rrr"
x = txt.strip(",.grt")
print(x)

And the output you would get is "banana" as it has removed the ,.grt that was found in between the double brackets( x = txt.strip(",.grt")).

For more info check out this page, and i recommend the info at the bottom for further help: https://www.w3schools.com/python/python_strings.asp

posting as an answer because my comment was hard to read

This is what I came up with, it's ugly and hacky but it does work.

import re

txt = "if x = 4: continue  \"'''hi'''\"  print(x) "
print(txt)
#find everything wrapped in double quotes
double_quotes = re.findall(r"\"(.+?)\"", txt)
for string in double_quotes:
    triple_single = re.findall(r"\'''(.+?)\'''", string)[0]
    full_comment = '"'+"'''" +triple_single+"'''"+'"'
    txt = txt.replace(full_comment, '')
    print(txt)

Prints:

if x = 4: continue  "'''hi'''"  print(x) 
if x = 4: continue    print(x)

Unassigned string literal can be considered as nodes on the source code's Abstract Syntax Tree (AST) representation. Then the problem is reduced to identifying these nodes and rewriting the AST without them, using the tools in the ast module.

Comments (# ...) are not parsed into the AST, so there is not need to code for them.

Unassigned string literals are nodes of type ast.Constant, and form part of the body attribute of nodes that have bodies, such as module definitions, function definitions and class definitions. We can identify these nodes, remove them from their parents' body's and then rewrite the AST.

import ast 
import io

from unparse import Unparser


with open('comments.py') as f:
    src = f.read()

root = ast.parse(src)

# print(ast.dump(root)) to see the ast structure.


def filter_constants(node):
    if isinstance(node, ast.Expr):
        if isinstance(node.value, ast.Constant):
            if isinstance(node.value.value, str):
                return None
    return node


class CommentRemover(ast.NodeTransformer):

    def visit(self, node):
        if hasattr(node, 'body'):
            node.body = [n for n in node.body if filter_constants(n)]
        return super().visit(node)


remover = CommentRemover()
new = remover.visit(root)
ast.fix_missing_locations(new)

buf = io.StringIO()
Unparser(new, buf)
buf.seek(0)
print(buf.read())

Calling the script on this code (comments.py):

"""Module docstring."""                                                                                                             


# A real comment                                                                                                                    
"""triple-double-quote comment"""                                                                                                   
'''triple-single-quote comment'''                                                                                                   

"'''weird comment'''"                                                                                                               
"\"\"\"\n comment  \"\"\""                                                                                                          

NOT_A_COMMENT = 'spam'                                                                                                              

42                                                                                                                                  


def foo():                                                                                                                          
    """Function docstring."""                                                                                                       
    # Function comment                                                                                                              
    bar = 'baz'                                                                                                                     
    return bar                                                                                                                      


class Quux:
    """class docstring."""

    # class comment

    def m(self):
        """method comment"""
        return

Gives this output:

NOT_A_COMMENT = 'spam'
42

def foo():
    bar = 'baz'
    return bar

class Quux():

    def m(self):
        return

Notes:

  • the unparse script can be found in your Python distribution's Tools/parser folder (in v3.8 - in previous versions it has been in Tools or in the Demo folder). It may also be downloaded from github - be sure that you download the version for your version of Python
  • As of Python 3.8, the ast.Constant class is used for all constant nodes; for earlier versions you may need to use ast.Num, ast.Str, ast.Bytes, ast.NameConstant and ast.Ellipsis as appropriate. So in filter_constants might look like this:

    def filter_constants(node):
        if isinstance(node, ast.Expr):
            if isinstance(node.value, ast.Str):
                return None
        return node
    
  • As of Python 3.9, the ast module provide an unparse function that may be used instead of the unparse script

    src = ast.unparse(new)
    print(src)
    
Related