HTML Parser removing all the strings separated by & (ampersands)

user_12569908 0
Viewed 81

I'm using a customization of the HTMLParser in order to remove HMTL tags and other things. The problem came out with this string Q&A

from html.parser import HTMLParser
from io import StringIO

class MyParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.reset()
        self.strict = False
        self.convert_charrefs= True
        self.text = StringIO()
    def handle_data(self, d):
        self.text.write(d)
    def get_data(self):
        return self.text.getvalue()

def strip_tags(html):
    s = MyParser()
    s.feed(html)
    return s.get_data()

strip_tags("Q&A")

The output is an empty string. In this case, I don't consider Q&A as a parsable string because it is the contraction of Question & Answer. The same occurs with other strings separated by ampersands like String&Otherstring.

It is probably recognized as a part of an URL (?). I tried to switch self.convert_charrefs= False and the output is only Q. I used this algorithm for many cases and it worked every time, this is the only case it doesn't work, so I'd like to change it as little as possible to preserve cases as [STRING]&[OTHER_STRING] (except URLs and HTML)

Any idea?

ampersand html-parsing parsing html python
1 Answers

HTMLParser returns an empty string, because it is expecting more data to 'complete' the HTML.

You should force the parser to process the string using HTMLParser.close().

from html.parser import HTMLParser
from io import StringIO


class MyParser(HTMLParser):
    def __init__(self):
        super().__init__()
        self.reset()
        self.strict = False
        self.convert_charrefs = True
        self.text = StringIO()

    def handle_data(self, d):
        self.text.write(d)

    def get_data(self):
        return self.text.getvalue()


def strip_tags(html):
    s = MyParser()
    s.feed(html)
    s.close() # <-- I added this
    return s.get_data()


strip_tags("Q&A")
user_77478370
Related