Validate (X)HTML in Python

cdleary_3594 0
Viewed 25409

What's the best way to go about validating that a document follows some version of HTML (prefereably that I can specify)? I'd like to be able to know where the failures occur, as in a web-based validator, except in a native Python app.

validation xhtml html python
9 Answers

XHTML is easy, use lxml.

from lxml import etree
from StringIO import StringIO
etree.parse(StringIO(html), etree.HTMLParser(recover=False))

HTML is harder, since there's traditionally not been as much interest in validation among the HTML crowd (run StackOverflow itself through a validator, yikes). The easiest solution would be to execute external applications such as nsgmls or OpenJade, and then parse their output.

John_Millikin_35600

You can decide to install the HTML validator locally and create a client to request the validation.

Here I had made a program to validate a list of urls in a txt file. I was just checking the HEAD to get the validation status, but if you do a GET you would get the full results. Look at the API of the validator, there are plenty of options for it.

import httplib2
import time

h = httplib2.Http(".cache")

f = open("urllistfile.txt", "r")
urllist = f.readlines()
f.close()

for url in urllist:
   # wait 10 seconds before the next request - be nice with the validator
   time.sleep(10)
   resp= {}
   url = url.strip()
   urlrequest = "http://qa-dev.w3.org/wmvs/HEAD/check?doctype=HTML5&uri="+url
   try:
      resp, content = h.request(urlrequest, "HEAD")
      if resp['x-w3c-validator-status'] == "Abort":
         print url, "FAIL"
      else:
         print url, resp['x-w3c-validator-status'], resp['x-w3c-validator-errors'], resp['x-w3c-validator-warnings']
   except:
      pass
user_622620

Try tidylib. You can get some really basic bindings as part of the elementtidy module (builds elementtrees from HTML documents). http://effbot.org/downloads/#elementtidy

>>> import _elementtidy
>>> xhtml, log = _elementtidy.fixup("<html></html>")
>>> print log
line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 1 column 7 - Warning: discarding unexpected </html>
line 1 column 14 - Warning: inserting missing 'title' element

Parsing the log should give you pretty much everything you need.

zacherates_26030

The html5lib module can be used to validate an HTML5 document:

>>> import html5lib
>>> html5parser = html5lib.HTMLParser(strict=True)
>>> html5parser.parse('<html></html>')
Traceback (most recent call last):
  ...
html5lib.html5parser.ParseError: Unexpected start tag (html). Expected DOCTYPE.
user_27297780

I think that HTML tidy will do what you want. There is a Python binding for it.

user_6190

This is a very basic html validator based on lxml's HTMLParser. It is not a complete html validator, but does a few basic checks, doesn't require any internet connection, and doesn't require a large library.

_html_parser = None
def validate_html(html):
    global _html_parser
    from lxml import etree
    from StringIO import StringIO
    if not _html_parser:
        _html_parser = etree.HTMLParser(recover = False)
    return etree.parse(StringIO(html), _html_parser)

Note that this will not check for closing tags, so for example, the following will pass:

validate_html("<a href='example.com'>foo")
> <lxml.etree._ElementTree at 0xb2fd888>

However, the following wont:

validate_html("<a href='example.com'>foo</a")
> XMLSyntaxError: End tag : expected '>', line 1, column 29
user_2342700
Related